[Contributors Welcome] Add missing torch::nn layers in C++ API

yf225: <p>Currently, PyTorch C++ API is missing many <code>torch::nn</code> layers that are available in the Python API. As part of the Python/C++ API parity project, we would like to add all of the following layers to C++:</p>
<h2>Containers</h2>
<ul>
<li>[ ] ModuleDict</li>
<li>[ ] ParameterList</li>
<li>[ ] ParameterDict</li>
</ul>
<h2>Convolution layers</h2>
<ul>
<li>[ ] ConvTranspose1d</li>
<li>[ ] ConvTranspose2d</li>
<li>[ ] ConvTranspose3d</li>
<li>[ ] Unfold</li>
</ul>
<h2>Pooling layers</h2>
<ul>
<li>[ ] MaxUnpool1d</li>
<li>[ ] MaxUnpool2d</li>
<li>[ ] MaxUnpool3d</li>
<li>[ ] FractionalMaxPool2d</li>
<li>[ ] LPPool1d</li>
<li>[ ] LPPool2d</li>
<li>[ ] AdaptiveMaxPool1d</li>
<li>[ ] AdaptiveMaxPool2d</li>
<li>[ ] AdaptiveMaxPool3d</li>
<li>[ ] AdaptiveAvgPool1d</li>
<li>[ ] AdaptiveAvgPool2d</li>
<li>[ ] AdaptiveAvgPool3d</li>
</ul>
<h2>Padding layers</h2>
<ul>
<li>[ ] ReflectionPad1d</li>
<li>[ ] ReflectionPad2d</li>
<li>[ ] ReplicationPad1d</li>
<li>[ ] ReplicationPad2d</li>
<li>[ ] ReplicationPad3d</li>
<li>[ ] ZeroPad2d</li>
<li>[ ] ConstantPad1d</li>
<li>[ ] ConstantPad2d</li>
<li>[ ] ConstantPad3d</li>
</ul>
<h2>Non-linear activations (weighted sum, nonlinearity)</h2>
<ul>
<li>[ ] ELU</li>
<li>[ ] Hardshrink</li>
<li>[ ] Hardtanh</li>
<li>[ ] LeakyReLU</li>
<li>[ ] LogSigmoid</li>
<li>[ ] MultiheadAttention</li>
<li>[ ] PReLU</li>
<li>[ ] ReLU</li>
<li>[ ] ReLU6</li>
<li>[ ] RReLU</li>
<li>[ ] SELU</li>
<li>[ ] CELU</li>
<li>[ ] Sigmoid</li>
<li>[ ] Softplus</li>
<li>[ ] Softshrink</li>
<li>[ ] Softsign</li>
<li>[ ] Tanh</li>
<li>[ ] Tanhshrink</li>
<li>[ ] Threshold</li>
</ul>
<h2>Non-linear activations (other)</h2>
<ul>
<li>[ ] Softmin</li>
<li>[ ] Softmax</li>
<li>[ ] Softmax2d</li>
<li>[ ] LogSoftmax</li>
<li>[ ] AdaptiveLogSoftmaxWithLoss</li>
</ul>
<h2>Normalization layers</h2>
<ul>
<li>[ ] GroupNorm</li>
<li>[ ] SyncBatchNorm</li>
<li>[ ] InstanceNorm1d</li>
<li>[ ] InstanceNorm2d</li>
<li>[ ] InstanceNorm3d</li>
<li>[ ] LocalResponseNorm</li>
</ul>
<h2>Linear layers</h2>
<ul>
<li>[ ] Identity</li>
<li>[ ] Bilinear</li>
<li>[ ] Flatten</li>
</ul>
<h2>Dropout layers</h2>
<ul>
<li>[ ] AlphaDropout</li>
</ul>
<h2>Distance functions</h2>
<ul>
<li>[ ] CosineSimilarity</li>
<li>[ ] PairwiseDistance</li>
</ul>
<h2>Vision layers</h2>
<ul>
<li>[ ] PixelShuffle</li>
<li>[ ] Upsample</li>
<li>[ ] UpsamplingNearest2d</li>
<li>[ ] UpsamplingBilinear2d</li>
</ul>
<h2>DataParallel layers (multi-GPU, distributed)</h2>
<ul>
<li>[ ] DataParallel</li>
<li>[ ] DistributedDataParallel</li>
</ul>
<h2>Utilities</h2>
<ul>
<li>[ ] clip_grad_norm_</li>
<li>[ ] clip_grad_value_</li>
<li>[ ] parameters_to_vector</li>
<li>[ ] vector_to_parameters</li>
<li>[ ] weight_norm</li>
<li>[ ] remove_weight_norm</li>
<li>[ ] spectral_norm</li>
<li>[ ] remove_spectral_norm</li>
<li>[ ] PackedSequence</li>
<li>[ ] pack_padded_sequence</li>
<li>[ ] pad_packed_sequence</li>
<li>[ ] pad_sequence</li>
<li>[ ] pack_sequence</li>
</ul>
<p>If you see any layers in the list that you are interested in using, please let me know whether you would also be interested in contributing to its implementation in C++ API. Most layers would just call the corresponding ATen functions internally (e.g. C++ MaxPoolNd module: <a href="https://github.com/pytorch/pytorch/pull/24860/files" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/24860/files</a>, thanks <a class="mention" href="/u/shahriarss">@ShahriarSS</a> for the contribution! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">) and doesn’t require in-depth knowledge about how the layer performs computation, so the barrier to entry should be considerably low for C++ developers of all levels.</p>
<p>Thanks!</p>

rvarm1: <p>Interested in <code>clip_grad_norm_</code> and <code>clip_grad_value_</code></p>

ptrblck: 

pascal.soveaux: <p>Linear || Containers</p>

Ashutosh_Chapagain1: <p>I’d love to start with ConvTranspose 1D.</p>

Abhishek_Nadgeri: <p><a class="mention" href="/u/yf225">@yf225</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> I would love to start with Non-linear activations: ReLU, Tanh, Sigmoid and maybe even MultiheadAttention <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

mailcorahul: <p><a class="mention" href="/u/yf225">@yf225</a> Interested in Convolution or Vision layers.</p>

Jon_Tau: <p><a class="mention" href="/u/yf225">@yf225</a> I’m interested in implementing the mentioned <strong>Distance functions</strong> <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:">.</p>

kolasanichaitanya: <p>I can work on linear layers</p>

Ankit-Dhankhar: <p>Interested in <code>AdaptiveMaxPool</code> layers</p>

latentcodes: <p><a class="mention" href="/u/yf225">@yf225</a> I’m interested in all of the Vision layers</p>

davidpfahler: <p>I‘d be interested in contributing to the non-linear activations.</p>

gsp-27: <p>I can get started with the distance functions cosine similarity and pairwise distance function</p>

dendisuhubdy: <p><a class="mention" href="/u/yf225">@yf225</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> I’m interested to contribute. Excited of the C++ frontend API. Other than <a href="https://github.com/pytorch/pytorch/pull/24860/files" rel="nofollow noopener">obviously this example</a>, do you have any other examples?</p>

Nishan_Srishankar: <p><a class="mention" href="/u/yf225">@yf225</a> I’m interested in the Convolution layers and non-linear activations (basic+other)</p>

yf225: <p>Thank you all so much for your interests and I really appreciated it. I am currently working on cutting out the task for each layer (most of them involve adding the corresponding <code>torch::nn::functional</code> layer as well), and I will share with all of you (along with example PRs to get started) as soon as possible.</p>

gtamba: <p><a class="mention" href="/u/yf225">@yf225</a> : Would be happy to attempt to contribute code to Convolutions,Vision and Non-linearities if all tasks aren’t already picked up</p>

tancl: <p>Great to see the work on C++ API, I am not sure whether I am able to contribute in this but I could give a try, but definitely I could contribute on the use case, in which I will integrate it with other OSS (such as Scilab) to make the call to C++ API easier for others, especially beginners who have less programming background.</p>
<p>Thanks.</p>

kuroko1t: <p>i want to contribute to Non-linear activations or dataparallel layers if still in time</p>

ptrblck: