Training a linear layer with a 2D input

username: agt
cooked: <p>If the first linear layer has <code>in_features = 1</code> and I input <code>[1, 2, 3]</code> into the model, how will that linear layer be trained? Will it train it independently on 1, 2, and 3 so the layer keeps track of the gradient for each input, and then the optimizer will use the average of all their gradients? If so, is there a way to tell the optimizer to use a custom function instead of the average to combine them?</p>

username: googlebot
cooked: <p>nn.Linear just wraps matrix multiplication, where right matrix is trainable in_features x out_features map. So, left matrix (input) must have in_features columns (last dimension), and arbitary number of rows (non-last dimensions reshaped into single one).</p>
<p>I assume your [1,2,3] denotes a python list. Normal conversion to a tensor would produce a tensor with 3 columns, not compatible with in_features=1. Adding extra tail dimension (shape (3,1)) would enable independent training, doing same transform to all scalars.</p>
<p>I’m not sure I understand what you want from gradients, but if your final network output still has that 3 element dimension, you can scale loss along that dimension, this is similar to having weighted samples.</p>

username: agt
cooked: <p>How are the values updated by default? Will it just be an average of the 3 gradients as if they form a batch in gradient descent?</p>
<blockquote>
<p>but if your final network output still has that 3 element dimension</p>
</blockquote>
<p>What if the output has a 1 element dimension, then how will the previous layers with 3 elements train?</p>
<blockquote>
<p>…having weighted samples</p>
</blockquote>
<p>How do you do that in pytorch?</p>
<p>Thank you for your help</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="agt" data-post="3" data-topic="90625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>How are the values updated by default? Will it just be an average of the 3 gradients as if they form a batch in gradient descent?</p>
</blockquote>
</aside>
<p>Yes</p>
<aside class="quote no-group" data-username="agt" data-post="3" data-topic="90625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>What if the output has a 1 element dimension, then how will the previous layers with 3 elements train?</p>
</blockquote>
</aside>
<p>This means that you combine elements somewhere, so they’re not independent. Your 3 intermediate outputs are just like hidden features, so it makes less sense to train them as independent (1x3 vs 3x3 map) or scale gradients, as they’ll be entangled later anyway.</p>
<aside class="quote no-group" data-username="agt" data-post="3" data-topic="90625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>How do you do that in pytorch?</p>
</blockquote>
</aside>
<p>Pytorch loss classes have reduction=‘mean’ argument. If you use reduction=‘none’, you can adjust elementwise losses before computing mean.</p>

username: agt
cooked: <blockquote>
<p>…(1x3 vs 3x3 map) or scale gradients, as they’ll be entangled later anyway</p>
</blockquote>
<p>Where does 3x3 come from?<br>
What does it mean to scale gradients? Do you mean in a weighted sum?<br>
I don’t understand what is “entangled” and why it affects scaling gradients.</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="agt" data-post="5" data-topic="90625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>Where does 3x3 come from?</p>
</blockquote>
</aside>
<p>Sorry, that was incorrect. With input shape (3,1) you have 1 x out_features map. With usual shape (*,3) you have 3 x out_features map.</p>
<aside class="quote no-group" data-username="agt" data-post="5" data-topic="90625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>What does it mean to scale gradients? Do you mean in a weighted sum?</p>
</blockquote>
</aside>
<p>If you multiply elementwise losses by some tensor C, all gradients are also multiplied by C. So, that’s one of the ways to change gradients.</p>
<aside class="quote no-group" data-username="agt" data-post="5" data-topic="90625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>I don’t understand what is “entangled” and why it affects scaling gradients.</p>
</blockquote>
</aside>
<p>Nevermind. Basically, for 3d → 1d function, expresed with a neural net, 3 inputs usually only work together.</p>

username: agt
cooked: <p>But if I do have 3D -&gt; 1D, then how will the training work? If the second layer has 3D, but  the output is 1D, how will the weights for the second layer get updated? Will it calculate 3  separate gradients for all 3 elements only for the second layer, and deeper layers will have only 1 shared gradient?</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="agt" data-post="7" data-topic="90625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>If the second layer has 3D, but the output is 1D, how will the weights for the second layer get updated? Will it calculate 3 separate gradients for all 3 elements only for the second layer, and deeper layers will have only 1 shared gradient?</p>
</blockquote>
</aside>
<p>That sounds right.</p>
<p>But this description somehow ignores hidden layer and non-linearity, needed to get a universal function approximator.</p>
<p>So, minimalistic network looks like: 3-&gt;64-&gt;relu-&gt;1, it has linear maps: 3x64,64x1. Thing is, your proposed 1d linear transformations, prepended to this network, won’t do anything beyond what 3x64 map does.</p>
<p>You can of course prepend 1d non-linear transformations to that network, but I don’t see a point.</p>

username: agt
cooked: <p>My goal is for the network to accept an arbitrary number of inputs and to learn how to do inference with information from all of the inputs. So if I apply a 1x5 linear transformation to input <code>[[1], [2], [3]]</code>, the output will be 3x5 tensor <code>[[a,b,c,d,e],[f,g,h,i,j],[k,l,m,n,o]]</code>. Then if I aggregate the 3 outputs into <code>[[v,w,x,y,z]]</code>, the later 5xM layers will learn to use information from all inputs, while early layer will learn to represent them in a way that allows for a useful aggregate.</p>
<p>Am I misunderstanding something? Can you elaborate so I can understand better.</p>
<p>Thank you</p>

username: googlebot
cooked: <p>Ok, with variable input lengths, this makes some sense. But you’ll probably need a deep network (MLP) for 1d-&gt;5d transformation, as 1x5 linear map just represents a line in 5d space, not sure if that’s of any use with your aggregation method… In most scenarios, another linear map 5xH will be applied to that 5 column tensor, so at least some non-linearity is needed<br>
(otherwise (input @ 1x5) @ 5xH = input @ (1x5 @ 5xH) = input @ (1xH), so first linear layer is not needed).</p>

username: agt
cooked: <p>Alright, that makes sense!</p>
<p>Can you point me to the pytorch docs that implies the gradients of the first layers (before aggregation) will be averaged?</p>
<p>Do you have any superior idea for doing this kind of aggregation?</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="agt" data-post="11" data-topic="90625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>Can you point me to the pytorch docs that implies the gradients of the first layers (before aggregation) will be averaged?</p>
</blockquote>
</aside>
<p>Don’t know if such a thing exists. That’s just how gradients work with tensor broadcasting (if scalar from one of inputs is used multiple times, you sum relevent derivatives).</p>
<aside class="quote no-group" data-username="agt" data-post="11" data-topic="90625">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>Do you have any superior idea for doing this kind of aggregation?</p>
</blockquote>
</aside>
<p>That’s a bit too abstract. Without positional info, you’re basically limited to mean() and exponential smoothing. Self-attention maybe. If you allow input position to affect output, you’re in the realm of usual sequence handlers: rnn, transformer/attention, causal convolutions etc.</p>

username: agt
cooked: <p>Hmm, I guess I will read about tensor broadcasting docs then.</p>
<p>In my problem only the first input matter, then the next inputs (2nd, 3rd, 4th, etc.) order does not matter, so I think we can use a weighted sum. How would exponential smoothing and self-attention work here?</p>

username: googlebot
cooked: <p>Exponential smoothing is just a recursive weighted mean. But it is not permutation invariant, to model that you should use simple mean(). If first input is special, you’re free to handle is separately.</p>
<p>Self-attention - I don’t understand it well enough to elaborate. Its use for aggregation is described in <a href="https://arxiv.org/abs/1901.05761" rel="nofollow noopener">Attentive Neural Processes</a> paper.</p>