Prepending layer to the input layer

vr25: <p>Given a neural network, with 10 neurons in the input layer, 3 in the hidden and 1 in the output layer, is there a way to prepend a layer to the input layer?</p>
<p>For instance, I have a dataset of five 2D tensors (3x10) with one label value (for instance, the task is regression) for each.</p>
<p>I want to create a 1D tensor (dimension size=10) and then pass this to the input layer with 10 neurons. In order to create a 1D tensor, each row in a 2D tensor is multiplied by a value in weight tensor of size 3. This creates three weighted tensors which are finally added to create a final 1d tensor of size 10, then fed to the input layer.</p>
<p>My question is how to prepend this weight tensor and make sure the values are updated.</p>
<p>Thanks!</p>

ptrblck: <p>Based on your description, it seems you would like to create a <code>weight</code> parameter, which will scale the inputs, and which should be trainable.<br>
If that’s the case, you could initialize it as:</p>
<pre><code class="lang-python">weight = nn.Parameter(torch.randn(3).to(device))
</code></pre>
<p>and pass it with all other parameters to the optimizer:</p>
<pre><code class="lang-python">optimizer = optim.Adam([weight] + list(model.parameters()), lr=1e-3)
</code></pre>
<p>This will make sure to optimizer all passed parameters.<br>
Autograd will take care of creating the computation graph based on the applied operations.</p>

vr25: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96d64b67ffcebf8356a9b0bcc1c7958d4098cb7a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/96d64b67ffcebf8356a9b0bcc1c7958d4098cb7a" title="Notes_200124_074441_0f5_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96d64b67ffcebf8356a9b0bcc1c7958d4098cb7a_2_324x500.jpeg" alt="Notes_200124_074441_0f5_1" data-base62-sha1="lwmK9ZD2fmXNjwhqoJkTNt4jwtI" width="324" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96d64b67ffcebf8356a9b0bcc1c7958d4098cb7a_2_324x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96d64b67ffcebf8356a9b0bcc1c7958d4098cb7a_2_486x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96d64b67ffcebf8356a9b0bcc1c7958d4098cb7a_2_648x1000.jpeg 2x" data-dominant-color="F9F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Notes_200124_074441_0f5_1</span><span class="informations">1000×1542 59.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks a lot for your reply. It does partially answer my question. I have attached an image of what I am intending to achieve.</p>
<p>There are 5 document tensors (each is 3 x 10).</p>
<ol>
<li>I want to apply weights to individual rows</li>
<li>Sum these weighted rows to get a document vector which then goes as input to the neural network.</li>
<li>At the same time, I want the initial weights <strong>w</strong> to be updated too.</li>
</ol>
<p>Can you please help me figure this out, thanks?</p>