Optimizer not taking Random Inputs

Ayush_Singhal: <p>I was just curious about the NNs. So I did a small experiment. I took a simple Linear Layer <code>nn.Linear(2 , 2)</code>. The training is simple</p>
<pre><code class="lang-auto">losses = []

for f , t in zip(fea , tar):

    preds = layer(f)

    loss = loss_f(preds , t)
    losses.append(loss)

    optim.zero_grad()
    loss.backward()

    optim.step()
</code></pre>
<p>Here the optimzers is <code>torch.optim.Adam(layer.parameters())</code> and loss is <code>nn.MSELoss()</code>. It works fine and updates the weights/parameters. But as soon as I do a slight change in optimizer as</p>
<pre><code class="lang-auto">x = nn.Parameter(torch.rand(2 , 2) , dtype = torch.float32)

optim = torch.optim.Adam([x])
</code></pre>
<p>The weights are not updated</p>
<pre><code class="lang-auto">Parameter containing:
tensor([[0.5747, 0.4457],
        [0.5285, 0.7974]], requires_grad=True)
</code></pre>
<pre><code class="lang-auto">{'params': [Parameter containing:
  tensor([[0.5747, 0.4457],
          [0.5285, 0.7974]], requires_grad=True)],
 'lr': 0.001,
 'betas': (0.9, 0.999),
 'eps': 1e-08,
 'weight_decay': 0,
 'amsgrad': False,
 'maximize': False,
 'foreach': None,
 'capturable': False,
 'differentiable': False,
 'fused': None}
</code></pre>
<p>Why even after the same size, it is not changing and even if the shape is different. do otpimizer really check if it is the actual weights of a layer …?</p>
<p>Is there any way, where I can make this thing work…?</p>

ptrblck: <aside class="quote no-group" data-username="Ayush_Singhal" data-post="1" data-topic="186861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ayush_singhal/48/62393_2.png" class="avatar"> Ayush_Singhal:</div>
<blockquote>
<p><code>x = nn.Parameter(torch.rand(2 , 2) , dtype = torch.float32)</code></p>
</blockquote>
</aside>
<p>Is wrong as <code>nn.Parameter</code> won’t accept a <code>dtype</code> argument and will fail with:</p>
<pre data-code-wrap="python"><code class="lang-python">TypeError: Parameter.__new__() got an unexpected keyword argument 'dtype'
</code></pre>
<p>Fixing this issue properly allows <code>x</code> to be optimized:</p>
<pre data-code-wrap="python"><code class="lang-python">
x = nn.Parameter(torch.rand(2 , 2, dtype = torch.float32))
optim = torch.optim.Adam([x], lr=1.)

for _ in range(10):
    print(x)
    out = x * 2
    loss = out.mean()
    optim.zero_grad()
    loss.backward()
    optim.step()
</code></pre>
<p>Output:</p>
<pre data-code-wrap="python"><code class="lang-python">Parameter containing:
tensor([[0.6680, 0.9391],
        [0.6531, 0.9501]], requires_grad=True)
Parameter containing:
tensor([[-0.3320, -0.0609],
        [-0.3469, -0.0499]], requires_grad=True)
Parameter containing:
tensor([[-1.3320, -1.0609],
        [-1.3469, -1.0499]], requires_grad=True)
Parameter containing:
tensor([[-2.3320, -2.0609],
        [-2.3469, -2.0499]], requires_grad=True)
Parameter containing:
tensor([[-3.3320, -3.0609],
        [-3.3469, -3.0499]], requires_grad=True)
Parameter containing:
tensor([[-4.3320, -4.0609],
        [-4.3469, -4.0499]], requires_grad=True)
Parameter containing:
tensor([[-5.3320, -5.0609],
        [-5.3469, -5.0499]], requires_grad=True)
Parameter containing:
tensor([[-6.3320, -6.0609],
        [-6.3469, -6.0499]], requires_grad=True)
Parameter containing:
tensor([[-7.3320, -7.0609],
        [-7.3469, -7.0499]], requires_grad=True)
Parameter containing:
tensor([[-8.3320, -8.0609],
        [-8.3469, -8.0499]], requires_grad=True)
</code></pre>

Ayush_Singhal: <p>Sorry for the typo in <code>nn.Parameter()</code></p>
<p>I have one last question. Does the shape play a vitol role…? Like does the working change if I give the shape of <code>(4 , 4)</code> rather than <code>(2 , 2)</code>…?</p>

ptrblck: <p>Yes, the shape is important for the used operation. A parameter of <code>4x4</code> will also work depending on the way you are planning on using this parameter.</p>

Ayush_Singhal: <p>Thank you for solving my issue <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>