The issue of registering parameters using torch.nn.Parameter

xv_rong: <p>I encountered a problem while reading and studying torch code, which I have simplified to the following model. I am attempting to train a model with only one parameter to fit y = 1000x. I tried two ways to register my parameter in the model. a1 can be trained normally, but obviously does not consider torch.exp() during forward propagation, while a2 causes the following error:</p>
<pre data-code-wrap="bash"><code class="lang-bash">RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
</code></pre>
<p>When I add retain_graph=True to backward() based on the error, the model can be trained normally, but I found that the model’s parameter a2 has not been updated, only a1 has. Here are some of the outputs:</p>
<pre data-code-wrap="bash"><code class="lang-bash">Parameter containing:
tensor([2.7183], requires_grad=True) tensor([2.7183], grad_fn=&lt;ExpBackward0&gt;)
x * a1 = tensor([2.7183], grad_fn=&lt;MulBackward0&gt;), x * a2 = tensor([2.7183], grad_fn=&lt;MulBackward0&gt;)
a1 = Parameter containing:
tensor([22.6639], requires_grad=True), a2 = tensor([2.7183], grad_fn=&lt;ExpBackward0&gt;)
x * a1 = tensor([22.6639], grad_fn=&lt;MulBackward0&gt;), x * a2 = tensor([2.7183], grad_fn=&lt;MulBackward0&gt;)
a1 = Parameter containing:
tensor([42.2106], requires_grad=True), a2 = tensor([2.7183], grad_fn=&lt;ExpBackward0&gt;)
x * a1 = tensor([42.2106], grad_fn=&lt;MulBackward0&gt;), x * a2 = tensor([2.7183], grad_fn=&lt;MulBackward0&gt;)
a1 = Parameter containing:
tensor([61.3664], requires_grad=True), a2 = tensor([2.7183], grad_fn=&lt;ExpBackward0&gt;)
x * a1 = tensor([61.3664], grad_fn=&lt;MulBackward0&gt;), x * a2 = tensor([2.7183], grad_fn=&lt;MulBackward0&gt;)
a1 = Parameter containing:
tensor([80.1391], requires_grad=True), a2 = tensor([2.7183], grad_fn=&lt;ExpBackward0&gt;)
x * a1 = tensor([80.1391], grad_fn=&lt;MulBackward0&gt;), x * a2 = tensor([2.7183], grad_fn=&lt;MulBackward0&gt;)
a1 = Parameter containing:
tensor([98.5363], requires_grad=True), a2 = tensor([2.7183], grad_fn=&lt;ExpBackward0&gt;)
x * a1 = tensor([98.5363], grad_fn=&lt;MulBackward0&gt;), x * a2 = tensor([2.7183], grad_fn=&lt;MulBackward0&gt;)
a1 = Parameter containing:
tensor([116.5656], requires_grad=True), a2 = tensor([2.7183], grad_fn=&lt;ExpBackward0&gt;)
x * a1 = tensor([116.5656], grad_fn=&lt;MulBackward0&gt;), x * a2 = tensor([2.7183], grad_fn=&lt;MulBackward0&gt;)

</code></pre>
<p>I would like to know what the fundamental differences are between these two methods? Here is the model code:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
class my_model(torch.nn.Module):
    def __init__(self):
        super(my_model, self).__init__()
        self.a1 = torch.nn.Parameter(torch.exp(torch.Tensor([1.0])), requires_grad=True)
        self.a2 = torch.exp(torch.nn.Parameter(torch.Tensor([1.0]), requires_grad=True))
    def forward(self, x):
        return x * self.a1, x * self.a2

# y = 1000 * x
x, y = torch.Tensor([1]), torch.Tensor([1000])
model = my_model()
model.train()
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
for i in range(1000):
    optimizer.zero_grad()
    y1, y2 = model(x)
    loss = criterion(y1, y) + criterion(y2, y)
    loss.backward(retain_graph=True)
    optimizer.step()
    print(f"x * a1 = {y1}, x * a2 = {y2}")
    print(f"a1 = {model.a1}, a2 = {model.a2}")
</code></pre>
<p>Actually, the code I read is as follows:"</p>
<pre data-code-wrap="python"><code class="lang-python">if self.learned_temperature is True:
    self.temperature = (self.model_dim * torch.exp(torch.clamp(nn.Parameter(
        torch.Tensor([1]), requires_grad=True), min=0.005, max=5))).cuda()
</code></pre>
<p>from <a href="https://github.com/AkariAsai/ATTEMPT/blob/main/attempt/third_party/models/t5/modeling_t5.py" class="inline-onebox" rel="noopener nofollow ugc">ATTEMPT/attempt/third_party/models/t5/modeling_t5.py at main · AkariAsai/ATTEMPT · GitHub</a> line 916</p>

divinho: <p>have you tried using</p>
<pre><code class="lang-auto">torch.nn.Parameter(torch.exp(torch.Tensor([1.0])), requires_grad=True)
</code></pre>
<p>once you call torch.exp on a nn.Parameter I don’t think it’s a parameter anymore</p>