In place functions by reference

pytorcher: <p>Is there a way to do in place operations when you are accessing a tensor by reference?  As an example I’d like to be able to change the original temp calling sigmoid (and possibly other operations more complex than +=  ) on temp2.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; temp = {0:torch.zeros(4), 1:torch.zeros(4)}
&gt;&gt;&gt; temp2 = temp[0]
&gt;&gt;&gt; temp2+=1
&gt;&gt;&gt; temp
{0: tensor([1., 1., 1., 1.]), 1: tensor([0., 0., 0., 0.])}
&gt;&gt;&gt; temp2 = temp2.sigmoid()
&gt;&gt;&gt; temp2
tensor([0.7311, 0.7311, 0.7311, 0.7311])
&gt;&gt;&gt; temp
{0: tensor([1., 1., 1., 1.]), 1: tensor([0., 0., 0., 0.])}

</code></pre>

KFrank: <p>Hi Pytorcher!</p>
<aside class="quote no-group" data-username="pytorcher" data-post="1" data-topic="121697" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>Is there a way to do in place operations when you are accessing a tensor by reference?</p>
</blockquote>
</aside>
<p>Yes, you can use <code>temp2.copy_ (temp2.sigmoid())</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

pytorcher: <p>That did it, thanks!</p>

pytorcher: <p>Actually, it looks like that worked for simple cases but now that I’m running a more complex model it is giving me a:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
<p>I ran with with torch.autograd.set_detect_anomaly(True) and it does look like its the copy operations, specifically I am running a batch_norm as you can see in this block of the stack.</p>
<pre><code class="lang-auto">  File "/home/my_layer.py", line 431, in forward
    mainOut.copy_( self.batchNormLayer(mainOut))
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/nn/functional.py", line 1670, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled

Traceback (most recent call last):
  File "main.py", line 617, in &lt;module&gt;
    main()
  File "main.py", line 144, in main
    main_worker(args.gpu, ngpus_per_node, args)
  File "main.py", line 319, in main_worker
    train(train_loader, model, criterion, optimizer, scheduler, epoch, args)
  File "main.py", line 431, in train
    loss.backward()
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/tensor.py", line 166, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/me/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 512, 7, 7]], which is output 0 of CudnnConvolutionBackward, is at version 3; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

</code></pre>

KFrank: <p>Hi Pytorcher!</p>
<aside class="quote no-group quote-modified" data-username="pytorcher" data-post="4" data-topic="121697" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
</blockquote>
</aside>
<p>Inplace operations can “break the computation graph” and therefore<br>
prevent proper backpropagation.  (The autograd engine flags this for<br>
you, I guess because it can lead to hard-to-debug errors.)</p>
<p>But you originally asked:</p>
<aside class="quote no-group">
<blockquote>
<p>Is there a way to do in place operations when you are accessing a tensor by reference?</p>
</blockquote>
</aside>
<p>Why is it that you want to perform an inplace operation, and how<br>
would you want this operation to interact with backpropagation?</p>
<p>Would it make sense not to use an inplace operation and just let<br>
autograd do its normal thing?</p>
<p>Best.</p>
<p>K. Frank</p>

pytorcher: <p>Yes, I do want autograd to just do its normal thing.</p>
<p>Basically the situation I have is that sometimes I want my module to get a single tensor, and sometimes a dictionary of tensors like in the example.  At the top of my call I want to do something like:</p>
<pre><code class="lang-auto">forward(input)
if type(input) is dictionary
    mainOut = input[0]
else:
    mainOut = input
...
do stuff with mainOut including either
mainOut = self.batchNormLayer(mainOut)
or
mainOut.copy_( self.batchNormLayer(mainOut))
...
return input
</code></pre>
<p>Maybe just at the end I should do another dictionary check and set input[0] to be mainOut again, i.e. you can’t do an actual reference like c++ with python like I was thinking here because of the way the in place operations work like that.</p>

KFrank: <p>Hi Pytorcher!</p>
<aside class="quote no-group" data-username="pytorcher" data-post="6" data-topic="121697" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>at the end I should do another dictionary check and set input[0] to be mainOut again</p>
</blockquote>
</aside>
<p>This is exactly right (if I understand your use case).  Thus, something<br>
like:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward (input):
    if type (input) is dict:
        some_tensor = input[0]
    else:
        some_tensor = input
    
    # do stuff with some_tensor
    # then do
    main_out = self.batchNormLayer (some_tensor)
    
    if type (input) is dict:
        input[0] = main_out
        return input
    else:
        return main_out

</code></pre>
<p>Best.</p>
<p>K. Frank</p>