Inplace Operations when using Pyrfm

Huzaifa_Capricorn: <pre><code> from pyrfm import OrthogonalRandomFeature,CompactRandomFeature,RandomFourier,FastFood

 class Kernel_Mapping(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
          transformer = RandomFourier(n_components=10,
                                kernel='rbf',
                                use_offset=True, random_state=0)
    

    
    inputs = input.detach().numpy()
    #inputs = input.numpy()
    inputs_trans = transformer.fit_transform(inputs)
    ## Do I want to save the transformed for backward?
    ctx.save_for_backward(input)
    return torch.as_tensor(inputs_trans, dtype=input.dtype)

@staticmethod
def backward(ctx,g):
    return g

 loader = DataLoader(dataset, batch_size=len(dataset), shuffle=False)
generator = torch.Generator()
#device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
#model = model.to(device)
for inputs, targets in loader:
    sensitive_attributes = (inputs[:, sensitive_attribute_idx])[:, None]
    inputs = drop_attribute_tensor(inputs,40)
    inputs = inputs.to(device)
    
    targets = targets.to(device)
    print(f'Local Contribution {type(inputs)}')
    #import pdb;pdb.set_trace()
   
    #targets = targets
    #inputs = inputs.1
    outputs = model(inputs)
    #outputs = model(inputs[:,:-1])
    Z_outputs= Kernel_Mapping.apply(outputs.clone())
    Z_sens_attr  = Kernel_Mapping.apply(sensitive_attributes)

    
    ## Are they sampled independently?
    phi_hat = torch.normal(0., 1.0/R, (len(dataset), R), generator=generator.manual_seed(random_seed))
    omega_hat = torch.normal(0., 1.0/T, (len(dataset), T), generator=generator.manual_seed(int(random_seed/2)))

    phi_sens_attr = phi_hat.T@Z_sens_attr
    phi_outputs = phi_hat.T@Z_outputs
    omega_sens_attr = omega_hat.T@Z_sens_attr
    omega_outputs = omega_hat.T@Z_outputs


 inner_arg_sens = matrices['phi_s']@ matrices['omega_s'].T
  inner_arg_output = matrices['omega_f'] @matrices['phi_f'].T
            
            inner_arg = inner_arg_sens @ inner_arg_output
            fair_loss = torch.trace(inner_arg)
            
            fair_loss = (params["fairness_weight"]/( (len(train_dataset) -1)**2) ) * fair_loss
            
            
            #reg_loss = (1.0)/(2.0*step_size) * (model_l2(subtract_models(client_model, global_model)))
            
            loss = loss_func(outputs, targets) + params["fairness_weight"]/( (len(train_dataset) -1)**2)  * torch.trace(inner_arg)
            
            #overall_loss +=loss.item()
            f_loss += fair_loss.item()
            #torch.autograd.set_detect_anomaly(True)
            loss.backward(retain_graph = True)



The error:   RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [32, 1]], which is output 0 of AsStridedBackward0, is at version 146; expected version 145 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>The Runtime error seems to be when I use feature_mapping on outputs. When I use :<br>
Z_outputs= Kernel_Mapping.apply(outputs.detach()) I see no error but that<br>
is wrong.</p>

KFrank: <p>Hi Huzaifa!</p>
<aside class="quote no-group quote-modified" data-username="Huzaifa_Capricorn" data-post="1" data-topic="188633" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/huzaifa_capricorn/48/51194_2.png" class="avatar"> Huzaifa_Capricorn:</div>
<blockquote>
<pre><code>            loss.backward(retain_graph = True)
            optimizer.step()
</code></pre>
<p>I get the following error when I use a NN but no error when using BinaryLogisticRegression:</p>
<pre><code> RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 1]], which is output 0 of TBackward, is at version 100; expected version 65 instead.
</code></pre>
</blockquote>
</aside>
<p>The likely immediate cause of your inplace-modification error is your use of<br>
<code>retain_graph = True</code>.  Try removing it.</p>
<p>Note that <code>optimizer.step()</code> counts as an inplace operation, so you normally<br>
don’t want the “retained graph” still around when you call it.</p>
<p>If you think that you actually do need <code>retain_graph = True</code>, make sure you<br>
that you understand why so that you can either avoid it or arrange things to<br>
avoid the inplace-modification error.</p>
<p>I don’t really understand what your code is doing – you can find some suggestions<br>
for debugging inplace-modification errors in this post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Good luck!</p>
<p>K. Frank</p>

Huzaifa_Capricorn: <p>Thanks K.Frank you see I do need to retain the graph But I have tried removing it and the issue still persists.Also, do you know why a NN would cause issue but not Logistic Regression in this case.</p>

Huzaifa_Capricorn: <p>Hi Frank so I figured out the section of code that was giving me error.Can you help me explain this error?</p>

KFrank: <p>Hi Huzaifa!</p>
<aside class="quote no-group" data-username="Huzaifa_Capricorn" data-post="3" data-topic="188633" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/huzaifa_capricorn/48/51194_2.png" class="avatar"> Huzaifa_Capricorn:</div>
<blockquote>
<p>I do need to retain the graph</p>
</blockquote>
</aside>
<p>Why?</p>
<aside class="quote no-group">
<blockquote>
<p>I have tried removing it and the issue still persists.</p>
</blockquote>
</aside>
<p>In that case, you might consider removing it (even if that isn’t correct for your<br>
use case) and debugging your inplace-modification error in this somewhat<br>
simpler setting.</p>
<aside class="quote no-group">
<blockquote>
<p>do you know why a NN would cause issue but not Logistic Regression in this case.</p>
</blockquote>
</aside>
<p>As a general comment not necessarily relevant to your issue, whether or not<br>
an inplace modification triggers an error depends on the details of how the<br>
modified tensor is used in the computation graph.  Perhaps you have a modified<br>
tensor in both cases but your “Logistic Regression” doesn’t use that modified<br>
tensor in a way that leads to inplace-modification error.</p>
<p>Best.</p>
<p>K. Frank</p>

KFrank: <p>Hi Huzaifa!</p>
<aside class="quote no-group" data-username="Huzaifa_Capricorn" data-post="4" data-topic="188633" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/huzaifa_capricorn/48/51194_2.png" class="avatar"> Huzaifa_Capricorn:</div>
<blockquote>
<p>I figured out the section of code that was giving me error.</p>
</blockquote>
</aside>
<p>It’s pretty hard to tell what’s going on with your code.  Not only has the code<br>
you originally posted been edited into something completely different (and the<br>
title of the post has completely changed), but most importantly, the error you<br>
quote:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [32, 1]], which is output 0 of AsStridedBackward0, is at version 146; expected version 145 instead. 
</code></pre>
</blockquote>
</aside>
<p>is quite different from the one you originally posted.  This suggests that you<br>
haven’t figured out the (only) section of code that is giving the error.</p>
<p>For reference, the original error you posted was:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 1]], which is output 0 of TBackward, is at version 100; expected version 65 instead.
</code></pre>
</blockquote>
</aside>
<p>(Different tensor shape, different backward operation, and different version<br>
numbers.)</p>
<aside class="quote no-group">
<blockquote>
<p>Can you help me explain this error?</p>
</blockquote>
</aside>
<p>Have you tried any of the debugging techniques I suggested in the post I linked<br>
to in my first reply?</p>
<p>My suggestion would be to figure out / guess which tensor is being modified<br>
based on its shape, and then use a divide-and-conquer strategy based on the<br>
<code>._version</code>s of the suspect tensor to located where it is modified inplace.  (And<br>
turn off <code>retain_graph = True</code> if doing so doesn’t make the error go away just<br>
to simplify the situation somewhat.)</p>
<p>Best.</p>
<p>K. Frank</p>

Huzaifa_Capricorn: <p>Yes I removed <code>retain_graph = True</code> in the code and completly reimplemented it.Now I get no such error and my code works.Thanks</p>