Implement bit-flips during training

username: soepblik
cooked: <p>Hi,<br>
I read multiple papers about implementing bitflips during training to make your neural network more resistant against random bit flips.</p>
<p>i found two tools:</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/pytorchfi/pytorchfi" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/13f7d31553aa9de1e4f2b7b4809866fde7c4fe13706925c89d2d2f5a725426dc/pytorchfi/pytorchfi" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorchfi/pytorchfi" target="_blank" rel="noopener nofollow ugc">pytorchfi/pytorchfi</a></h3>


  <p><span class="label1">A runtime fault injection tool for PyTorch :fire:. Contribute to pytorchfi/pytorchfi development by creating an account on GitHub.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
and</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/bfgoldstein/torchfi" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://repository-images.githubusercontent.com/174930342/72796780-ee89-11e9-8433-9062eea7d803" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/bfgoldstein/torchfi" target="_blank" rel="noopener nofollow ugc">bfgoldstein/torchfi</a></h3>


  <p><span class="label1">TorchFI is a fault injection framework build on top of PyTorch for research purposes.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>they are mainly focussing on images.</p>
<p>I just have a small NN(one relu layer) with numerical inputs and outputs</p>
<p>I made a kind of algorithm that takes one bias/weight from the input/relu or output layer and converts that to binary. Do a bitflip and converts it back.</p>
<p>I was thinking of implementing this after the feedforward stage. Now everytime i get a runtime error:</p>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [8, 4]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>This is my code:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    running_loss = 0
    for i, (inputs, labels) in enumerate(train_loader):
        inputs = torch.from_numpy(inputs).to(torch.float32)
        labels = torch.from_numpy(labels).to(torch.float32)

        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        
        net = bitflip(net)

        loss.backward()
        
        optimizer.step()

        running_loss += loss.item()
</code></pre>
<p>How can i implement the bitflip algorithm during training without getting the error?</p>

username: soepblik
cooked: <p>Someone who can help me?</p>