akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Super(model) in init

username: sugh7020
cooked: <pre><code class="lang-auto">class LinearRegressionModel(nn.Module):
   def __init__(self, input_dim, output_dim):
      super(LinearRegressionModel, self).__init__()
      self.linear = nn.Linear(input_dim, output_dim)

   def forward(self, x):
      out = self.linear(x)
      return out
</code></pre>
<p>Can any of you please explain the use of the statement:</p>
<pre><code class="lang-auto">super(LinearRegressionModel, self).__init__()
</code></pre>
<p>Thanks!</p>

username: ptrblck
cooked: <p>The <code>super</code> call delegates the function call to the parent class, which is <code>nn.Module</code> in your case.<br>
This is needed to initialize the <code>nn.Module</code> properly. Have a look at the <a href="https://docs.python.org/2/library/functions.html#super">Python docs</a> for more information.</p>

username: KarthikR
cooked: <p>Here is a link to learn more about ‘Super’ =&gt; <a href="https://realpython.com/python-super/" rel="nofollow noopener">https://realpython.com/python-super/</a></p>

username: cehrett
cooked: <p>Based on reading the link <a class="mention" href="/u/karthikr">@KarthikR</a> provided, it seems to me that in the context of the original code, in Python 3 the original <code>super</code> statement is completely equivalent to just:</p>
<pre><code class="lang-auto">super().__init__()
</code></pre>
<p>(and per the recommendations at that link, this simpler version is in fact preferred).</p>
<p>Can anyone confirm that this is correct?</p>

username: lfywork
cooked: <p>I think you are right. More information ref to <a href="https://stackoverflow.com/questions/19776056/the-difference-between-super-method-versus-superself-class-self-method" rel="noopener nofollow ugc">Martijn Pieters</a></p>

username: paulwings
cooked: <p>The reason to use super is so that child classes that may be using cooperative multiple inheritance will call the correct next parent class function in the Method Resolution Order (MRO).</p>
<p>In Python 3, we can call it like this:</p>
<pre><code class="lang-auto">class ChildB(Base):
    def __init__(self):
        super().__init__()
</code></pre>
<p>In Python 2, you were required to call super like this with the defining class’s name and self, but you’ll avoid this from now on because it’s redundant, slower (due to the name lookups), and more verbose (so update your Python if you haven’t already!):</p>
<p><code>        super(ChildB, self).__init__()</code></p>
<p>Without super, you are limited in your ability to use multiple <a href="http://net-informations.com/faq/oops/inheritancetype.htm" rel="noopener nofollow ugc">inheritance</a> because you hard-wire the next parent’s call:</p>
<p><code>        Base.__init__(self) # Avoid this.</code></p>
High throughput ResNet50 model for distributed training with Horovod

username: whatdhack
cooked: <p>Looking for  a high   throughput  (images/sec)  ResNet50 PyTorch model for distributed training with Horovod on GPUs .  The Horovod example code ( using torchvision ResNet50 , <a href="https://github.com/horovod/horovod/blob/master/examples/pytorch/pytorch_imagenet_resnet50.py" class="inline-onebox" rel="noopener nofollow ugc">horovod/pytorch_imagenet_resnet50.py at master · horovod/horovod · GitHub</a> )   is there, but looks like  it  does not run some functions ( e.g.  cross_entropy , may be optimizer  also ) in GPUs.</p>

username: H-Huang
cooked: <p>I’m not sure how Horovod works but can you modify the example with <code>.to(device)</code> so that the functions that you want to use GPU are using GPU?</p>
<p>You may find more info in the Horovod forums or stack overflow as many of the members here are proficient in <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" rel="noopener nofollow ugc">PyTorch Distributed Training</a> rather than Horovod.</p>

username: whatdhack
cooked: <p>Thanks. Is there a way to  globally move all  functions  that are gpu compatible to gpu ? The piecemeal approach could be error prone. Looks like you are saying there is no PyTorch ResNet50+Horovod  “official” example code ? Would that be the right conclusion to draw ?</p>
Jacobian Based Saliency Map implementation

username: sait
cooked: <p>I am trying to implement <a href="https://arxiv.org/pdf/1511.07528.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1511.07528.pdf</a> . That is Jacobian based saliency map attack for generation of adversarial examples. Is there any reference implementation. If not can somone tell how to compute derivative of logits wrt to inputs.Thanks in advance</p>

username: jarvico
cooked: <p>Hi,</p>
<p>For MNIST data, where input image is 28x28 and you have 10 classes, you may use this code:</p>
<p>def find_Jacobian(my_input, my_nn_model):</p>
<pre><code>inp = my_input.detach().clone()
Jn = torch.zeros((784, 10))  # loop will fill in Jacobian
Jn = Jn.float()

inp.requires_grad_()

preds = my_nn_model(inp)

for i in range(10):
    grd = torch.zeros((1, 10))  # same shape as preds
    grd[0, i] = 1  # column of Jacobian to compute
    preds.backward(gradient=grd, retain_graph=True)
    Jn[:, i] = inp.grad.view(784).float()  # fill in one column of Jacobian
    inp.grad.zero_()  # .backward() accumulates gradients, so reset to zero

return Jn</code></pre>

username: Linkang_Du
cooked: <p>A simple implementation in torch 1.8.1 <a href="https://github.com/link-zju/JSMA-16-SP--implement" rel="noopener nofollow ugc">JSMA</a></p>
Facing issue with forward method while combining two blocks

username: Nakkhatra
cooked: <p>I have two model blocks, one is a pretrained resnet moel (resEncoder) and the other is a custom decoder model that I wrote. Now, to initialize the decoder model, x.size() is taken as a parameter which is called input_size.</p>
<p>Here, x is actually the output that I get from the layer before the avgpool layer from the resnet model. So, in forward pass, I want to add self.decoder and create an object of the decoderModel() class. But, to do that, I need the input the size of that tensor which I get from before the avgpool layer of resnet18. One way is to hardcode it, which I don’t wanna do. Now I can call it like this, but will it work in training if I don’t create self.decoder in <strong>init</strong>()?</p>
<pre><code class="lang-auto">class fullModel(nn.Module):
  def __init__(self, addDecoder: bool):
    super(fullModel, self).__init__()
    self.addDecoder = addDecoder 

    self.resEncoder = pretrainedModelBlock(model_name=params["model_name"], classes=params["num_classes"], addDecoder= self.addDecoder)
    
    # if self.addDecoder:
    #   self.decoder = decoderModel(input_size= beforeAvgPool.size(), depth = 5)

  def forward(self, x):
    if self.addDecoder:
      beforeAvgPool, pool, resnet_features, out = self.resEncoder(x)
      out = decoderModel(input_size= beforeAvgPool.size(), depth = 5)(beforeAvgPool)

    else:
      out = self.resEncoder(x)
    
    return out
</code></pre>
<p>Now it works this way, but since I do not have self.decoder in <strong>init</strong> of fullModel(), will it work if I create a model object from this fullModel class?</p>
<pre><code class="lang-auto">x = torch.randn((64, 3, 32, 32))
fullModel(addDecoder = True)(x).size()
</code></pre>

username: suraj.pt
cooked: <p>Hi! Do you expect the output size of your encoder to change on the fly? If not, you can try passing a random input to the encoder in the constructor itself to get <code>beforeAvgPool.size()</code> and use that to init your decoder. Though I’m really not sure why you are taking this approach <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

username: Nakkhatra
cooked: <p>I can do that actually and that is what I meant by ‘hardcoding it’. Thing is, I want to use the same classes whether I use resnet18 or resnet50 or anything else. So, if I fix the output size there, the code will not be as flexible.</p>
How to calculate the output size after Conv2d in pytorch?

username: XiaXuehai
cooked: <p>I am new about pytorch.<br>
the example like this</p>
<pre><code class="lang-auto">inputs = torch.rand(1,1,10,10)
mod = nn.Conv2d(1,32,3,2,1)
out = mod(inputs)
print(out.shape)
</code></pre>
<p>the output is <code>torch.Size([1, 32, 5, 5])</code><br>
I think  new_width = (old_width+2*padding-kernal_size)/stride +1.<br>
but it cann’t divisible.<br>
So how to calculate it in pytorch?</p>

username: ptrblck
cooked: <p>The complete formula for the output size is given in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d">docs</a>. If it’s not divisible, the output size seems to be rounded down.</p>
<p>EDIT: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d">new link to the Conv2d docs</a>.</p>

username: jw3126
cooked: <p>Is there a built in way to compute the output size of layers, without actually running the layer? I am looking for something like:</p>
<pre><code class="lang-python">output_tensor_size = compute_output_size(layer, input_tensor_size)
</code></pre>

username: ptrblck
cooked: <p>You could take a look at the different approaches in <a href="https://discuss.pytorch.org/t/utility-function-for-calculating-the-shape-of-a-conv-output/11173">this thread</a>.</p>

username: yuezuegu
cooked: <p>Following package calculates the output dimensions of common torch.nn operations:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pypi.org/project/torchshape/">
  <header class="source">
      <img src="https://pypi.org/static/images/favicon.6a76275d.ico" class="site-icon" width="32" height="30">

      <a href="https://pypi.org/project/torchshape/" target="_blank" rel="noopener nofollow ugc">PyPI</a>
  </header>

  <article class="onebox-body">
    <img src="https://pypi.org/static/images/twitter.6fecba6f.jpg" class="thumbnail onebox-avatar" width="300" height="300">

<h3><a href="https://pypi.org/project/torchshape/" target="_blank" rel="noopener nofollow ugc">torchshape</a></h3>

  <p>Calculates the output shape of Pytorch operations</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>You can install it by:</p>
<pre><code class="lang-auto">pip install torchshape
</code></pre>
<p>Disclaimer: I’m the creator of this package.</p>