akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom Criterion (Loss) - Custom Layer

fsh: <p>Hi all,</p>
<p>I want to create a new criterion as a black box (which uses numpy). Since autograd cannot compute the gradient, I need to define both the forward() and backward() functions. The new criterion gets the output of the network; then, together with some other values, it computes both the loss and the gradients to the input. Now,</p>
<ul>
<li>Do I need to inherit from a module in Pytorch?</li>
<li>Do I need to iterate a for-loop to compute the loss for each sample within a batch and then compute the average loss?</li>
</ul>
<p>Any help would be highly appreciated.</p>

G.M: <p>You can define your own autograd function, as described <a href="https://pytorch.org/docs/1.0.1/notes/extending.html#extending-torch-autograd" rel="nofollow noopener">here</a>.</p>

fsh: <p>Thank you so much for the hint. I implemented a function as below. <code>loss_and_gradients</code> is a numpy function which computes both the loss and gradients w.r.t to <em>input</em>. Once the function is called in the forward path, it computes the gradients as well. Do I have to call <code>loss_and_gradients</code> function in the backward path again? I need the gradient only w.r.t to  <em>input</em>, and not the other input values (i.e. imgL, cost, imgD).</p>
<pre><code class="lang-auto">class MyCustomLoss(Function):
    
    @staticmethod
    def forward(self, input, imgL, cost, imgD):
        self.loss, self.gradients = loss_and_gradients(imgL, cost, imgD, input)
        return self.loss
    
    @staticmethod
    def backward(self, grad_output):
       return self.gradients, None, None, None
</code></pre>

G.M: <p>No, u don’t have to call it again, u only have to make sure <code>backward</code> returns the gradients. This being so, u don’t have to save <code>.loss</code> to <code>self</code>. Also, the number of arguments  of <code>backward</code> and its number of returns should be equal to the number of outputs of <code>forward</code>.</p>
Pre-trained model does not train

Dasha_Dasha: <p>Hello!<br>
Is anyone know the reason why pre-trained model starts with good accuracy but after just jumping around this value and does not train.<br>
On the image that I attached, I tried AlexNet with different numbers of batch, epochs, and optimizers.<br>
I also tried other (VGG, ResNet) but results were similar.</p>
<p>The dataset that I use: Scratches on concrete surface (around 2000 images)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b4c0e447ac466850f34ca956946187d074b531ba.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b4c0e447ac466850f34ca956946187d074b531ba" title="ALEXNET"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4c0e447ac466850f34ca956946187d074b531ba_2_544x500.jpeg" alt="ALEXNET" data-base62-sha1="pN1ddR5bkbpmSE7AyoJTHX9ZD6y" width="544" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4c0e447ac466850f34ca956946187d074b531ba_2_544x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4c0e447ac466850f34ca956946187d074b531ba_2_816x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4c0e447ac466850f34ca956946187d074b531ba_2_1088x1000.jpeg 2x" data-dominant-color="F5F5F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ALEXNET</span><span class="informations">1434×1318 333 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

alx: <p>using a pretrained model often means your need a different lr. Have you changed it?</p>

Dasha_Dasha: <p>Nope, I always use lr=0.001, but I also use scheduler</p>
Nested Model Inheritence

Aviv_Shamsian: <p>Hi all,<br>
I have an abstract class which inherits from <code>nn.Module</code> and parent class which inherits from <code>AbstractClass</code> when I try to assign the model to CUDA I don’t get any error but by looking at nvidia-smi the model is clearly not assigned. When I tried to use <code>is_cuda</code> I got an error that my model does not recognize this method. I think the model is not inheriting the <code>nn.Module</code> methods at all.<br>
How can I fix this?</p>
<p>simple code to reproduce the problem:</p>
<pre><code class="lang-auto">    class AbstractModel(nn.Module):
        def __init__(self):
            super().__init__()
            self.max_objects_in_frame = 15
            self.em_in_dim = 5
            self.em_out_dim = 4

    class Linear(AbstractModel):
        def __init__(self, input_f, output_f):
            super().__init__()
            self.input_f = input_f
            self.output_f = output_f
            self.layer = nn.Linear(input_f, output_f)
            
        def forward(self, x):
            return self.layer(x)

if __name__ == '__main__':
    model = Linear(10,20)
    print(model.em_in_dim)
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model = model.to(device)
    print(model.is_cuda)
</code></pre>

ptrblck: <p><code>.is_cuda</code> is a tensor attribute and cannot be used on a module, as a module can hold parameters on different devices.<br>
Instead of calling it on the model, you could check some internal parameters. e.g. <code>model.layer.weight.is_cuda</code>.</p>
Error when try to install torchvision

Cergo: <p>I am trying to install torchvision (Windows) with pip:<br>
<code>pip install torchvision</code><br>
When torch is already installed but I get the following error:</p>
<pre><code class="lang-auto">Collecting torchvision
  Using cached https://files.pythonhosted.org/packages/b7/ff/091b4503d5f228bd1120db784e2c071617211b965a8a78018e75750c7199/torchvision-0.3.0-cp37-cp37m-win_amd64.whl
Requirement already satisfied: numpy in c:\users\cergo\appdata\local\programs\python\python37\lib\site-packages (from torchvision) (1.16.4)
Collecting six (from torchvision)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Requirement already satisfied: pillow&gt;=4.1.1 in c:\users\cergo\appdata\local\programs\python\python37\lib\site-packages (from torchvision) (6.1.0)
Collecting torch&gt;=1.1.0 (from torchvision)

  ERROR: Could not find a version that satisfies the requirement torch&gt;=1.1.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch&gt;=1.1.0 (from torchvision)
</code></pre>

peterjc123: <p>Please check out the new commands to install torchvision 0.3.0 on <a href="https://pytorch.org" rel="nofollow noopener">https://pytorch.org</a>.</p>

Gaussian01: <p>If you use this does it work?</p>
<p>pip install conda<br>
conda install -c pytorch torchvision</p>
When i firstly install pytorch in jupyter notebook ,there is a problem?

cjwang: <p>i  try to run those conmmands in anaconda3 prompt: 1.conda activate pytorch  2.conda install nb_conda<br>
3.jupyter notebook  then my conmmads  run as follows:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88674f09e2733c9b73c2360d150b077dbe8a1687.png" data-download-href="https://discuss.pytorch.org/uploads/default/88674f09e2733c9b73c2360d150b077dbe8a1687" title="B@5XU4M2I8DP8PVP`2UIM65"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88674f09e2733c9b73c2360d150b077dbe8a1687.png" alt="B@5XU4M2I8DP8PVP`2UIM65" data-base62-sha1="jsGgZ2T9gd7wUG4DUlXydFTONQH" width="690" height="204" data-dominant-color="0F0F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">B@5XU4M2I8DP8PVP`2UIM65</span><span class="informations">1066×316 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>This error seems to be unrelated to PyTorch and is apparently a fix for the <code>tornado</code> server inside of Jupyter for Windows users.<br>
I would recommend to try to update <code>asyncio</code> and if that doesn’t help, create an issue on the jupyter repository.</p>
Adding a new hidden layer

Ibrahim_Banat: <p>How can i add more hidden layer to this code, and i added more hidden layer should it increase the accurcy of the model, I’ve got 76%<br>
and can i contact you in anthor place to review the code with me ?</p>
<p>this the code:</p>
<pre><code class="lang-auto">class Model(nn.Module):

    def __init__(self, embedding_size, num_numerical_cols, output_size, layers, p=0.4):
        super().__init__()
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
        self.embedding_dropout = nn.Dropout(p)
        self.batch_norm_num = nn.BatchNorm1d(num_numerical_cols)

        all_layers = []
        num_categorical_cols = sum((nf for ni, nf in embedding_size))
        input_size = num_categorical_cols + num_numerical_cols

        for i in layers:
            all_layers.append(nn.Linear(input_size, i))
            all_layers.append(nn.ReLU(inplace=True))
            
            all_layers.append(nn.BatchNorm1d(i))
            all_layers.append(nn.Dropout(p))
            input_size = i

        all_layers.append(nn.Linear(layers[-1], output_size))

        self.layers = nn.Sequential(*all_layers)

    def forward(self, x_categorical, x_numerical):
        embeddings = []
        for i,e in enumerate(self.all_embeddings):
            embeddings.append(e(x_categorical[:,i]))
        x = torch.cat(embeddings, 1)
        x = self.embedding_dropout(x)

        x_numerical = self.batch_norm_num(x_numerical)
        x = torch.cat([x, x_numerical], 1)
        x = self.layers(x)
        return x
</code></pre>

ptrblck: <p>Adding more layers to your model doesn’t necessarily improve the accuracy so you would need to experiment with your model for your use case.</p>
<p>Based on your code it seems you could simply pass more feature sizes in <code>layers</code> to add more linear layers.</p>
Is torch.nn.Sequential able to take a list as the input?

songyuc: <p>Hi, guys,<br>
I am working on reimplementing the DeepLabV3+ these days,<br>
and I am thinking about whether the torch.nn.Sequential can take a <code>list</code> of modules as the input?<br>
I think this would be an awesome feature!</p>
<p>Any answer or idea will be appreciated!</p>

ptrblck: <p>We had discussions in the past about passing multiple inputs to <code>nn.Sequential</code> e.g. <a href="https://github.com/pytorch/pytorch/pull/9979#">in this PR</a> and <a href="https://github.com/pytorch/pytorch/issues/19808#">here</a>, but in the end the changes were rejected.</p>
<p>You could of course use the posted custom sequential containers for your model, if that makes its definition easier.</p>
Cannot overtfit on a single training sample

farhanrahman: <p>Hello there! For my own project, I have a dataset of RGB (3 channel) images. And the ground truth is 12 channels. In each of these channels there can be values ranging from [0-1]. And the channels are independent of each other, and not one hot encoded of course.</p>
<p>I was trying to train a U-Net on this data. And for experiment I took a couple of samples and trained on one and validated on the other. I was expecting to get close to perfect accuracy on the training sample since the model should overfit after a sufficient number of epochs, if I am not mistaken. However the loss starts to flatten out after a while and its not that close to ground truth either. Note that I am using MSE as the loss function. Is there anything that I might be missing? Or should I increase the complexity of the network, or use a different encoder backbone for the U-Net?</p>
<p>Thanks in advance, looking forward for your opinions and suggestions!</p>

aauker: <p>Just to make sure I understand, you are trying to predict 12 continuous valued masks from an RGB image?  To me that’s a pretty intense objective, and you might just be running up against your irreducible error.  I’d try the whole setup and see if the model is generalizing to validation data first. You might need a deeper U-net or wider decoding layers.</p>

farhanrahman: <p>Yes that is correct. Lets say for 1 channel, each values in that channel would tell a certain property of all the pixels. More specifically if the output is 500x500x12, then the for a pixel there are 12 values that describe a 12 properties for that particular pixel.</p>

aauker: <p>I see, so this objective is requiring a lot out of your network. I suspect a wider decoder will eventually be useful, but I’d not be concerned with that yet.  Try a full scale train/validation and see how it goes.  If you pulled an off-the-shelf U-net model it should mostly work for your application.</p>
Loss collection for outputs on multiple GPUs

martinferianc: <p>Hi,</p>
<p>I have the following problem. I am trying to propagate multiple outputs out of my network which are scalars, for example, latency or memory consumption of respective layers in addition to the output itself. These outputs I would then like to add to the main loss, let’s say cross-entropy.</p>
<p>With a single GPU, I am using a <code>@dataclass</code> to accumulate the respective scalar layer outputs in an accumulator that I then add to the loss, which contents I add to the main loss. However, I do have multiple GPUs that I could utilise for training and I am not sure how to propage the respective scalars and combine them such that I could use <code>.backward()</code>. Any help is much appreciated. Thanks.</p>

ptrblck: <p>If you are using <code>nn.DataParallel</code> the model will be replicated to each GPU and each model will get a chunk of your input batch.<br>
The output will be gathered on the default device, so most likely you wouldn’t have to change anything.</p>
<p>However, I’m not sure about the use case.<br>
How are you calculating the memory consumption and is this operation differentiable?<br>
I assume it’s not differentiable so that your accumulated loss will in fact just be the <code>nn.CrossEntropyLoss</code>.</p>

martinferianc: <p>Thank you for getting back to me. I forgot to mention that the scalars are multiplied by a parameter that I would like to learn (I am experimenting with neural architecture search).</p>
<p>When I did some small-scale experiments, I did not observe any errors so it has to be my implementation that is wrong, nevertheless, thank you for your clarification.</p>