akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Stop updating some weight during the training of an NN model

Sihabul_Islam: <p>During training a model from the below code, I am trying not to update the 1st weight parameter (0.14) of the 2nd layer (linear2). And the other weights will update according to the gradient descent. How can I do this? I am trying this in the following way. But it is updating all weights of the model.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class MyNet(nn.Module):
    def __init__(self):
            super(MyNet, self).__init__()
            self.linear1 = nn.Linear(2, 2,  bias=None)
            self.linear1.weight = torch.nn.Parameter(torch.tensor([[0.11, 0.21], [0.12, 0.08]]))
            self.linear2 = nn.Linear(2, 1,  bias=None)
            self.linear2.weight = torch.nn.Parameter(torch.tensor([[0.14, 0.15]]))

    def forward(self, inputs):
            out = self.linear1(inputs)
            out = self.linear2(out)
            return out

losses = []
loss_function = nn.L1Loss()
model = MyNet()
optimizer = optim.SGD(model.parameters(), lr=0.05)
input = torch.tensor([2.0,3.0])
print('weights before backpropagation = ',   list(model.parameters()))

for epoch in range(1):
        result = model(input )
        loss = loss_function(result , torch.tensor([1.00],dtype=torch.float))
        print('result = ', result)
        print("loss = ",   loss)
        
        with torch.no_grad():
                model.linear2.weight[0, 0] = .14

        model.zero_grad()
        loss.backward()
        print('gradients =', [x.grad.data  for x in model.parameters()] )
        optimizer.step()
        print('weights after backpropagation = ',   list(model.parameters()))</code></pre>
How to free the gpu memory of tensor list obtained by all_gather_object api?

roy_zhang: <p>Following the example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; # Note: Process group initialization omitted on each rank.
&gt;&gt;&gt; import torch.distributed as dist
&gt;&gt;&gt; # Assumes world_size of 3.
&gt;&gt;&gt; gather_objects = ["foo", 12, {1: 2}] # any picklable object
&gt;&gt;&gt; output = [None for _ in gather_objects]
&gt;&gt;&gt; dist.all_gather_object(output, gather_objects[dist.get_rank()])
&gt;&gt;&gt; output
['foo', 12, {1: 2}]
</code></pre>
<p>I used similar way to gather tensors into an output list during the training. These tensors occupied to much gpu memory and made CUDA OOM in the next steps. I tried to use <code>del</code> and <code>torch.cuda.empty_cache()</code>, but the memory is not released. Does it has another way to free the gpu memory?</p>

pbelevich: <p><a class="mention" href="/u/roy_zhang">@roy_zhang</a> could you please show how did you detect that memory was not released?</p>

roy_zhang: <p>I just watch the nvidia-smi. But, this problem is solved, I use all_gather in a complex scenario, the cuda tensor are not actually transfer to the target gpu even the target process could get all tensors, I guess it should be mapping? I am sure that each process creates context in all gpus making the gpu memory increasing. I tried to move tensors on cpu in gather_objects, then it is running…</p>

vince62s: <p>Hi <a class="mention" href="/u/roy_zhang">@roy_zhang</a> ,<br>
What syntax are you using ?<br>
I am facing a similar situation where while gathering different parts of a model state dict, I am getting OOM.<br>
BUT reading the the docs, all_gather needs to have the objects on GPU when using nccl.<br>
My code is : torch.distributed.all_gather_object(full_model, model_state_dict)<br>
so should I just add  .cpu() to full_model ?</p>
Question about the use of ddp_model.no_sync() with torch.utils.checkpoint.checkpoint

wangchengtao: <p>Question about the use of ddp_model.no_sync() with torch.utils.checkpoint.checkpoint</p>
<p>Background: The ddp_model is a TwoLinearLayerMLP</p>
<h2><a name="if-i-use-torchutilscheckpointcheckpoint-with-the-whole-ddp_model-1" class="anchor" href="#if-i-use-torchutilscheckpointcheckpoint-with-the-whole-ddp_model-1"></a>If I use torch.utils.checkpoint.checkpoint with the whole ddp_model：</h2>
<p>Code1</p>
<pre><code class="lang-python">with ddp_model.no_sync():
    for i in range(1, micro_batch_size):
        output = cp.checkpoint(ddp_model, inputs[i])
        loss = loss_fn(output, labels[i])
        loss.backward()
output = cp.checkpoint(ddp_model, inputs[0])
loss = loss_fn(output, labels[0])
loss.backward()
</code></pre>
<p>Code2</p>
<pre><code class="lang-python">outputs = [0] * micro_batch_size
outputs[0] = cp.checkpoint(ddp_model, inputs[0])
with ddp_model.no_sync():
    for i in range(1, micro_batch_size):
        outputs[i] = cp.checkpoint(ddp_model, inputs[i])
    for i in range(1, micro_batch_size):    
        loss = loss_fn(outputs[i], labels[i])
        loss.backward()
loss = loss_fn(outputs[0], labels[0])
loss.backward()
</code></pre>
<p>Both of the above codes can work properly.</p>
<h2><a name="if-i-use-torchutilscheckpointcheckpoint-with-the-two-linear-layers-that-belong-to-the-twolinearlayermlp-like-this-2" class="anchor" href="#if-i-use-torchutilscheckpointcheckpoint-with-the-two-linear-layers-that-belong-to-the-twolinearlayermlp-like-this-2"></a>If I use torch.utils.checkpoint.checkpoint with the two linear layers that belong to the TwoLinearLayerMLP, like this:</h2>
<p>Code3</p>
<pre><code class="lang-python">def forward(self, input):
      a1 = functional.relu(cp.checkpoint(self.linear1, input))
      a2 = cp.checkpoint(self.linear2, a1)
      return input + a2
</code></pre>
<p>This code can still work properly:<br>
Code4</p>
<pre><code class="lang-python">with ddp_model.no_sync():
      for i in range(1, micro_batch_size):
          output = ddp_model(inputs[i])
          loss = loss_fn(output, labels[i])
          loss.backward()
  output = ddp_model(inputs[0])
  loss = loss_fn(output, labels[0])
  loss.backward()
</code></pre>
<p>But this code works wrongly. Gradients across DDP processes can’t be synchronized:<br>
Code5</p>
<pre><code class="lang-python">outputs = [0] * micro_batch_size
outputs[0] = ddp_model(inputs[0])
with ddp_model.no_sync():
    for i in range(1, micro_batch_size):
        outputs[i] = ddp_model(inputs[i])
    for i in range(1, micro_batch_size):
        loss = loss_fn(outputs[i], labels[i])
        loss.backward()
loss = loss_fn(outputs[0], labels[0])
loss.backward()
</code></pre>
<h1><a name="my-questions-3" class="anchor" href="#my-questions-3"></a>My questions:</h1>
<ol>
<li>I want to let the Code5 work properly. How can I solve this problem? And there are any matters needing attention when I use ddp_model.no_sync() with torch.utils.checkpoint.checkpoint.</li>
<li>What’s the difference between Code1 and Code2? I want to use ddp_model.no_sync() like Code2.</li>
</ol>
Can __torch_function__ only intercept specific API?

dilililiwhy: <p>I defined a custom DeviceContext (like the one in torch/utils/_device.py) based on <strong>torch_function</strong>. I find that the context will intercept calling of all torch apis, which decrease the performance of training especially in model with multiple small ops like InceptionV3.</p>
<p>Is there a way to limit the interception？</p>
Torch.compile does not work with importlib

coincheung: <p>The directory structure is like this:</p>
<pre><code class="lang-auto">.
├── main.py
└── models
    └── model.py

</code></pre>
<p>The content of <code>main.py</code> is:</p>
<pre><code class="lang-python">import torch
import importlib


def get_model():

    pth = './models/model.py' # this is assigned as input argument
    spec = importlib.util.spec_from_file_location('model', pth)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)

    class Child(mod.Block):

        def __init__(self):
            super(Child, self).__init__()

        @torch.compile
        def forward(self, x):
            outputs = super().forward(x)
            return outputs

    return Child

model = get_model()()
model = torch.compile(model)

optim = torch.optim.SGD(model.parameters(), lr=1e-3)

for i in range(10):
    inten = torch.randn(2, 32, 128, 128)
    outen = torch.randn(2, 32, 128, 128)

    optim.zero_grad()
    out = model(inten)
    loss = (out - outen).mean()
    loss.backward()
    optim.step()
    print(loss.item())

</code></pre>
<p>And the content of <code>models/model.py</code> is:</p>
<pre><code class="lang-python">import torch

class Block(torch.nn.Module):

    def __init__(self):
        super().__init__()
        self.conv1 = torch.nn.Conv2d(32, 32, 3, 1, 1)
        self.conv2 = torch.nn.Conv2d(32, 32, 3, 1, 1)

    def forward(self, x):
        return self.conv2(self.conv1(x))
</code></pre>
<p>The error message is:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/projects/useless/main.py", line 55, in &lt;module&gt;
    out = model(inten)
  File "/home/.conda/envs/py39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/.conda/envs/py39/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
    return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)
  File "/home/.conda/envs/py39/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
    return fn(*args, **kwargs)
TypeError: forward() missing 1 required positional argument: 'x'
</code></pre>
<p>I am working on ubuntu system, with pytorch 2.0.1 installed from conda, the cuda versio is 11.8.</p>
<p>If I comment out the line of <code>@torch.compile</code>, the code can work as expected. Would you tell me how could I make this work?</p>
Torchvision's ViT question about number of classes for fine-tuning

Vitalii_Kostenko: <p>Is this the right way I change num_classes in vit?</p>
<pre><code class="lang-auto">self.model = models.vit_b_16(pretrained=True)  # Load with 1000 classes first
        
if hasattr(self.model.heads, "head") and isinstance(self.model.heads.head, nn.Linear):
    self.model.heads.head = nn.Linear(self.model.heads.head.in_features, num_classes)
</code></pre>

tom: <p>The <code>pretrained=True</code> wants to be replaced by an appropriate <code>weights=</code> parameter eventually, but other than that, it looks reasonable. I might change the <code>if</code> into an <code>assert</code> because, quite likely, you don’t want to continue if the <code>head</code> isn’t a linear that you can replace with your own.</p>
<p>Best regards</p>
<p>Thomas</p>
Best way to track potential reasons for crashes during training

LMeyling: <p>Hi,</p>
<p>I am training a rather large model with pytorch (Estimated forward/backward around 10 GB), and I am facing the problem that the training sometimes crashes at around ~100/800Epochs (computer shuts down). I am aware there can be multiple reasons like the GPU overheating, CPU getting trashed etc. So i was wondering what the best metrics would be to track down the issue, and if possible how to track these metrics efficently e.g. what liabries/commands to use.</p>
<p>For instance tracking the cpu usage is rather hard, since it is mostly used when loading images with the dataloader, so i am not sure where to call it. Looking forward for ideas!</p>
<p>Kind Regards</p>

ptrblck: <p>I would first check logs, e.g. in <code>dmesg</code>, to see if the system reported any errors or why it’s shutting down. It would also be interesting to learn more about this “shut down” behavior. Is the system reporting any errors before shutting itself down or is it just crashing and turns directly off? The latter case is often caused by an underpowered or defect PSU, so you might want to swap it (even temporarily for debugging) and see if this helps.</p>
Convolve over image with small additional tensor

mlterpstra-umc: <p>Hi all,</p>
<p>I have a question regarding a specific convolution operation I want to perform. Suppose I have some image tensor with shape (1, 12, 256, 256) that I want to convolve with kernel with size 5x5. I can simply do this in PyTorch by doing</p>
<pre><code class="lang-auto">import torch
img = torch.randn(1, 12, 256, 256)
conv = torch.nn.Conv2d(in_channels=12, out_channels=32, kernel_size=5, padding=2)
out = conv(img)
</code></pre>
<p>Now suppose I have an additional image with the same number of channels and an image size equal to the kernel, i.e., (1, 12, 5, 5).<br>
Now I want the convolution operation to be applied to both images. If they were the same size I could concatenate along the channel dimension into a 24 channel tensor, but  as they have different spatial dimensions this operation is not possible. Moreover, padding is not possible because I want the small additional image to be centered on the convolution kernel.</p>
<p>Is there any way to “append” this tensor to every convolution operation?</p>
<p>Thanks.</p>

ptrblck: <aside class="quote no-group" data-username="mlterpstra-umc" data-post="1" data-topic="187672">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mlterpstra-umc/48/8710_2.png" class="avatar"> mlterpstra-umc:</div>
<blockquote>
<p>Moreover, padding is not possible because I want the small additional image to be centered on the convolution kernel.</p>
</blockquote>
</aside>
<p>This is possible using <code>F.pad</code> and sounds like a valid approach for your use case:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1, 1, 5, 5)
x = F.pad(x, (9, 10, 9, 10))
print(x.shape)
# torch.Size([1, 1, 24, 24])
</code></pre>
Maximum number of channels?

ljeonjko: <p>Hi all!</p>
<p>I trained two GAN models (the actual python scripts are copies of each other) - one using 7 and the other 9-channel images (stored as .tiff). When testing, the 7-channeled model works as expected, but the 9-channeled one returns all NaN values, even though during training the loss values were actual numbers. Is there a maximum number of channels PyTorch can handle?</p>
<p>Thanks!</p>

ptrblck: <p>No, PyTorch doesn’t set the max. number of channels to 9 and you can easily test it with a few unit tests.</p>
"Log of sum over Gaussian likelihood" loss function + DNN

AhmadHalimi: <p>Hi. I have just started working with Pytorch and ML stuff. I have a loss function as below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac0dfa5522a7336aab26b802cb537990c208c2a6.png" data-download-href="https://discuss.pytorch.org/uploads/default/ac0dfa5522a7336aab26b802cb537990c208c2a6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac0dfa5522a7336aab26b802cb537990c208c2a6_2_690x92.png" alt="image" data-base62-sha1="oy44u6rfi4JVxslM0XShRI7Azau" width="690" height="92" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac0dfa5522a7336aab26b802cb537990c208c2a6_2_690x92.png, https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac0dfa5522a7336aab26b802cb537990c208c2a6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac0dfa5522a7336aab26b802cb537990c208c2a6.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1024×137 15.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I guess I can easily use the nn.GaussianNLLLoss for the first term but do you know how can I implement the second term which is the logarithm of the sum over Gaussian likelihood?</p>
How to merge variables without using additional space?

BearBiscuit05: <p>I currently have two feature files, each with a size of 3GB, and an 8GB machine. Now, I want to load both files into memory one after the other and merge them into a single variable. I would like to know how to do this without exceeding the available memory. I tried using <code>torch.cat</code> , but it seems that the <code>cat</code> function might allocate additional memory for merging, which leads to memory overflow.</p>
Modifying the data type of model parameters or buffers

iammano: <p>Hi,<br>
I’m trying to give a model as input to Pytorch-DDP for distributed training. Here my script throws an error of invalid scalar type. when I look into the model(GPT-NEO 1.3B) buffers it is boolean and loop through the buffer and modify the data type to int8. Now DDP is working fine.</p>
<p>problem:<br>
After training, I’m not getting any response from the model. The model is not generating any inference/prediction here. I don’t know whether what am I doing is correct or wrong.</p>
<p>Kindly clarify my understanding with your reply, Thanks.</p>
<p>Note: For synchronisation of weights among the nodes DDP need a datatype of float32 or int. Boolean is not accepted.</p>
Computing gradients on sparse matrix eigensolves

Judd: <p>Hi, does anyone have any suggestions for computing gradients on an eigensolve of a sparse matrix.</p>
<p>The lobpcg function does not support this, the documentation explicitly mentions that the backward method is not supported for sparse inputs (<a href="https://pytorch.org/docs/stable/generated/torch.lobpcg.html" class="inline-onebox" rel="noopener nofollow ugc">torch.lobpcg — PyTorch 2.0 documentation</a>).</p>
<p>Thanks!</p>
Error adding model with multiple inputs to tensorboard

Goldname: <p>I am trying to take a look at this huggingface model in their documentation using tensorboard:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/docs/transformers/v4.32.1/en/model_doc/time_series_transformer#transformers.TimeSeriesTransformerForPrediction.forward.example">
  <header class="source">

      <a href="https://huggingface.co/docs/transformers/v4.32.1/en/model_doc/time_series_transformer#transformers.TimeSeriesTransformerForPrediction.forward.example" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://huggingface.co/front/thumbnails/docs/transformers.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/docs/transformers/v4.32.1/en/model_doc/time_series_transformer#transformers.TimeSeriesTransformerForPrediction.forward.example" target="_blank" rel="noopener nofollow ugc">Time Series Transformer</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>As you can see, the model takes in a lot of inputs:</p>
<p>outputs = model( … past_values=batch[“past_values”], … past_time_features=batch[“past_time_features”], … past_observed_mask=batch[“past_observed_mask”], … static_categorical_features=batch[“static_categorical_features”], … static_real_features=batch[“static_real_features”], … future_values=batch[“future_values”], … future_time_features=batch[“future_time_features”])</p>
<p>So I try to add a model to tensorboard like this:</p>
<p>writer.add_graph(model, (batch[“past_values”],batch[“past_time_features”],<br>
batch[“past_observed_mask”],<br>
batch[“static_categorical_features”],<br>
batch[“static_real_features”],<br>
batch[“future_values”],<br>
batch[“future_time_features”],<br>
))</p>
<p>But I get this error: :Dictionary inputs to traced functions must have consistent type. Found Tensor and Tuple[Tensor, Tensor, Tensor]</p>
<p>Any ideas on the proper syntax?</p>
Is there a bitcount operation in pytorch

ynjiun_wang: <p>I try to write a simple hamming distance function in pytorch:</p>
<pre><code class="lang-auto">xorimg = torch.bitwise_xor(img1,img2)
for i in range(bitlen) : 
        hdist = hdist + (xorimg &amp; 1)
        xorimg = xorimg &gt;&gt; 1
</code></pre>
<p>I wonder if there is a simple bitcount operation to count the 1s bits to get rid of the for loop such as:</p>
<pre><code class="lang-auto">xorimg = torch.bitwise_xor(img1,img2)
hdist = torch.bitcount(xorimg)
</code></pre>
<p>or any other equivalent way to get rid of time consuming for-loop?</p>
<p>or pytorch support hamming distance directly such as:</p>
<blockquote>
<p>hdist = torch.hamming(img1,img2)</p>
</blockquote>
<p>that would be even better.</p>
<p>Thanks for your help.</p>

ynjiun_wang: <p>See answer <a href="https://stackoverflow.com/a/77019942/17610195" rel="noopener nofollow ugc">here</a></p>
<p>and <a href="https://davidstutz.de/implementing-custom-pytorch-tensor-operations-in-c-and-cuda/" rel="noopener nofollow ugc">here</a></p>
Training multiple models simutaneously

Kai_Lan: <p>I have RTX 6000 with about 48 GB. Right now I am training three models each with a separate python script. So they are three processes, each taking 15 GB.</p>
<p>My question is if training all three models simultaneously would be slower than if training one at a time? I have enough memory for all three, but maybe they are competing for resources and causing slow-down?</p>

ptrblck: <p>Yes, each process will compete for memory (which seems to be fine) and compute resources. Assuming each process is not fully saturating the GPU’s compute resources you would see a speedup, but it depends on the overall setup and how high the GPU utilization of each process is.</p>
How to reduce the loss any further

Jazzy66: <p>Hi, I am using Google’s <a href="https://github.com/lucidrains/lion-pytorch" rel="noopener nofollow ugc">Lion</a> to minimize a HuberLoss in my model. The learning rate I use is 6e-4, weight decay is 1e-2, β1 and β2 are 0.9 and 0.99 as the optimizer recommended.</p>
<p>I performed a 10000 epoch run with batch size 60, iteration early stopped at epoch 2903 under the loss criterion 3e-5. As far as I know, the mean loss has never been less than 1e-5.</p>
<p>So my question is this situation could get any better? Any advice or experience would be saving my life.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0ff44544ffa2e6634e4dec9452b41fa7f97b0a1f.png" data-download-href="https://discuss.pytorch.org/uploads/default/0ff44544ffa2e6634e4dec9452b41fa7f97b0a1f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0ff44544ffa2e6634e4dec9452b41fa7f97b0a1f_2_690x394.png" alt="image" data-base62-sha1="2h8vA10Fi3PEbFAx9A12bV3eLcr" width="690" height="394" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0ff44544ffa2e6634e4dec9452b41fa7f97b0a1f_2_690x394.png, https://discuss.pytorch.org/uploads/default/original/3X/0/f/0ff44544ffa2e6634e4dec9452b41fa7f97b0a1f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/f/0ff44544ffa2e6634e4dec9452b41fa7f97b0a1f.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">871×498 28.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/caa8a0941cf6260a9b64024b57f90e59586f1afb.png" data-download-href="https://discuss.pytorch.org/uploads/default/caa8a0941cf6260a9b64024b57f90e59586f1afb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/caa8a0941cf6260a9b64024b57f90e59586f1afb_2_690x415.png" alt="image" data-base62-sha1="sUNJkZTWf5ioerTSchzCvAicRBN" width="690" height="415" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/caa8a0941cf6260a9b64024b57f90e59586f1afb_2_690x415.png, https://discuss.pytorch.org/uploads/default/original/3X/c/a/caa8a0941cf6260a9b64024b57f90e59586f1afb.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/a/caa8a0941cf6260a9b64024b57f90e59586f1afb.png 2x" data-dominant-color="E3EDF4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">884×532 29.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Either "backward through graph a second time" error or out of memory error if i set retain_graph=True

emei: <p>I get an error ( <a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time</a>) which requires me to set retain_graph=True. However if i do so i get an out of memory error. Is there someway to release the memory after the loss.backward(). Or some other way to work around the error other than retain_graph=True.</p>
<p>This is representattive of what I am trying to do :</p>
<pre><code class="lang-auto">    latents = torch.randn(
        (batch_size, 4, height // 8, width // 8),
        generator=generator,
    )
    latents = latents.to(torch_device, dtype=torch.float16)

    for j in range(num_iter):
        latents_new = return_unet(latents, text_embeddings)
        loss, model_input = return_model(latents_new)
        loss.backward()

        img2 = (model_input['pixel_values']) - lr_rate * (model_input['pixel_values'].grad)
        m = torch.nn.Upsample(scale_factor=3.5, mode='nearest')
        loss2 = sum(sum(sum(sum(img2 - m(latents_new)))))
        loss2.backward(retain_graph=True)

        text_embeddings = text_embeddings - lr_rate * text_embeddings.grad
</code></pre>

ptrblck: <p>Based on the code it seems <code>loss.backward()</code> deletes the intermediates of <code>return_unet</code> and <code>return_model</code>, which will then cause the issue in <code>loss2.backward()</code> since it uses <code>model_input</code> which was created by using both models.<br>
You could <code>sum</code> the losses together and call <code>.backward()</code> on the sum or you could call <code>loss.backward(retain_graph=True)</code> and afterwards <code>loss.backward()</code>, but I would probably prefer the former approach.</p>

emei: <p>The reason I dont sum is that i need to get the gradients on model_input using loss.backward() first and then compute loss2 for gradients on text_embeddings. And strangely i get an error if i use retain_graph=True on loss.backward and not on loss2.backward. But I dont get an error if i use retain_graph=True on loss2.backward as in the code snippet above.</p>
How improve accuracy on this CNN image classifier?

seberino: <p>I implemented the CNN here:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://blog.paperspace.com/writing-cnns-from-scratch-in-pytorch/">
  <header class="source">
      <img src="https://blog.paperspace.com/content/images/size/w256h256/2018/07/Paperspace-Circle.png" class="site-icon" width="256" height="256">

      <a href="https://blog.paperspace.com/writing-cnns-from-scratch-in-pytorch/" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 08 September 2021">Paperspace Blog – 8 Sep 21</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:670/400;"><img src="https://blog.paperspace.com/content/images/2021/07/cnns-pytorch-cifar10.png" class="thumbnail" width="670" height="400"></div>

<h3><a href="https://blog.paperspace.com/writing-cnns-from-scratch-in-pytorch/" target="_blank" rel="noopener nofollow ugc">Writing CNNs from Scratch in PyTorch</a></h3>

  <p>One of the best ways to learn about convolutional neural networks (CNNs) is to write one from scratch! In this post we look to use PyTorch and the CIFAR-10 dataset to create a new neural network.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It creates a classifier for CIFAR-10 dataset.</p>
<p>The accuracy on training data is about 87%.<br>
The accuracy on testing data is about 68%.</p>
<p>What can I do to try to increase the accuracy? Should I focus on<br>
making the CNN more complex?</p>
<p>Here is the current CNN:</p>
<pre><code class="lang-auto">N_CLASSES  = 10
LEARN_RATE = 0.001
MOMENTUM   = 0.9
N_EPOCHS   = 20
BATCH_SIZE = 64
ERROR      = torch.nn.CrossEntropyLoss()

class CNN(torch.nn.Module):
        """
        Defines a convolutional neural network.
        """

        def __init__(self, N_CLASSES):
                super(CNN, self).__init__()
                self.conv_1 = torch.nn.Conv2d( 3, 32, 3)
                self.conv_2 = torch.nn.Conv2d(32, 32, 3)
                self.pool_1 = torch.nn.MaxPool2d(2, stride = 2)
                self.conv_3 = torch.nn.Conv2d(32, 64, 3)
                self.conv_4 = torch.nn.Conv2d(64, 64, 3)
                self.pool_2 = torch.nn.MaxPool2d(2, stride = 2)
                self.line_1 = torch.nn.Linear(1600, 128)
                self.relu_1 = torch.nn.ReLU()
                self.line_2 = torch.nn.Linear(128, N_CLASSES)

        def forward(self, inputs):
                results = inputs
                for f in [self.conv_1,
                          self.conv_2,
                          self.pool_1,
                          self.conv_3,
                          self.conv_4,
                          self.pool_2]:
                        results = f(results)
                results = results.reshape(results.size(0), -1)
                for f in [self.line_1, self.relu_1, self.line_2]:
                        results = f(results)

                return results

</code></pre>
Why is cpu faster the gpu in this case

niydu: <p>I tested this code both cpu and gpu mode<br>
For gpu mode I set device=0, for cpu I set device=-1</p>
<p>Why is gpu mode slower by 1 second<br>
Am I doing something incorrect, I thought gpu was meant to be faster</p>
<pre><code class="lang-auto">from transformers import pipeline

model="facebook/bart-large-mnli"

classifier = pipeline(
                      task="zero-shot-classification",
                      model=model,
                      device=0
                    )

text_piece = "I went to the beach"

candidate_labels=["games", "holiday", "accident"]

result = classifier(text_piece, candidate_labels)

print(result)
</code></pre>
<hr>
<pre><code class="lang-auto">time python3 on_gpu.py
{'sequence': 'I went to the beach', 'labels': ['holiday', 'accident', 'games'], 'scores': [0.980344831943512, 0.013516747392714024, 0.0061384160071611404]}
real    0m6.021s
user    0m5.915s
sys     0m2.293s
</code></pre>
<hr>
<pre><code class="lang-auto">time python3 on_cpu.py
{'sequence': 'I went to the beach', 'labels': ['holiday', 'accident', 'games'], 'scores': [0.9803448915481567, 0.01351669616997242, 0.006138448603451252]}
real    0m4.948s
user    0m6.966s
sys     0m1.570s
</code></pre>

ptrblck: <p>You could properly profile the actual workload and get rid of the initialization and warmup time.</p>
What is the difference __init__() and forward() in a network model?

shrbrh: <p>I have the following code for a neural network. I am confused about what is the difference between the use of <strong>init</strong>() and forward() methods. Does the <strong>init</strong>() method behave as the constructor? If so, what is the significance of the forward() method? Is it necessary to use both while creating the network?</p>
<pre><code class="lang-auto">class MyNeuralNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MyNeuralNetwork, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, output_size)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        out = self.sigmoid(out)
        return out
</code></pre>

knoriy: <p><code>__init__</code> is a constructor method used to initialize the parameters of the network. It is executed when an object of the class is created. For example, in PyTorch, this method is used to define the layers of the network, such as convolutional layers, linear layers, activation functions, etc.</p>
<p><code>forward</code> is the method that defines the forward pass of the neural network. This method takes the input data and passes it through the layers of the network to produce the output. This method is executed whenever the model is called to make a prediction or to compute the loss during training.</p>
<p>In other words, <code>__init__</code> sets up the network’s structure by defining the layers, while <code>forward</code> specifies how the data flows through the network. Both methods are required to create a neural network in PyTorch and serve different purposes.</p>

Biscuit: <p><a class="mention" href="/u/knoriy">@knoriy</a></p>
<p>If I need to create a tensor based on the arguments in the constructor, should I create this tensor in the constructor or in <code>def forward()</code>? This tensor won’t change with the input data for <code>forward()</code>. Is it better to create it in the constructor because we don’t need to create it again and again for each batch? Thanks!</p>

knoriy: <p>Yes, if you need to create a tensor, based on arguments that do not change during the forward pass and remain constant throughout the lifetime of the module, it is generally better to create this tensor in the constructor (<code>__init__</code> ) rather than in the <code>forward</code> method. This is because the constructor is called only once when you initialize the model, whereas the <code>forward</code> method is called for each forward pass, which can be for every batch of data during training or inference.</p>