akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cuda Out of memory error on Squad

Shivam_Marathe: <p>Hello all,<br>
I am fine tuning T5 Conditional Generation model with squad dataset on kaggle with gpu.<br>
But i am getting this cuda out of memory error for the following code</p>
<pre><code class="lang-auto">class SquadQAModel(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = Model


    def forward(self, input_ids, attention_mask, labels=None):
        output = self.model(
            input_ids, 
            attention_mask=attention_mask,
            labels=labels)

        return output.loss, output.logits

    def training_step(self, batch, batch_idx):
        input_ids = batch['input_ids']
        attention_mask=batch['attention_mask']
        labels = batch['labels']
        loss, outputs = self.forward(input_ids, attention_mask, labels)
        self.log("train_loss", loss, prog_bar=True, logger=True)
        return {"loss": loss, "predictions":outputs, "labels": labels}

    def validation_step(self, batch, batch_idx):
        input_ids = batch['input_ids']
        attention_mask=batch['attention_mask']
        labels = batch['labels']
        loss, outputs = self.forward(input_ids, attention_mask, labels)
        self.log("val_loss", loss, prog_bar=True, logger=True)
        return loss

    def test_step(self, batch, batch_idx):
        input_ids = batch['input_ids']
        attention_mask=batch['attention_mask']
        labels = batch['labels']
        loss, outputs = self.forward(input_ids, attention_mask, labels)
        self.log("test_loss", loss, prog_bar=True, logger=True)
        return loss

    def configure_optimizers(self):
        optimizer = AdamW(self.model.parameters(), lr=0.0001)
        return optimizer
    
    def training_epoch_end(self, outputs):
        print("Epoch {} =======&gt;&gt;&gt;&gt; {}".format(self.trainer.current_epoch, self.trainer.progress_bar_dict  ))

    def validation_epoch_end(self, outputs):
        print('EPOCH {} ======={}&gt;&gt;&gt;&gt;'.format(self.trainer.current_epoch, self.trainer.callback_metrics))

</code></pre>
<p>can anyone please see into it ? Why the 16 GB gpu keeps feeling after loading every thing on cuda device</p>

ptrblck: <p>The memory footprint of your training seems to be too high for 16GB so you could reduce the batch size, use a smaller model, or use e.g. <code>torch.utils.checkpoint</code> to trade compute for memory.</p>
RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[64, 1, 32, 32] to have 3 channels, but got 1 channels instead

CasellaJr: <p>I am working on SVHN dataset, and I got this error, only during training phase. During the instantiation of the model, it works.</p>
<pre><code class="lang-auto">RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[64, 1, 32, 32] to have 3 channels, but got 1 channels instead
</code></pre>
<p>To be sure of having 3 channels, I wrote that Grayscale(3) transformation.</p>
<pre><code class="lang-auto"># Compose transformations
data_transform = transforms.Compose([
  transforms.Resize((32,32)),
  transforms.Grayscale(num_output_channels=3),
  transforms.RandomHorizontalFlip(),
  transforms.ToTensor(),
])

# Compose transformations
test_transform = transforms.Compose([
  transforms.Resize((32,32)),
  transforms.Grayscale(num_output_channels=3),
  transforms.ToTensor(),
])

# Load MNIST dataset with transforms
train_set_svhn = torchvision.datasets.SVHN(root=base_dir, split='train', download=True, transform=data_transform, target_transform=None)
test_set_svhn = torchvision.datasets.SVHN(root=base_dir, split='test', download=True, transform=test_transform)
</code></pre>
<pre><code class="lang-auto">class VGG16(nn.Module):

    def __init__(self, num_classes):
        super(VGG16, self).__init__()

        # calculate same padding:
        # (w - k + 2*p)/s + 1 = o
        # =&gt; p = (s(o-1) - w + k)/2

        self.block_1 = nn.Sequential(
            nn.Conv2d(in_channels=3,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      # (1(32-1)- 32 + 3)/2 = 1
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(in_channels=64,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
           .....
           .....
           .....
        )
</code></pre>
<p>I have no idea where that 1 comes from</p>
<p>Moreover, this is the shape of the output of the model without classifier (fully connected layers):</p>
<pre><code class="lang-auto">output = model1(test_x)
output.shape
</code></pre>
<pre><code class="lang-auto">torch.Size([1, 256, 4, 4])
</code></pre>
<p>And indeed I will pass 256x4x4 as input to the first FC.</p>

ptrblck: <p>The error message is created by a conv layer with 64 kernels expecting 3 input channels and a kernel size of 3, which maps to your first conv layer in <code>block1</code>.<br>
Could add an <code>assert</code> statement checking if all inputs have 3 channels?</p>

CasellaJr: <p>I think the error was due to previous instantiation of the model, for another dataset. I restarted the kernel, and I solved. Thanks</p>
Is it possible to no_grad() a downstream part of a network while still allowing the gradients to flow to the upstream part?

Sam_Lerman: <p>Say I have two DNN modules <code>F</code> and <code>G</code>.</p>
<p><code>out = F(G(x_1), x_2)</code></p>
<p>And a loss function like <code>loss(out, label)</code>.</p>
<p>I want to disable gradients for <code>F</code> during one particular update (though not necessarily in others).</p>
<p>I tried this during that update:</p>
<pre><code class="lang-auto">g = G(x_1)
with torch.no_grad():
   out = F(g, x_2)
</code></pre>
<p>But (correct me if I’m wrong), it seems gradients don’t propagate back to <code>G</code> in this case. I need gradients to still propagate back to <code>G</code>.</p>

ptrblck: <p>Your code won’t work and will raise:</p>
<pre><code class="lang-python">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>if you try to call <code>backward()</code> on <code>out</code> or an output of any successive operation on it, since it was calculated in the <code>no_grad</code> block:</p>
<pre><code class="lang-python">G = nn.Linear(1, 1)
F = nn.Linear(1, 1)
x = torch.randn(1, 1)

g = G(x)
with torch.no_grad():
   out = F(g)

out.backward()
&gt; RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

print(out.grad_fn)
&gt; None
</code></pre>
<p>To disable the gradient calculation in <code>F</code> you could set the <code>requires_grad</code> attributes of all parameters of <code>F</code> to <code>False</code>:</p>
<pre><code class="lang-python">G = nn.Linear(1, 1)
F = nn.Linear(1, 1)
x = torch.randn(1, 1)

for param in F.parameters():
    param.requires_grad = False

g = G(x)
out = F(g)

out.backward()
print(out.grad_fn)
&gt; &lt;AddmmBackward object at &gt;
print(G.weight.grad)
&gt; tensor([[0.2877]])
print(F.weight.grad)
&gt; None
</code></pre>
Is it possible to treat the first 2 dims as batch dims without changing view or reshaping?

Sam_Lerman: <p>I use <code>torch.expand</code> on a big tensor:</p>
<p><code>tensor = tensor.unsqueeze(1).expand(-1, 1000, -1)</code></p>
<p>and then I would normally change the view so that the first 2 dim becomes one <em>batch</em> dim, <em>e.g.</em>,</p>
<p><code>new_tensor = tensor.view(tensor.shape[0] * 1000, -1)</code>,</p>
<p>but I can’t do this because <code>torch.expand</code> requires contiguous shaping.</p>
<p>However, using <code>torch.reshape</code> throws me a CUDA out-of-memory error on account of the new contiguous copy! Is there any way I can batch process <code>tensor</code> without creating a memory-expensive copy?</p>
<p>My goal is to process <code>new_tensor</code> in a module with batch processing.</p>

ptrblck: <p>If depends on the used module if contiguous tensors are needed and depending on the used operation you could check, if a slower but more memory-efficient alternative could be used.</p>