akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
shape '[-1, 12544]' is invalid for input of size 115200

Jai_chand_Patel: <p>class MyModel(nn.Module):<br>
def <strong>init</strong>(self, num_classes: int = 50, dropout: float = 0.7) → None:</p>
<pre><code>    super().__init__()

    self.conv1 = nn.Conv2d(3,16,3,1,1)
    self.conv2 = nn.Conv2d(16,32,3,1,1)
    self.conv3 = nn.Conv2d(32,64,3,1,1)
    self.conv4 = nn.Conv2d(64,128,3,1,1)
    self.conv5 = nn.Conv2d(128,256,3,1,1)
    self.dropout = nn.Dropout(0.4)


    self.pool = nn.MaxPool2d(2, 2)

  
    self.fc1 = nn.Linear(256*7*7,512)
    self.fc2 = nn.Linear(512, 50)
    
    self.dropout = nn.Dropout(0.4)

def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
 
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    x = self.pool(F.relu(self.conv4(x)))
    x = self.pool(F.relu(self.conv5(x)))


    # Flatten image into vector, pass to FC layers
    # print(x.shape)# [32, 64, 28, 28]
    x = x.view(-1, 256*7*7)


    x = self.dropout(x)
    x = self.fc1(x)
    x = F.relu(x)
    x = self.dropout(x)
    x = self.fc2(x)
    return x
</code></pre>

ptrblck: <p>Your flattening operation is wrong:</p>
<pre><code class="lang-python">    # Flatten image into vector, pass to FC layers
    # print(x.shape)# [32, 64, 28, 28]
    x = x.view(-1, 256*7*7)
</code></pre>
<p>since it cannot flatten an input with <code>115200</code> elements to <code>[-1, 256*7*7]</code>.<br>
But even if the input would have the printed shape of <code>[32, 64, 28, 28]]</code>, it would be most likely wrong since you are increasing the batch size:</p>
<pre><code class="lang-python">x = torch.randn(32, 64, 28, 28)
x.view(-1, 256*7*7).shape
# torch.Size([128, 12544])
</code></pre>
<p>Use <code>x = x.view(x.size(0), -1)</code> to keep the batch size equal and to flatten the other dimensions before passing it to the linear layer.<br>
If you are then running into a shape mismatch, make sure the <code>in_features</code> of <code>self.fc1</code> match the flatten activation feature size.</p>
How to do distributed training when each data loader may have a different # of samples?

Vedant_Roy: <p>Let’s say I have a dataset with 800 samples. I train on 8 GPUs. This launches 8 training processes, each of which consumes a shard of 100 samples.</p>
<p>However, there’s an issue where a sample has, let’s say, a 10% chance of being faulty.<br>
This means some dataloaders, purely by luck, will finish faster than other dataloaders.</p>
<p>Now, I’m not 100% sure, but I <em>think</em> this is causing the following deadlock: <a href="https://github.com/pytorch/pytorch/issues/85775" class="inline-onebox" rel="noopener nofollow ugc">Timed out receiving the shared seed from the distribtued store on Rank 2 · Issue #85775 · pytorch/pytorch · GitHub</a>.</p>
<p>Basically, one data loader is finishing faster than the others. This is causing some sort of deadlock related to shared state.</p>
<p>Does anyone have experience training with differently sized dataloaders?</p>