akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: non-empty 3D or 4D input tensor expected but got ndim: 4

charmander1: <p>When I started feeding data through my Pytorch net, I got this error:  <code>RuntimeError: non-empty 3D or 4D input tensor expected but got ndim: 4</code> .</p>
<p>This is my net:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 5, 1)
        self.conv2 = nn.Conv2d(32, 64, 5, 1)
        self.conv3 = nn.Conv2d(64, 128, 5, 1)
        self.fc1 = nn.Linear(3*3*3*128, 512)
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, 16)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv3(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 3*3*3*128)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.log_softmax(x, dim=1)
</code></pre>
<p>Complete message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "model.py", line 145, in &lt;module&gt;
    outputs = net(batch_X)
  File "C:\Users\-\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "model.py", line 97, in forward
    x = F.max_pool2d(x, 2, 2)
  File "C:\Users\-\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\_jit_internal.py", line 181, in fn
    return if_false(*args, **kwargs)
  File "C:\Users\-\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\functional.py", line 487, in _max_pool2d
    return torch.max_pool2d(
RuntimeError: non-empty 3D or 4D input tensor expected but got ndim: 4
</code></pre>
<p>Would anybody know what I can do to fix it?</p>

ptrblck: <p>This error would be raised, if you pass an empty tensor to the pooling layer as shown here:</p>
<pre><code class="lang-python">pool = nn.MaxPool2d(2)
x = torch.randn(1, 3, 24, 24)
out = pool(x) # works

y = torch.tensor([[[[]]]])
out = pool(y)
&gt; RuntimeError: non-empty 3D or 4D input tensor expected but got ndim: 4
</code></pre>
<p>Could you check the shape of each tensor you are passing to the pooling layers?</p>
How can I get the data from dataloader so I do clustering?

Mah: <p>I am beginner. I want to do a clustering my image dataset. How can I get the data from dataloader so I do it?</p>
<p>tfms = transforms.Compose([<br>
transforms.Resize((sz, sz)),  # PIL Image<br>
transforms.ToTensor(),        # Tensor<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
])</p>
<p>train_ds = datasets.ImageFolder(trn_dir, transform=tfms)<br>
train_dl = torch.utils.data.DataLoader(train_ds, batch_size=batch_size,<br>
shuffle=True, num_workers=0)</p>
<p>from sklearn.cluster import KMeans<br>
num_class=2<br>
kmeans = KMeans(n_clusters=num_class, random_state=0).fit(???)<br>
center=kmeans.cluster_centers_</p>

ptrblck: <p>The <code>fit()</code> methods expects a numpy array of the complete dataset, so you would need to get all samples of the <code>DataLoader</code> before feeding them to <code>KMeans().fit()</code>.<br>
You could use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.cluster.MiniBatchKMeans.html">MiniBatchKMeans</a> and use the <code>partial_fit</code> method, if you want to feed each batch to it.</p>
IoU metric 0 and -1

ngupta: <p>I am following <a href="https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb#scrollTo=Lkmb3qUu6zw3" rel="nofollow noopener">this</a> tutorial for faster RCNN on a custom dataset. I am not sure why my IoU metric shows 0 here.<br>
Could someone please help?</p>
<pre><code class="lang-auto">IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
</code></pre>
How to partition a neural network into sub-networks?

TinfoilHat0: <p>I’d like to partition a neural network into two sub-networks using Pytorch. To make things concrete, consider this image:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30eb7acf6d0e8f69dbdbd01c8dd71fa43739dc83.png" data-download-href="https://discuss.pytorch.org/uploads/default/30eb7acf6d0e8f69dbdbd01c8dd71fa43739dc83" title="partition_networks"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30eb7acf6d0e8f69dbdbd01c8dd71fa43739dc83_2_641x500.png" alt="partition_networks" data-base62-sha1="6YLqKXtkYa0JwJ02vSg42TvITab" width="641" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30eb7acf6d0e8f69dbdbd01c8dd71fa43739dc83_2_641x500.png, https://discuss.pytorch.org/uploads/default/original/3X/3/0/30eb7acf6d0e8f69dbdbd01c8dd71fa43739dc83.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/0/30eb7acf6d0e8f69dbdbd01c8dd71fa43739dc83.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">partition_networks</span><span class="informations">762×594 48.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In 1, I’ve a 3x4x1 neural network. What I want is, for example during epoch 1, I’d only like to update the weights in the sub-network 1, i.e., the weights that appear in the sub-network 2 must be frozen. Then again, in epoch 2, I’d like to train the weights that appear in sub-network 2 while the rest should frozen.</p>
<p>How can I do that?</p>
Understanding broadcasting with gradients

btimar: <p>Hi all- just want to check I understand the memory consumption of some code that involves broadcasting; it increases by an order of magnitude when I request gradients.</p>
<p>Using a gpu,</p>
<pre data-code-wrap="python"><code class="lang-python">device = torch.device('cuda:0')
</code></pre>
<p>I build two “batched” tensors and a fixed tensor to apply to them:</p>
<pre data-code-wrap="python"><code class="lang-python">d0, d1 = 100, 1000
batch_size = 800

x_ng = torch.randn(batch_size, d0, 1, requires_grad=False).to(device)
x_g = torch.randn(batch_size, d0, 1, requires_grad=True).to(device)
U = torch.randn(1, d1, d0).to(device)
</code></pre>
<p>Then I check gpu memory allocation for performing a matmul - it spikes when a broadcast is required with an input requiring gradients:</p>
<pre data-code-wrap="python"><code class="lang-python">check_mem = lambda s : print(f"{s}: {torch.cuda.memory_allocated()/1e6:.3f} MB")


check_mem("Before computation")

#requires broadcasting U, no grads
y_ng = torch.matmul(U, x_ng)
check_mem("After no-grad matmul")
del y_ng

#requires broadcasting U, grads
y_g = torch.matmul(U, x_g)
check_mem("After grad matmul")
del y_g

# no broadcast, grads
y_g = torch.matmul(U.squeeze(), x_g)
check_mem("After grad matmul (no broadcast)")
del y_g
</code></pre>
<p>with the result</p>
<blockquote>
<p>Before computation: 8.923 MB<br>
After no-grad matmul: 12.123 MB<br>
After grad matmul: 332.123 MB<br>
After grad matmul (no broadcast): 12.123 MB</p>
</blockquote>
<p>Here’s what I think is going on: when grads are required, pytorch needs to store the tensor multiplying x, and this cached version is taken <em>after</em> the broadcast - hence the large size. Is this accurate? Note that the size of the output tensor is the same in each case.</p>
<p>I don’t see anything related in the <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="noopener nofollow ugc">broadcasting docs</a>.</p>