akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Different CNN architectures tips

Jose-Bastos: <p>So I have been experimenting with multiple CNN architectures and wanted to clear some ideas. I did the usual CNN+flattening+Linear Layers 1st and that beats the baselines but not by a large margin. Suppose I have an Image with N points. I tried the following:</p>
<ol>
<li>CNN-&gt; no flattening → Linear Layer to reduce the number of channels(rehspaping the input of this linear layer to have the number of channels as the last dimension to be in accordance with pytorch’s convention) → output an image with the same dimensions as the input</li>
<li>
<pre><code>CNN -&gt; no flattening -&gt; Convolutional Layer to reduce the number of channels to only 1--&gt; Flattening the one channel left-&gt; Linear layers -&gt; output of N values (N is the number of pixels in the image)
</code></pre>
</li>
<li>
<pre><code>CNN -&gt; no flattening -&gt; having N parallel linear layers(with in_features=num_channels at the end of the CNN) that look into the same pixel in all channels -&gt; Output N values and reshape them into an image
</code></pre>
</li>
<li>
<pre><code>CNN -&gt; Grouped convolutions with kernel size=1 . This one I haven't managed to get running as intended. But it is the same idea as the architecture before. Having parallel parameters for each pixel after the initial CNN. I don't know if it is possible to do that with grouped convolutions...
</code></pre>
</li>
</ol>
<p>And the results were:</p>
<ol>
<li>Model wasn’t able to learn anything. Training loss stayed constant. I checked initializations and each layer output and it doesn’t seem to be a vanishing gradients problem. Adding more parameters did not work.</li>
<li>
<pre><code>Almost same results as the initial try(conventional CNN model).
</code></pre>
</li>
<li>
<pre><code>Here with just one group of N parallel linear layers it managed to decrease the Loss a bit. But when I added another group of N linear layers it got very slow while training. I suspect it is because I use nested for loops to feed each linear layer the desired inputs.
</code></pre>
</li>
</ol>
<p>I’m looking into einsum notation to maybe get rid of the for loops. Problem is: I don’t know if I can have only one weight matrix for all my linear layers and do the multiplication “by hand” with the outputs of the CNN using einsum notation to help me get the desired output for each parallel linear layer.</p>
<p>I have tried to use architectures that do not flatten the output of the convolutions to see if maintaining the information of spatial patterns learnt helps, because, in the problem at hand, I think it should help. But I haven’t really got good results. Is it worth not flattening, and why it isn’t(or is)?</p>
<p>So, what I’m asking is: what can I do better in those architectures that I’ve tried already? What new architectures do you recommend? Do you have any resources that I can read about CNNs without flattening(or why should we always flatten)?</p>
<p>Ps: If it helps, I’m trying to do weather forecast on a 2D spatial grid using previous time steps.</p>
<p>Sorry for the long post(and the bad English). Thank you for your time!</p>
Error: block: [0,0,0], thread: [4,0,0] Assertion `t >= 0 && t < n_classes` failed

sesale: <p>I am using an LSTM for multi-class many-to-one prediction (20 classes).<br>
When trying to calculate the loss using CrossEnthropyLoss, I receive the following error:</p>
<pre><code class="lang-auto">Error: block: [0,0,0], thread: [4,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.
</code></pre>
<p>My model output has size [batch_size,number_of_classes] (number_of_classes=output_dim in the below LSTM) and my target has size [batch_size]. My model output partially contains negative values: Is this an issue?.<br>
Below is my model:</p>
<pre><code class="lang-auto">class LSTM(Model):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, dropout = None):
        super(LSTM, self).__init__()

        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.output_dim = output_dim
        self.dropout = dropout
        self.lstm = nn.LSTM(self.input_dim, self.hidden_dim, self.layer_dim, batch_first=True, dropout= self.dropout if self.dropout else None)

        self.fclstm = nn.Linear(self.hidden_dim, self.output_dim)

    
    def forward(self, x):
        out = x.float()

        h0 = torch.zeros(self.layer_dim, out.size(0), self.hidden_dim).requires_grad_().to(next(self.parameters()).device)
        c0 = torch.zeros(self.layer_dim, out.size(0), self.hidden_dim).requires_grad_().to(next(self.parameters()).device)   
        out, _ = self.lstm(out, (h0.detach(), c0.detach()))

        # get last time step's hidden state (many-to-one)
        out = self.fclstm(out[:, -1, :])
        return out
</code></pre>
<p>But even when adding a ReLU after the fclstm Linear layer in order to squeeze all ouput values between 0 and 1, I still get the same error.<br>
Does anyone know what is causing the error?<br>
Thanks!</p>

InnovArul: <aside class="quote no-group quote-modified" data-username="sesale" data-post="1" data-topic="144999">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/cdc98d/48.png" class="avatar"> sesale:</div>
<blockquote>
<p><code>Assertion </code>t &gt;= 0 &amp;&amp; t &lt; n_classes``</p>
</blockquote>
</aside>
<p>Make sure that the <code>target</code> is between <code>0</code> and  <code>C-1</code> (inclusive), where <code>C = number of classes</code>.</p>
Logsumexp trick for log(2 / (1 + exp(x)))

Assefa_Seyoum: <p>I am computing the similarity between embeddings by converting the similarities to probability and using cross entropy loss.</p>
<p>Is there anything like the <a href="https://pytorch.org/docs/stable/generated/torch.logsumexp.html" rel="noopener nofollow ugc">logsumexp</a> trick I can do to avoid computing <code>exp(x)</code> in <code>log(2 / (1 + exp(x)))</code>?</p>

InnovArul: <p>I am not sure about the exact formulation of the loss function.<br>
From what I understand, you can use <code>logsumexp</code> with <code>log(2 / (1 + exp(x)))</code> too.</p>
<p><code>log(2 / (1 + exp(x))) = log 2 - log (1 + exp(x))               = log 2 - logsumexp([0, x])</code></p>
<pre><code class="lang-auto">import torch
x = torch.rand(5) * 10

actual = torch.log(2 / (1 + torch.exp(x)))

# concat 0 with x
catzero = torch.cat((torch.zeros(5,1), x.view(5,1)), dim=1) # 5x2
logexptrick = torch.log(torch.tensor(2)) - torch.logsumexp(catzero, dim=1)

</code></pre>
Inserting TORCH.CLAMP or range restriction into MobileNetV3

Jonas_ar: <p>Hello everyone,</p>
<p>I am currently testing the influence from soft error on the MobileNetV3 large. I am basically injecting bitflips into the model weights after the training has been completed, afterwards I am testing the impacts of the soft errors with the test dataset, whether there has been a drop in accuracy or not.<br>
My result shows that especially the MSB can have an large effect for float32 model weights since values can change from 0.0xx to 1*e^x for example.<br>
I want to prevent the model from basically computing the output with such large weights. And I came across the troch.clamp function.</p>
<p>From my understanding torch.clamp sets a min and max range for a tensor. So would it be possible to insert torch.clamp to basically mitigate the effect of large numbers after the training actually has completed since I don’t want to clamp the weights during the training. I want to prevent the model to “use” these large weights which completely fall out of the scale of the rest.</p>
<p>As a side note, i am using the already implemented MobileNetV3_large model from torchvision. So maybe there is a way to implement the range restriction without actually changing the models architecture.</p>
<p>For reference I am thinking of something similar to this <a href="https://github.com/DependableSystemsLab/Ranger" class="inline-onebox" rel="noopener nofollow ugc">GitHub - DependableSystemsLab/Ranger</a></p>
<p>Any help or suggestions would be appreciated.</p>

Jonas_ar: <p>Right now I am using a workaround. I am using torch.clamp after I injected the fault and before evaluating with the Test-Data, this seems to mitigate values which are higher than -1 and 1 (for example) that occur due to a bitflip.<br>
Values which are not within these borders are set either to the min value or to the max value in this case -1 and 1.<br>
The next step would be to implement this procedure into the architecture itself. Keeping others that might be interested up to date.</p>
A question about `padding` in `nn.MaxPool2d`

Ren_Pang: <p>According to Google’s pytorch implementation of Big Data Transfer, there is subtle difference between the following 2 approaches. Could anyone explain the difference? Is it some different strategy for boundary pixels?</p>
<p>What’s the purpose of spliting <code>padding</code> parameter from <code>nn.MaxPool2d</code> and making it a separate <code>nn.Pad</code> layer before the pooling?</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/google-research/big_transfer/blob/140de6e704fd8d61f3e5ea20ffde130b7d5fd065/bit_pytorch/models.py#L121-L124">
  <header class="source">

      <a href="https://github.com/google-research/big_transfer/blob/140de6e704fd8d61f3e5ea20ffde130b7d5fd065/bit_pytorch/models.py#L121-L124" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/google-research/big_transfer/blob/140de6e704fd8d61f3e5ea20ffde130b7d5fd065/bit_pytorch/models.py#L121-L124" target="_blank" rel="noopener nofollow ugc">google-research/big_transfer/blob/140de6e704fd8d61f3e5ea20ffde130b7d5fd065/bit_pytorch/models.py#L121-L124</a></h4>



    <pre class="onebox">      <code class="lang-py">
        <ol class="start lines" start="121" style="counter-reset: li-counter 120 ;">
            <li>('pad', nn.ConstantPad2d(1, 0)),</li>
            <li>('pool', nn.MaxPool2d(kernel_size=3, stride=2, padding=0)),</li>
            <li># The following is subtly not the same!</li>
            <li># ('pool', nn.MaxPool2d(kernel_size=3, stride=2, padding=1)),</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Image dataloader error

lanka: <pre><code class="lang-auto">traindir_A = 'F:/rgb_images/'
traindir_B = 'F:/rgb_images/'

train_data_t=300
test_data_t=44

class ConcatDataset(Dataset):
    def __init__(self, dataa,datab):
        self.datasets=dataa + datab


    def __getitem__(self, i):
        return tuple(d[i] for d in self.datasets)


    def __len__(self):
        return min(len(d) for d in self.datasets)

train_loader = torch.utils.data.DataLoader(ConcatDataset( datasets.ImageFolder(traindir_A), datasets.ImageFolder(traindir_B) ),
             batch_size=batch_size, shuffle=True,
             num_workers=workers, pin_memory=True)

def main():
    for i, (input, target) in enumerate(train_loader):
        print(input.shape,targrt.shape)

if __name__ == '__main__':
    main()
</code></pre>
<p>############ output error</p>
<p>File “F:\image_load.py”, line 5, in <br>
from torch.utils.data import Dataset, DataLoader, random_split, sampler<br>
File “C:\Users\Anaconda3\envs\py36\lib\site-packages\torch_<em>init</em>_.py”, line 124, in <br>
raise err<br>
OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading “C:\Users\Anaconda3\envs\py36\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll” or one of its dependencies.<br>
return _default_context.get_context().Process._Popen(process_obj)<br>
File “C:\Users\Anaconda3\envs\py36\lib\multiprocessing\context.py”, line 322, in _Popen<br>
return Popen(process_obj)<br>
File “C:\Users\Anaconda3\envs\py36\lib\multiprocessing\popen_spawn_win32.py”, line 65, in <strong>init</strong><br>
reduction.dump(process_obj, to_child)<br>
File “C:\Users\Anaconda3\envs\py36\lib\multiprocessing\reduction.py”, line 60, in dump<br>
ForkingPickler(file, protocol).dump(obj)<br>
BrokenPipeError: [Errno 32] Broken pipe</p>
<p>is this pytorch version error ?</p>

Dazitu616: <p>Maybe related <a href="https://discuss.pytorch.org/t/winerror-1455-the-paging-file-is-too-small-for-this-operation-to-complete/131233">here</a></p>
Regularization when using LBFGS optimizer

bananenpampe: <p>I am looking for a way to add regularization to the model weights when optimizing using the LBFGS optimizer. My validation error shows, that I am overfitting after ~20 epochs.<br>
Is there a way to introduce regularization, other than early stopping in pytorch?</p>

Dazitu616: <p>Not an expert here. According to the document, LBFGS has a <code>step</code> function that takes in the loss_function. Maybe you can directly apply the L2 regularization in the loss_function? Write code to compute the sum of the L2 norm of all the params and return</p>
<pre><code class="lang-auto">l2_norm = torch.tensor(0.)
for p in model.parameters():
    l2_norm += (p**2).sum()  # something similar, you get the point
</code></pre>
Will DDP still work if I call model.module?

Dazitu616: <p>Hi! I’m currently using DDP training. My model looks like:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):

    def __init__(self):
        ...

    def forward(self, data_dict):
        ...
        return out_dict

    def loss_func(self, data_dict):
        out_dict = self.forward(data_dict)
        loss = compute_loss(data_dict, out_dict)  # a function that calculates loss, not important here
</code></pre>
<p>Since <code>loss_func</code> is a class function of <code>MyModel</code>, I should use <code>loss = model.module.loss_func(xxx)</code> in the training code. I was wondering if PyTorch still performs DDP sync in this case (e.g. sync the gradient, or if I use <code>all_gather</code> on some output in <code>out_dict</code>, can I still gather them)?</p>
<p>I saw a related post <a href="https://discuss.pytorch.org/t/dataparallel-doesnt-work-when-calling-model-module-some-attribute/47556">here</a>, but it’s about DataParaller not DDP. And I understand that, because in DDP each training process is separated, so I won’t encounter that problem. I just don’t know if sync among process can still be achieved in my case. Thanks in advance!</p>

Dazitu616: <p>Ok so I verify it myself. DDP sync always work as long as that value is a CUDA tensor, and it appears in each process. It doesn’t matter if that tensor is e.g. created from a int value or even a numpy array etc.</p>
How to convert an integer to a tensor?

Prakhar_Sharma: <p>I know I can create a new torch.int8 tensor. But I want to convert an integer from python user input to a tensor for some specific task.</p>
<p>.to() or torch.from_int() doesn’t exist.</p>

ptrblck: <p>You can directly pass the <code>int</code> to a tensor and convert it afterwards to an <code>int8</code>.<br>
I’m not sure how you are getting <code>int8</code> input in Python so could you describe your use case a bit more and what exactly is not working?</p>

Dazitu616: <pre><code class="lang-auto">&gt;&gt;&gt; a = 1
&gt;&gt;&gt; torch.tensor(a, dtype=torch.int8)
tensor(1, dtype=torch.int8)
</code></pre>
<p>I think this will work</p>
How to synchronize to calculate loss in DDP?

oasjd7: <p>Within a mini-batch, different types of loss are calculated according to some conditions.<br>
Because of this, synchronization does not match and the gpu becomes stuck.<br>
I tried to solve this using <code>dist.barrier()</code>, but it didn’t work.<br>
How can I sync properly?</p>
<pre><code class="lang-auto">total_loss = 0
for i, idx in enumerate(idx_in_batch)
    if condition_A:
       loss_A = ...
        total_loss = total_loss + loss_A
    else:
        loss_B = ...
        total_loss = total_loss + loss_B

total_loss /= imgs.size(0)
print(total_loss)
# dist.barrier()
losses.update(total_loss.item(), imgs.size(0))

# compute gradient and do SGD step
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
</code></pre>
Error $ Torch: not enough memory: you tried to allocate 42GB. Buy new RAM!

Morphinepill: <p>This is the error:<br>
RuntimeError: $ Torch: not enough memory: you tried to allocate 42GB. Buy new RAM! at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/THGeneral.c:218</p>
<p>I imported everything separately and these are my transofrms</p>
<pre><code class="lang-auto">train_transform_list = [transforms.RandomRotation(30),
                        transforms.RandomResizedCrop(224),
                        transforms.RandomHorizontalFlip(),
                        transforms.ToTensor()
                       ]

test_transform_list = [transforms.Resize(255),
                       transforms.CenterCrop(224),
                       transforms.ToTensor()
                      ]

train_transform = transforms.Compose(train_transform_list)
test_transform = transforms.Compose(test_transform_list)


train_set = torchvision.datasets.CIFAR10(root='./data', download=True, transform = train_transform)
trainloader = torch.utils.data.DataLoader(train_set, batch_size = 64)


test_set = torchvision.datasets.CIFAR10(root='./data', download=True, transform = test_transform)
testloader = torch.utils.data.DataLoader(test_set, batch_size = 64)

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
</code></pre>
<p>And this is my network</p>
<pre><code class="lang-auto">device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(150528, 75264)
        self.fc2 = nn.Linear(75264, 64)
        self.fc3 = nn.Linear(64, 10)

        
    def forward(self, x):
        x = x.view(x.shape[0], -1)
        
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))

        
        x = F.log_softmax(self.fc3(x), dim=1)
        return x


model = Network()  
</code></pre>
<p>If I use smaller parameters I get another error which is<br>
RuntimeError: size mismatch, m1: [64 x 150528], m2: [4096 x 1024] at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generic/THCTensorMathBlas.cu:249<br>
(Using 4096 as the first parameter)<br>
How do I choose the right numbers for nn.Linear exactly?</p>

ptrblck: <p>The <code>in_features</code> are set as the number of features in the incoming activation tensor. To reduce this value you would need to reduce the activation shape e.g. by using smaller inputs, a more aggressive pooling etc.</p>
KeyError 0 in custom collate_fn

praggya: <p>def my_collate(batch):<br>
data = [item[0] for item in batch if item[0].shape[1]==256 and item[0].shape[2]==256]<br>
target = [item[1] for item in batch if item[0].shape[1]==256 and item[0].shape[2]==256]<br>
return [data, target]<br>
I tried creating a custom collate_fn for my dataloader such that the stack contains only images of [3,256,256] shape.Can anyone please tell me why i am facing KeyError 0 ?<br>
It’s Urgent.</p>

ptrblck: <p>Check what <code>batch</code> and <code>item</code> contains as it seems you cannot index the latter with <code>[0]</code>.</p>
How do I apply non maximum suppression to multiple images from a dataloader?

SU801T: <p>Hi,</p>
<p>I’m struggling to apply the Non Maximum Suppression (NMS) to multiple images from a dataloader.</p>
<p>I have a function where I apply NMS to one predicted image as seen here, the function defined below:</p>
<pre><code class="lang-auto">def apply_nms(orig_prediction, iou_thresh=0.3):
    
    # torchvision returns the indices of the bboxes to keep
    keep = torchvision.ops.nms(orig_prediction['boxes'], orig_prediction['scores'], iou_thresh)
    
    final_prediction = orig_prediction
    final_prediction['boxes'] = final_prediction['boxes'][keep]
    final_prediction['scores'] = final_prediction['scores'][keep]
    final_prediction['labels'] = final_prediction['labels'][keep]
    
    return final_prediction
</code></pre>
<p>Here I take one image from the validation set and infer some predictions, where I then apply NMS to that one image:</p>
<pre><code class="lang-auto"># one image from the test set
img, target = valid_dataset[2]
# put the model in evaluation mode
model.eval()
with torch.no_grad():
    prediction = model([img.to(device)])[0]

nms_prediction = apply_nms(prediction, iou_thresh=0.1)
</code></pre>
<p>How can I do this for multiple images (i.e. a batch of images)?</p>
<p>I currently have this set up:</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
  for images, targets in valid_data_loader:
    images = list(img.to(device) for img in images)
    predictions = model(images)
</code></pre>
<p>However, am struggling to understand how I can apply NMS to multiple images without using for loops…</p>
Pytorch Geometric AssertionError

Gianmarco_Luchetti_S: <p>Hello everybody,<br>
I’m learning to use PyTorch Geometric, I tried to replicate the WL kernel written by the PyG developers on GitHub (<a href="https://github.com/pyg-team/pytorch_geometric/blob/master/examples/wl_kernel.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch_geometric/examples/wl_kernel.py at master · pyg-team/pytorch_geometric · GitHub</a>), but when i run the program I get this error which I can’t solve:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97c279af77731ea3cf27a27dcaa31d0335233f96.png" data-download-href="https://discuss.pytorch.org/uploads/default/97c279af77731ea3cf27a27dcaa31d0335233f96" title="Schermata 2022-02-24 alle 22.01.01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97c279af77731ea3cf27a27dcaa31d0335233f96_2_689x219.png" alt="Schermata 2022-02-24 alle 22.01.01" data-base62-sha1="lEwL5AxfdCnJ0BJ0LuUBQc8SvOK" width="689" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97c279af77731ea3cf27a27dcaa31d0335233f96_2_689x219.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97c279af77731ea3cf27a27dcaa31d0335233f96_2_1033x328.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/97c279af77731ea3cf27a27dcaa31d0335233f96.png 2x" data-dominant-color="393A3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Schermata 2022-02-24 alle 22.01.01</span><span class="informations">1212×386 49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Which one had the same problem and was able to solve it? Thank you all!</p>
Speech recognition gets stuck

Ash23733: <p>Hi<br>
Im trying to run a model similar to the one shown here <a href="https://www.assemblyai.com/blog/end-to-end-speech-recognition-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Building an end-to-end Speech Recognition model in PyTorch</a><br>
It keeps getting stuck at 2.8 loss with 5 rnn’s but if i decrease it to 4 it actually learns but with a terrible error rate of 70 percent. Is there any way i could fix this.</p>
About dcgan.py example running

chensquan: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/83adfdf38439d1d7a92905aa4bfcd0d35441ebff.png" data-download-href="https://discuss.pytorch.org/uploads/default/83adfdf38439d1d7a92905aa4bfcd0d35441ebff" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/3/83adfdf38439d1d7a92905aa4bfcd0d35441ebff.png" alt="image" data-base62-sha1="iMTk8oaj7PYIPS8OJgxZ75eU70z" width="690" height="133" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1370×266 17.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
when I run the dcgan.py example，I set the batch_size=32, my gpu type is 1080Ti，it runs 8 epochs ,then call CUDA out of memory, how can I solve this problem？then I set the batch_size=16, the code can run，but the generator image is poor,the sigma I set is 0.7.</p>

karthikprasad: <p>Duplicate of <a href="https://discuss.pytorch.org/t/about-dcgan-py-results/144824" class="inline-onebox">About dcgan.py results</a>. Please direct all discussoons to that thread. This thread will not be monitored.</p>
Create a correct Tensor shape for BraTS

Anmar_Karmush: <p>I’m extremely new to deep learning so please forgive if this question is far to dumb or doesn’t make any sense.</p>
<p>But, I have download the BraTS data-set. And these images come in format (240,240,155). So the images are 240x240 in height x width and there are 155 slices per image. There are also 4 masks. I have converted all of these to numpys. And I now want to create a simple UNet.</p>
<p>So I would to like to create a function that has the following parameters:</p>
<pre><code class="lang-auto">def simple_unet_model(IMG_HEIGHT, IMG_WIDTH, IMG_DEPTH, IMG_CHANNELS, num_classes):
    #Create Tensor with parameters IMG_HEIGHT, IMG_WIDTH, IMG_DEPTH, IMG_CHANNELS
 
</code></pre>
<p>So I would like to create this input tensor. And, when I look at its size or shape. I would like to get (240,240,155,1,3).<br>
The 1 is there (I think) since I’m only working with T1-images and they are all in grayscale. Is this possible in PyTorch?</p>
<p>Again, Sorry if this question is bad, but maybe someone understands me.</p>

ptrblck: <p>Yes, creating this method would be possible and the more interesting part would be the model definition itself. You could check for popular UNet implementations in PyTorch and reuse them, if they would fit your use case.</p>

Anmar_Karmush: <p>Thanks for replying ptrblck.</p>
<p>I think I’m on the right track but I have a problem which probably is really simple, but I can’t seem to solve it.<br>
I have a U-net now and I want to test it.<br>
So i uploaded a 2D image of size (240, 240). However, my unet takes 4 arguments.<br>
[batch_size, channels, height, width]. How can I add these two dimensions so when i take for example image.size() I get torch.size([1,1,240,240]).</p>

ptrblck: <p>You could either use <code>unsqueeze</code> or index the tensor with <code>None</code>s (as is also done in e.g <code>numpy</code>):</p>
<pre><code class="lang-python">x = torch.randn(240, 240)
print(x.size())
# &gt; torch.Size([240, 240])

y1 = x.unsqueeze(0).unsqueeze(0)
print(y1.size())
# &gt; torch.Size([1, 1, 240, 240])

y2 = x[None, None, :, :]
print(y2.size())
# &gt; torch.Size([1, 1, 240, 240])
</code></pre>

Anmar_Karmush: <p>Thank you for all your help!</p>
Torchvision SSD how to put data on GPU?

ekmungi: <p>Hi,</p>
<p>I am trying to use the <a href="https://github.com/pytorch/vision/blob/33db2b3ebfdd2f73a9228f430fa7dd91c3b18078/torchvision/models/detection/ssd.py" rel="noopener nofollow ugc">SSD implementation on torchvision</a></p>
<p>As a test I wanted to use the following function from <a href="https://github.com/pytorch/vision/blob/33db2b3ebfdd2f73a9228f430fa7dd91c3b18078/torchvision/models/detection/ssd.py#L522" rel="noopener nofollow ugc">line 522</a>:</p>
<pre><code class="lang-auto">def ssd300_vgg16(pretrained: bool = False, progress: bool = True, num_classes: int = 91,
                 pretrained_backbone: bool = True, trainable_backbone_layers: Optional[int] = None, **kwargs: Any):
</code></pre>
<p>I have images of different sizes so I am creating a list of tensors for image.</p>
<p>The <code>SSD.py</code> documentation states on <a href="https://github.com/pytorch/vision/blob/33db2b3ebfdd2f73a9228f430fa7dd91c3b18078/torchvision/models/detection/ssd.py#L108" rel="noopener nofollow ugc">line 108</a>:</p>
<pre><code class="lang-auto">During training, the model expects both the input tensors, as well as a targets (list of dictionary),
    containing:
        - boxes (``FloatTensor[N, 4]``): the ground-truth boxes in ``[x1, y1, x2, y2]`` format, with
          ``0 &lt;= x1 &lt; x2 &lt;= W`` and ``0 &lt;= y1 &lt; y2 &lt;= H``.
        - labels (Int64Tensor[N]): the class label for each ground-truth box
</code></pre>
<p>I had couple of question, I was hoping someone could help:</p>
<ul>
<li>how to put the variable sized images that are currently in a list of tensors to GPU?</li>
<li>The function expects list of dictionaries containing bboxes and labels… how do I put this on GPU?</li>
</ul>
<p>Thank you… any help would be super appreciated.</p>

suraj.pt: <p>Hey <a class="mention" href="/u/ekmungi">@ekmungi</a></p>
<p>You can move your tensors to the GPU before packing them in a list or dict eg:<br>
<code>input_list = [img.cuda() for img, _ in loader]</code></p>
<p>This tutorial is a good reference <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.10.1+cu102 documentation</a> for creating your dataset and loader. When packing inputs for your model, use</p>
<pre><code class="lang-auto">images = list(image.cuda() for image in images)
targets = [{k: v.cuda() for k, v in t.items()} for t in targets]
</code></pre>
<p>Hope this helps!</p>

ekmungi: <p>Thank you Suraj. That worked for me.<br>
The link was very useful. I have an idea on how to modify my dataset <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
<p>Anant.</p>
Using BCEWithLogitsLoss in training and BCELoss for test

HCH: <p>Hi,</p>
<p>I have a binary model. I used the Sigmoid activation function and BCELoss for loss function. when I normalized my vector(input), I did not get good prediction (the prediction dropped but without normalization it works properly). So I remove sigmoid from my network and instead use BCEWithLogitsLoss in training phase but keep BCELoss for the test. I am getting this error in test phase : 'all elements of input should be between 0 and '. I know why it is happening, becuase the prediction in test is not between 0 - 1. My question is why I am getting prediction in test out of 0-1 range? should I use BCEWithLogitsLoss for test as well? if so, I do not understand why.</p>
<p>Thanks</p>

chiragjn: <aside class="quote no-group" data-username="HCH" data-post="1" data-topic="144922">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hch/48/46648_2.png" class="avatar"> HCH:</div>
<blockquote>
<p>why I am getting prediction in test out of 0-1 range? should I use BCEWithLogitsLoss for test as well? if so, I do not understand why.</p>
</blockquote>
</aside>
<p>Simply because without the sigmoid activation your model will give you logits that are not guaranteed to be bounded between 0 and 1.</p>
<p>As the name implies <code>BCEWithLogitsLoss</code> can compute binary cross-entropy from the raw logits while the <code>BCELoss</code> needs a binary Tensor as mentioned in the docs (<a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss" class="inline-onebox" rel="noopener nofollow ugc">BCELoss — PyTorch 2.1 documentation</a>)</p>
<p>See past discussion here: <a href="https://discuss.pytorch.org/t/bceloss-vs-bcewithlogitsloss/33586" class="inline-onebox">BCELoss vs BCEWithLogitsLoss</a></p>
<p>So there are two options:</p>
<ol>
<li><code>model(input)</code> → logits → <code>BCEWithLogitsLoss</code> → loss</li>
<li><code>model(input)</code> → logits → <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.sigmoid.html" rel="noopener nofollow ugc">F.sigmoid</a> → <code>BCELoss</code> → loss</li>
</ol>
<p>I would recommend using the same steps during both training and test to avoid discrepancies</p>
3-D Scatter (Add)

nitish: <p>Is there a way to implement 3-D scatter_/scatter_add_ in pytorch?<br>
Specifically, I have a tensor <code>X</code> (shape <code>(B, M, N)</code>), an index tensor <code>I</code> of shape <code>(B, C, D)</code> giving indices into the <code>X</code>, and a values tensor <code>V</code> the same shape of as indices, <code>(B, C, D)</code>. For indices <code>(b, c, d)</code> in <code>I</code>, I want to add <code>V[b,c,d]</code> to <code>X[b,c,d]</code>.</p>
<p>Is there a way to implement this? The regular <code>scatter_</code> function only indices using a single dim.</p>

ptrblck: <p>Is <code>I</code> holding the indices or are the actual indices stored in <code>b</code>, <code>c</code> and <code>d</code>?<br>
In the latter case, would this work?</p>
<pre><code class="lang-python">b = 10
m, n = 5, 6
c, d = 4, 2
x = torch.zeros(b, m, n)
b_idx = torch.empty(b, 1, 1, dtype=torch.long).random_(b)
c_idx = torch.empty(c, 1, dtype=torch.long).random_(c)
d_idx = torch.empty(d, dtype=torch.long).random_(d)
v = torch.randn(b, c, d)

x[b_idx, c_idx, d_idx] += v[b_idx, c_idx, d_idx]
</code></pre>
<p>If not, could you post a small sample, what’s stored in <code>I</code>?</p>

mok33: <p>Hello,<br>
I want to do the same thing. However, its in my training so I want to retain the gradient. I get a <code>RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</code> because X is my CNN outputs. So I have to use torch.scatter but I can’t wrap my head on how to do that ?</p>
How to debug forward method of your custom model head?

Muhammad_Hamza_Shafi: <p>Hi,<br>
I am pretty new to pytorch and neural networks. So far have only worked with pretrained model in pytorch. I want to implement a cusstom model to my already existing pipeline. I have found deeplabv3 plus implementation and trying to modify it for my pipeline. I have following implementation and I am getting this error.</p>
<pre><code class="lang-auto">class DeepLabHeadV3Plus(nn.Module):
    def __init__(self, in_channels, low_level_channels, num_classes, aspp_dilate=[12, 24, 36]):
        super(DeepLabHeadV3Plus, self).__init__()
        self.project = nn.Sequential( 
            nn.Conv2d(low_level_channels, 48, 1, bias=False),
            nn.BatchNorm2d(48),
            nn.ReLU(inplace=True),
        )

        self.aspp = ASPP(in_channels, aspp_dilate)

        self.classifier = nn.Sequential(
            nn.Conv2d(304, 256, 3, padding=1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, num_classes, 1)
        )

    def forward(self, feature):
        low_level_feature = self.project( feature['low_level'] )
        output_feature = self.aspp(feature['out'])
        output_feature = F.interpolate(output_feature, size=low_level_feature.shape[2:], mode='bilinear', align_corners=False)
        return self.classifier( torch.cat( [ low_level_feature, output_feature ], dim=1 ) )
</code></pre>
<p>My error is:</p>
<blockquote>
<p>low_level_feature = self.project( feature[‘low_level’] )<br>
IndexError: too many indices for tensor of dimension 4</p>
</blockquote>
<p>I get it that I have tensor of size 04 but I am trying to dimension it an extra. but how can I get rid of it. My dataloader has shape of tensor (B,C,H,W) = (32, 3, 320, 320). Any suggestion or help would be a lot of help.</p>

chiragjn: <p>you can stick in a <code>pdb.set_trace()</code> in forward to debug interactively. Of course don’t forget to import pdb.</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, feature):
        pdb.set_trace()
        low_level_feature = self.project( feature['low_level'] )
        ...
</code></pre>
<aside class="quote no-group" data-username="Muhammad_Hamza_Shafi" data-post="1" data-topic="144937">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muhammad_hamza_shafi/48/46989_2.png" class="avatar"> Muhammad_Hamza_Shafi:</div>
<blockquote>
<p>low_level_feature = self.project( feature[‘low_level’] )<br>
IndexError: too many indices for tensor of dimension 4</p>
</blockquote>
</aside>
<p>from the error message, it seems your dataloader is giving you tensors with more than 4 dimensions. I would recommend double-checking your input loading and conversion.</p>
TCP initialization for AWS different Regions

HuYang719: <p>Hi~ I want to use <a href="https://pytorch.org/docs/stable/distributed.html#tcp-initialization" rel="noopener nofollow ugc">TCP initialization</a> in <code>torch.distributed.init_process_group</code> for different AWS regions. I am sure I open all traffic in AWS EC2 instances and they could be communicated with socket using public IP. However, when I use TCP initialization to initial the process group, the master node will only store the <strong>private IP</strong> in the TCP_Store from other nodes, thus I guess why they cannot communicate.</p>
<p>Could I force the initial process to store public IP from other nodes? Or is there any other methods to let nodes in different area (but can communicate with public IP) to use torch.distributed? Thanks for any help!</p>

Yanli_Zhao: <p>cc <a class="mention" href="/u/cbalioglu">@cbalioglu</a> , do you have any insight about it?</p>

cbalioglu: <p><a class="mention" href="/u/huyang719">@HuYang719</a> How do you initiate your jobs right now? Using <code>python -m torch.distributed.run</code>?</p>
Encoder-decoder issues

Leo2: <p>Hi,<br>
I am attempting to implement an abstractive summarisation deep learning model on the gigaword dataset.  I’m using an lstm for both encoding and decoding and I’ve tried to also include Bahdanau attention. On top of this i’ve got a pointer mechanism to allow for some words to be copied rather than generated.</p>
<p>my model seems to not be doing very well in training. Both greedy and Beam search results in repeated tokens and shows little to no improvement over each batch.</p>
<p>Would anybody be kind enough to have a look at my colab notebook or perhaps i could paste the code here?</p>
<p>regards<br>
Leo</p>
Autograd on the window size of Hann window of STFT

JXuan: <p>Hello,</p>
<p>I have a question to the <code>requires_grad=</code> option of <code>torch.hann_window()</code>: the window size of Hann window is supposed to be positive integers and therefore the Hann window function is not differentiable. How does the autodiff/autograd work here? I scratched my head for quite some time but could not figure out how the operation is legitimate.</p>
<p>Many thanks in advance!</p>

ptrblck: <p>I don’t think the actual tensor creation will be differentiable, but <code>requires_grad</code> would change the output tensor attribute in the same way as other factory methods such as <code>torch.randn</code> work:</p>
<pre><code class="lang-python">torch.hann_window(8)
# &gt; tensor([0.0000, 0.1464, 0.5000, 0.8536, 1.0000, 0.8536, 0.5000, 0.1464])
torch.hann_window(8, requires_grad=True)
# &gt; tensor([0.0000, 0.1464, 0.5000, 0.8536, 1.0000, 0.8536, 0.5000, 0.1464],
#         requires_grad=True)

torch.randn(5)
# &gt; tensor([-2.0748,  0.8152, -1.1281,  0.8386, -0.4471])
torch.randn(5, requires_grad=True)
# &gt; tensor([-0.5538, -0.8776, -0.5635,  0.5434, -0.8192], requires_grad=True)
</code></pre>

JXuan: <p>Many thanks for the clarification!</p>
Indexing | how to index the gray regions

Assefa_Seyoum: <p>How to index the gray regions? The indexing has to be generic, i.e., I can add D1, D2, D3 and it should still work.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09ac2918c8d01e6ec27489e1269be16ee6e60f2c.png" alt="image" data-base62-sha1="1nz9gAM7t7yEW44ToVWBwo2HdE0" width="409" height="335"></p>

Yuv: <p>Assuming you want blocks of the size mxm in matrix  of the size nxn :</p>
<pre><code class="lang-auto">for i in range round(n/m):
   elements[:,:,i] = matrix[m*(i-1):i*m ,m*(i-1):i*m]
</code></pre>
<p>Another way and maybe easier is using <a href="https://pytorch.org/docs/stable/generated/torch.block_diag.html" rel="noopener nofollow ugc">torch.block_diag</a>:</p>
<pre><code class="lang-auto">A = torch.tensor([[1, 1], [1, 1]])
B = torch.block_diag(A, A, A)
</code></pre>
<p>This will result with :</p>
<pre><code class="lang-auto">tensor([[1, 1, 0, 0, 0, 0],
        [1, 1, 0, 0, 0, 0],
        [0, 0, 1, 1, 0, 0],
        [0, 0, 1, 1, 0, 0],
        [0, 0, 0, 0, 1, 1],
        [0, 0, 0, 0, 1, 1]])
</code></pre>
<p>And maybe use B == 1</p>
How random is AdamOptimizer

zzk: <p>after 5 iteration，the backward results are different with same input</p>

ptrblck: <p>I don’t think <code>Adam</code> adds randomness. Could you describe your issue with more details, please?</p>

zzk: <p>I use manual_seed_all to make all randoms be constant in the net, so randoms initialized for each run are the same.<br>
The input dataset and the initial values for the variables of AdamOptimizer are also the same, but i can not  align the values ​​include losses 、weights of conv and gradient after 5 iter or 10 iter.<br>
How is it that?</p>

ptrblck: <p>Take a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> which explain how deterministic behavior can be achieved.</p>
Torch.equal does not have the same behavior as ==

GuillaumeTong: <p>Here is a small code snippet demonstrating a case where torch.equal does not behave like the == syntax:</p>
<pre><code class="lang-auto">import torch
t1 = torch.tensor([1])
t2 = torch.tensor([[1,2,3]])
print(t1==t2)
# tensor([[ True, False, False]])
print(torch.equal(t1,t2))
# False
</code></pre>
<p>I was trying to use torch.equal because the output of == syntax gets interpreted as boolean type by my IDE, and it then tends to warn me when I try to do tensor operations to the result.</p>
<p>What Pytorch function can I use to mimic the == syntax? Maybe I should manually broadcast the tensors before torch.equal?</p>

ptrblck: <p><code>torch.eq(t1, t2)</code> should work.</p>

GuillaumeTong: <p>After a little digging into the documentation, it turns out <code>torch.equal</code> is indeed meant to always output a single boolean:</p>
<pre><code class="lang-auto">print(torch.equal(t2,t2))
# True
</code></pre>
<p>Instead, I should have used <code>torch.eq</code>:</p>
<pre><code class="lang-auto">print(torch.eq(t1,t2))
# tensor([[ True, False, False]])
</code></pre>
<p>Although simple enough, I have to say that the role of <code>torch.equal</code> is very counter-intuitive, considering how other functions are defined, such as <code>torch.greater_equal</code>, <code>torch.less_equal</code>, and <code>torch.not_equal</code>.</p>