akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Splitting ImageFolder into data and labels?

Michel_Sfeir: <p>Hello all, I’m new here and sorry if this question has been asked before, I have tried searching for a long time now and cannot seem to find the solution.</p>
<p>When I load a standard dataset from pytorch such as CIFAR, I can access the images and labels separately with dataset.data and dataset.targets. When I create a dataset with ImageFolder however, I only get back one list that does not contain .data or .label fields. I know that I can access the images and labels separately by doing dataset[i][0] or [1] to get respectively the ith image or label. But this list structure is bothersome because a) I cannot take the mean or std of my training set easily and b) I cannot pass the images and labels separately to stratifiedshufflesplit to create a validation set. (I know that I can use random_split but that does not guarantee equal distribution of classes in the validation set).</p>
<p>The way that I was using was to do:</p>
<p>images = []<br>
labels = []<br>
for i in range(len(my_data)):<br>
images.append(my_data[i][0])<br>
labels.append(my_data[i][1])</p>
<p>Which would give me separate lists of just the data or labels that I could then turn into a tensor with tensor.stack() for the data and a numpy array for the labels, both much nicer to work with.</p>
<p>This method works fine but uses up a lot of memory and is very slow. When trying to run it on Kaggle, 90% of the time my notebook restarts due to trying to use too much memory.</p>
<p>Can someone help me divide up my imagefolder into data and labels in a neater/more efficient way? Or am I not supposed to try and do that? Is there something conceptually wrong with what I am attempting here?</p>
<p>I also considered editing ImageFolder itself to have .data and .targets fields but I don’t know how I would go about doing that.</p>

Isaac_Kargar: <p>Could you please explain about your folder structure? data, labels, …</p>

Michel_Sfeir: <p>It goes like this:</p>
<p>/root/train/[class names]/[images]<br>
/root/test/[only one class]/[testing images]</p>
<p>Since my test data is not labeled, I’m only reading the train folder into ImageFolder and trying to split that into a training and validation set</p>

Isaac_Kargar: <p>I usually use a custom dataset class for this type of problems and in the <code>__init__</code> method read the name of images and do sorting and other stuff if needed and then in the <code>__getitem__</code> method, load the image and do the processing and return the image and its label. You can have a lot of freedom and can control everything in the class. You can check <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">this</a> tutorial for more details.</p>
How should I compute the Jacobian matrix of a layer with respect its input?

S_M: <p>hello guys<br>
I’m trying to compute the Jacobian of a hidden layer with respect to its input so I found `torch.autograd.functional.vjp’, but I’m not sure that is what I am searching about that.<br>
‘vjp’ gives three input arguments (func, input,v) the first argument in my problem is the function which produces the input of mentioned hidden layer (e.g. f(x*w)+b), the second argument is “x” and the last is the hidden layer’s weight matrix.<br>
shall I use ‘vjp’? any help would be appreciated.</p>
Loss goes down on GPU0, but up on other 3 when using DistributedDataParallel

Martun_Karapetyan: <p>Hi everyone,</p>
<p>I have stumbled upon a problem when using DistributedDataParallel. Strangely, after a few epochs of successful training, loss goes up for a while. I noticed that both train/validation losses got down for the batches which are on GPU0, but go up for the other 3 GPUs. I believe I’m doing something wrong with DistributedDataParallel, but can’t find a bug. Did anyone see a similar problem, or can guess what the reason can be?</p>
<p>In the chart you can see training and validation losses for GPU0 and average of all 4.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c51901ec4bee4067e6dac1bf77663d2ab02ec9fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/c51901ec4bee4067e6dac1bf77663d2ab02ec9fd" title="Screenshot from 2020-05-18 05-16-58"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c51901ec4bee4067e6dac1bf77663d2ab02ec9fd_2_690x388.png" alt="Screenshot from 2020-05-18 05-16-58" data-base62-sha1="s7BDGfZRbrYQ4R0CPe6i9XAxcUB" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c51901ec4bee4067e6dac1bf77663d2ab02ec9fd_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c51901ec4bee4067e6dac1bf77663d2ab02ec9fd_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c51901ec4bee4067e6dac1bf77663d2ab02ec9fd_2_1380x776.png 2x" data-dominant-color="DED7D0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-05-18 05-16-58</span><span class="informations">1920×1080 323 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mrshenli: <p>Hey <a class="mention" href="/u/martun_karapetyan">@Martun_Karapetyan</a>, DDP should have kept all model replicas in sync, i.e., all model replicas should have the same parameter values. Could you please check if this is true in your use case, say by using <code>all_gather</code> to collect all model parameters into one rank and compare?</p>

Martun_Karapetyan: <p>Thanks for the help.</p>
<p>I checked the model parameters, they were in perfect sync.</p>
<p>I had another stupid bug. I used ReduceLROnPlateau when the validation accuracy plateaued, but each process looked at the validation accuracy of its subset of data. 1st process reduced the learning rate first, the others reduced it 1 epoch later, hence the problem.</p>
Linear NN for MNIST

xzlhqed: <p>Hi</p>
<p>I apologise for the basic question. I am trying to implement a linear NN (linear activation, MSE error minimisation) to train MNIST on and I am having difficulties that I haven’t been able to resolve from googling. My model was initially defined as follows</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28*28, 50)
        self.fc1_drop = nn.Dropout(0.2)
        self.fc2 = nn.Linear(50, 50)
        self.fc2_drop = nn.Dropout(0.2)
        self.fc3 = nn.Linear(50, 10)

    def forward(self, x):
        x = x.view(-1, 28*28)
        x = self.fc1_drop(x)
        x = self.fc2_drop(x)
        return F.log_softmax(self.fc3(x), dim=1)

model = Net().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.5)
criterion = nn.MSELoss()
</code></pre>
<p>but when I tried training I got an error from using MSELoss because the output of the neural net (for batch size of 32) has size (32, 10) and the target has size (32). I tried using target.view(1, -1).float(), and the model ran, but I got nonsensical values for the loss.</p>
<p>I changed the model so that <code>self.fc3 = nn.Linear(50, 1)</code> to return a single value, but when I train the network every value returned is 0. I understand that for MSELoss to work, the tensors must match in trailing dimensions, but I am currently struggling to figure out how to make it actually work for MNIST.</p>

satyajitghana: <p>try printing out the output of the model and the target, i think the model is outputing probabilities of each of the possible number [1-10] , you’ll have to do i convert the target to one hot and then apply a loss function,</p>
<p>also take a look here (uses the MNIST dataset from pytorch) <a href="https://github.com/AvivSham/Pytorch-MNIST-colab/blob/master/Pytorch_MNIST.ipynb" rel="nofollow noopener">https://github.com/AvivSham/Pytorch-MNIST-colab/blob/master/Pytorch_MNIST.ipynb</a></p>
Applying convolution inside dataloader

smani: <p>I am trying to apply convolution/pooling inside data loader as follows:</p>
<pre><code class="lang-auto">def __getitem__(self, index):
        target = self.lbls[index]
        # read image and convert to PIL image
        I = openImage(self.fnArr[index]) 
        I = TF.to_pil_image(I, mode='RGB')
        # apply transformations, including totensor()
        I = self.transform(I)
        # here I need to transfer I to GPU so that I can apply pooling 
        I = I.cuda() # I am getting error here
        re = self.pool(I)
</code></pre>
<p>But I am getting the following error: " Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method".<br>
What should I do to overcome this?</p>

jcbrouwer: <p>DataLoaders spawn multiple processes under the hood to load your data in parallel and like the error says, pytorch doesn’t support multiprocessing + CUDA without using torch.multiprocessing.</p>
<p>Why does the convolution/pooling need to be done in the dataloader? You could just do it as the first step in your main process as soon as you get the data from the dataloader.</p>

smani: <p>I dont use multiprocessing.  Inside dataloader I need to load large images of different sizes, preprocess them and concatenate sub images extracted from them. For preprocessing I need to calculate average of windows. For that I need to apply pooling.</p>

jcbrouwer: <p>Ok, then I think you want to write a custom Dataset class and then use a Dataloader with num_workers=0.</p>
<p>That way the dataloader shouldn’t do any multiprocessing and so you can use your GPU to preprocess in the Dataset class.</p>

smani: <p>Great. Thanks a lot.</p>
Count nonzeros element along an axis

jpainam: <p>Hi!<br>
I have a <code>(32x750)</code> tensor</p>
<pre><code class="lang-python">tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0043],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0043],
        [ 0.0000,  0.0044,  0.0000,  ...,  0.0044,  0.0000,  0.0000],
        ...,
        [ 0.0059,  0.0000,  0.0059,  ...,  0.0059,  0.0000,  0.0000],
        [ 0.0059,  0.0000,  0.0059,  ...,  0.0059,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0056,  0.0000]], device='cuda:0')
</code></pre>
<p>And I want to get  the number of nonzero elements along each rows. Something like that<br>
<code>[12 47 0 5 .... 8 7 50]</code><br>
This <a href="https://discuss.pytorch.org/t/find-non-zero-elements-in-a-tensor/4493/2">discussion</a> didn’t solve my problem and concerned the number of nonzero elements for 1-D tensor.</p>
<p>Thanks</p>

agadetsky: <p>Maybe something like this:</p>
<p><code>750 - (tensor == 0).sum(dim=1)</code></p>

agadetsky: <p>Oh, you want differentiable counting</p>

jpainam: <p>Thanks,<br>
It worked<br>
I’m using smooth label regularization and the idea is to normalize the output with target values which will contribute to the grad</p>

steysie: <p><code>(tensor != 0).sum(dim=1)</code> works as well and you don’t have to mention dimentionality (750).</p>
CUDA out of memory when changing first conv layer of resnet

Xiaotong_Chen: <p>Hi,</p>
<p>I am trying to change the first layer in resnet-50-fpn backbone inside Mask R-CNN provided in torchvision library. I changed it to 4 channels for RGB-depth input data while keeping the weights for first 3 channels using the following code:</p>
<pre><code class="lang-auto">self.backbone = resnet_fpn_backbone(arch, pretrained)
# change first layer to 4 channel for early fusion with 1 channel depth, load pretrained weights on RGB channels
conv1_weight_old = self.backbone.body.conv1.weight
conv1_weight = torch.zeros((64, 4, 7, 7))
conv1_weight[:, 0:3, :, :] = conv1_weight_old
avg_weight = conv1_weight_old.mean(dim=1, keepdim=False)
conv1_weight[:, 3, :, :] = avg_weight
self.backbone.body.conv1.weight = torch.nn.Parameter(conv1_weight)
</code></pre>
<p>The problem is very weird: it gives ‘CUDA out of memory error’ after training 1 or 2 epochs and starts a new one. I checked gpu memory is full when I stop the program in debug mode on the line it passes forward the backbone. I made a test to train 10 epochs using 10 images. It sometimes works fine and pass all epochs, but in most cases it gives memory error.</p>
<p>I also tried commenting two lines calculating average but it makes no difference.<br>
The first conv layer in resnet was like this before changing its shape</p>
<pre><code class="lang-auto">nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
</code></pre>

Isaac_Kargar: <p>I think you need to decrease your batch size. When you add one additional channel, the network size will increase compared to original resnet. You gpu memory is not enough for this batch size.</p>

Xiaotong_Chen: <p>Actually I don’t think it’s memory size problem. And I found somehow I am able to run it in ‘run’ mode without this bug anymore, but cannot go through it in ‘debug’ mode, which are two modes in PyCharm IDE. It’s weird but it happens. Thanks anyway!</p>

Isaac_Kargar: <p>I have the same problem with pycharm debugging too. I try to debug my code without parallelization and with just fewer batch size or and data. It consumes too much memory in debug I think. I also tried vscode. It is lighter and faster for such problems.</p>

Xiaotong_Chen: <p>Oh I can try it! I tried it before but found it could not go inside functions related to neural networks.</p>
How to use Mask-RCNN model to extract features of boxes?

cjwcommuny: <p>I’d like to use <code>torchvision.models.detection.MaskRCNN</code> model to detect objects and extract the corresponding features of bounding boxes of objects?</p>
<p>Anyone has any instruction or idea on how to do it properly?</p>

Isaac_Kargar: <p>I think you can use the bounding box coordinates and select the corresponding area of featuremap using those coordinates.</p>

cjwcommuny: <aside class="quote no-group" data-username="Isaac_Kargar" data-post="2" data-topic="81771" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/isaac_kargar/48/9424_2.png" class="avatar"> Isaac_Kargar:</div>
<blockquote>
<p>I think you can use the bounding box coordinates and select the corresponding area of featuremap using those coordinates.</p>
</blockquote>
</aside>
<p>But if I use the bounding box coordinates and select the corresponding area of featuremap by using those coordinates, I would get features of different size for different boxes. What I want is features with same dimension.</p>

Isaac_Kargar: <p>I think you need to find a way to handle that. Maybe max pooling, average pooling, summation or sth like that. I’m not sure if resizing all of them to a fixed size would work or not.</p>
Shape of Sigmoid activation function in NN

liran: <p>Hi,<br>
I try to build NN for binary classification. here is my code:</p>
<pre><code class="lang-auto">CLASSES_LENGTH = 1

class NN_FullyConnected(nn.Module):
    def __init__(self):
        super().__init__()
        self.Input_to_Hidden1 = nn.Linear(DIMENSION, int(DIMENSION))
        self.DropOut1 = nn.Dropout(p=0.5)
        self.Hidden1_to_Hidden2 = nn.Linear(int(DIMENSION), int(DIMENSION))
        self.DropOut2 = nn.Dropout(p=0.5)
        self.Hidden2_to_Hidden3 = nn.Linear(int(DIMENSION), int(DIMENSION))
        self.DropOut3 = nn.Dropout(p=0.5)
        self.Hidden3_to_Output = nn.Linear(int(DIMENSION), CLASSES_LENGTH)

    def forward(self, X):
        # set_trace()
        x_inp = nn.ReLU()(self.Input_to_Hidden1(X))
        x_d1 = self.DropOut1(x_inp)
        x_hid1_2 = nn.ReLU()(self.Hidden1_to_Hidden2(x_d1))
        x_d2 = self.DropOut2(x_hid1_2)
        x_hid2_3 = nn.ReLU()(self.Hidden2_to_Hidden3(x_d2))
        x_d3 = self.DropOut3(x_hid2_3)
        x_out = self.Hidden3_to_Output(x_d3)
        y =nn.Sigmoid()(x_out)
        return y
</code></pre>
<p>In the debug, I saw that the output shape (y) is ([10003,1])</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88a763540bcb03b66d6d582e34b54d7c25de0c9d.png" alt="image" data-base62-sha1="juTyUDbccCGp0mGSZC8iFQhCdjL" width="243" height="26"></p>
<p>I expect to get only ([10003]). what I`m doing wrong?<br>
Thank you!!</p>

Isaac_Kargar: <p>Why do you expect to get that shape and not [10003, 1]? What if the number of calsses is 2? It would be [10003, 2], right? So for one class I think it’s ok to have [10003, 1].</p>

liran: <p>Hi <a class="mention" href="/u/isaac_kargar">@Isaac_Kargar</a>, Thank you for your answer.<br>
I know, but I got a problem in the loss function:<br>
my code:<br>
<code>loss = criterion(y_pred, y_real.to(device))</code></p>
<p>my criterion is BCE</p>
<p><code>ValueError: Target size (torch.Size([10003])) must be the same as input size (torch.Size([10003, 1]))</code></p>
<p>Thats why I started to consider about the output from the NN.<br>
Do I need to reshape the output before using the criterion function?</p>
<p>Thank you</p>

Isaac_Kargar: <p>Yes you can do it using y_real.view(-1, 1).</p>
How to link release and debug version of libtorch library with CMake?

RuFAI: <p>Hi everybody,</p>
<p>i wrote some code using the libtorch library. For this I used the debug version of libtorch. I linked it with cmake as decribed on the tutorial page ‘using the c++ frontend’. Now, I want to bring my code to release version. How do I link both the debug and the release version of the library via CMake so that the correct library is used for the particular purpose? As we set the CMAKE_PREFIX_PATH where the library is saved, there may be an opportunity to set one or two CMake variables to the two different library paths with kind of a flag wich one should be used for debugging or release mode. Do you have any suggestions?</p>
<p>I found this thread on a popular website but I don’t know how to deal with this problem according to libtorch, because there is only one variable TORCH in the config files.<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/2209929/linking-different-libraries-for-debug-and-release-builds-in-cmake-on-windows" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/157525/gct" target="_blank" rel="nofollow noopener">
    <img alt="gct" src="https://www.gravatar.com/avatar/5820228fd7aefaba2a8bfb0503b9e2a8?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/2209929/linking-different-libraries-for-debug-and-release-builds-in-cmake-on-windows" target="_blank" rel="nofollow noopener">Linking different libraries for Debug and Release builds in Cmake on windows?</a>
</h4>

<div class="tags">
  <strong>c++, windows, cmake</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/157525/gct" target="_blank" rel="nofollow noopener">
    gct
  </a>
  on <a href="https://stackoverflow.com/questions/2209929/linking-different-libraries-for-debug-and-release-builds-in-cmake-on-windows" target="_blank" rel="nofollow noopener">07:47PM - 05 Feb 10 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

ntatsisk: <p>I have the same issue. Did you find a solution eventually?</p>
One hot a multi-dim tensor?

Ge0rges: <p>Hello all, I have an n*m tensor with float values on which I would like to perform a row-wise one-hot operation, that is every row should be a one hot vector.</p>
<p>What’s the most efficient way to accomplish this?</p>
<p>I’ve been doing:</p>
<pre><code class="lang-auto">
 _, indices = torch.max(my_matrix, 1)
binary_out = torch.nn.functional.one_hot(indices, num_classes=nclasses)
</code></pre>

ptrblck: <p>The code looks alright. You could alternatively use <code>torch.zeros(batch_size, num_classes).scatter_(1, indices, 1)</code>, but that shouldn’t be faster than <code>F.one_hot</code>.</p>
Which layer(s) is(are) being trained with new weights during this transfer learning?

SubhankarHalder: <p>Hello!<br>
I am using Squeezenet 1_1 Model pretrained model to train a custom dataset with 4 classes. I used the following commands to instantiate the model</p>
<pre><code class="lang-auto">MODEL = models.squeezenet1_1(pretrained=True)
MODEL.classifier[1] = nn.Conv2d(512, self.num_classes, kernel_size=(1, 1), stride=(1, 1))
</code></pre>
<p>Then, I write my training script, which is quite boilerplate. I do not freeze any layers.</p>
<p>So, is only the Conv2d layer getting trained with new weights? Do the MaxPool2d layers in Squeezenet have the old weights? Is there a command which we can use that gives the weights of the entire net before and after the training process?</p>

ptrblck: <p>Since you didn’t freeze any parameters, all should be trained as long as you pass them to the optimizers.<br>
Pooling layers do not have any parameters.</p>
<p>You can check the updated parameters by creating a deepcopy of the <code>state_dict</code> before training and compare it to the <code>state_dict</code> after training:</p>
<pre><code class="lang-python">state_dict_reference = copy.deepcopy(model.state_dict())

# training
# ...

# compare
state_dict = model.state_dict()
for key in state_dict:
    print(key, (state_dict_reference[key] - state_dict[key]).abs().max())
</code></pre>
Extract Only Batch Norm Parameters

wwaayyaaww: <p>Hi. I would like to extract all batch norm parameters from the pre-trained model?<br>
May I know if you have any proper way to form a list of batch norm parameters?<br>
This is due to the reason that I would like to retrain these parameters separately.</p>
<pre><code class="lang-auto"># for layer in resnet.modules():
#    if isinstance(layer,torch.nn.modules.batchnorm.BatchNorm2d): 
#        .....

</code></pre>

ptrblck: <p>Your code should work to check for all batchnorm layers in the model.<br>
Would you like to store the affine parameters only (<code>weight</code> and <code>bias</code>) or also the running estimates?<br>
Where are you stuck at the moment? Are you trying to append all wanted parameters to a list?</p>

wwaayyaaww: <p>Hi thank you for the prompt response.</p>
<p>I would like to append all those BN parameters to form a list for separate training with a different learning rate.</p>
<p>Is my implementation reasonable?</p>
<pre><code class="lang-auto">bn_paras = list()
for layer in resnet.modules():
    if isinstance(layer,torch.nn.modules.batchnorm.BatchNorm2d): 
        bn_paras.append(layer.parameters())
print(len(bn_paras))

</code></pre>

ptrblck: <p>Assuming you would like to pass this list to an optimizer, this approach should work:</p>
<pre><code class="lang-python">bn_paras = []
for layer in resnet.modules():
   if isinstance(layer,torch.nn.modules.batchnorm.BatchNorm2d): 
       bn_paras.extend(list(layer.parameters()))
       
optimizer = torch.optim.SGD(bn_paras, lr=1.)
</code></pre>

wwaayyaaww: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="81762">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">for layer in resnet.modules():
   if isinstance(layer,torch.nn.modules.batchnorm.BatchNorm2d): 
       bn_paras.extend(list(layer.parameters()))
</code></pre>
</blockquote>
</aside>
<p>Thank you very much~~ :")))</p>
Expected object of scalar type Float but got scalar type Half for argument #3 'weight'

rayleichen: <p>I’m using CrossEntropyLoss in my object detection model shown as below. Then I’m using APEX amp to enable “O2”-level auto mixed precision training.</p>
<pre><code class="lang-auto">class PredictionHead():
    def __init__(self, class_weights):
        super(PredictionHead, self).__init__()

        class_weights = torch.tensor(class_weights, requires_grad=False)
        self.loss_module_class = torch.nn.CrossEntropyLoss(weight=class_weights, reduction="none")

    def forward(self, x):
         # ...

    def loss(self, predictions, labels):
        return self.loss_module_class(predictions, labels)
</code></pre>
<p>But I get the error as follows. It seems that APEX amp converted “class_weights” into half floats, but CrossEntryLoss computation requires full floats.</p>
<pre><code class="lang-auto">  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py", line 932, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2317, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2115, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of scalar type Float but got scalar type Half for argument #3 'weight' in call to _thnn_nll_loss_forward
  In call to configurable 'train_model' (&lt;function train_model at 0x7f111c9a7f28&gt;) in scope 'train'
  In call to configurable 'train' (&lt;function _run_job at 0x7f1153a08730&gt;)
</code></pre>
<p>I also posted in APEX github issue: <a href="https://github.com/NVIDIA/apex/issues/837" rel="nofollow noopener">https://github.com/NVIDIA/apex/issues/837</a>. Any suggestion to skip conversion from full floats to half floats for CrossEntryLoss weights?</p>

rayleichen: <aside class="quote no-group" data-username="rayleichen" data-post="1" data-topic="81757">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rayleichen/48/36213_2.png" class="avatar"> rayleichen:</div>
<blockquote>
<p><code>nll_loss</code></p>
</blockquote>
</aside>
<p>It’s working fine now after I move CrossEntropyLoss from init() to loss().</p>
JIT for simple linear models?

mjoux: <p>What can be expected from JIT for simple linear models?</p>
<p>It’s possible I’m using JIT trace_module wrong, but here’s my model:</p>
<pre><code class="lang-python">model = nn.Sequential(
        nn.Linear(784, 200),
        nn.ReLU(),
        nn.Linear(200, 200),
        nn.ReLU(),
        nn.Linear(200, 10),
)
model = model.to(DEVICE)
model = torch.jit.trace_module(
        model, {'forward': torch.randn(BATCH_SIZE, 784, device=DEVICE)})

for inputs, targets in data:
  optimizer.zero_grad()
  outputs = model(inputs)
  loss = loss_fn(outputs, targets)
  loss.backward()
  optimizer.step()
</code></pre>
<p>A simple linear model for MNIST.</p>
<p>I have a few observations:</p>
<ol>
<li>From using Nsight systems, it’s quite clear that without JIT, the performance is quite bad. This is normal since the kernels are very small, so there’s nothing much for the CPU or GPU to do.</li>
<li>I’d expect JIT to be able to fuse quite a few of these kernels though. However, when using <code>trace</code> as shown above, the situation is exactly the same as without JIT. How come? Am I using JIT correctly here? And what kernels can I expect to be fused for this (simple) model ?</li>
</ol>

zheng-xq: <p>Thanks for bringing this to our attention.</p>
<p>Supporting matmul + relu on CPU, among other types of fusion, in a few months is on our roadmap. Then we might look into GPU. We will let you know if it is close to being ready.</p>

mjoux: <p>Thanks for the quick answer and yes, it would be good to know when this is ready <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> !</p>
The difference in usage between nn.ModuleList and python list

sirius27: <p>I can’t figure out what the purpose of nn.ModuleList is because as far as I know, list can be used in most scenarios I come up with.<br>
For example, for the newtork structure below,</p>
<pre><code class="lang-auto">	def __init__(self):
		super(MyNN, self).__init__()
		fcs = [nn.Sequential(
				 nn.Linear(fc_input_size, fc_hidden_size),
				 nn.ReLU(),
				 nn.Linear(fc_hidden_size,num_classes)
		) for fc_hidden_size in fc_hidden_sizes]
		#self.network = nn.ModuleList(fcs)
		self.network = fcs
</code></pre>
<p>I don’t see the difference between list and nn.ModuleList.<br>
Can anyone please give an example of where nn.ModuleList has advantage and explain it’s designed purpose?</p>

smth: <p>When you ask for <code>model.parameters()</code>, the parameters of layers inside <code>nn.ModuleList</code> will be returned.<br>
But if it is regular list, we dont look inside the list for <code>nn.Module</code> layers.</p>
<p>In your example, if you do:</p>
<p><code>list(MyNN().parameters())</code> then you will see that the weight of the nn.Linear in <code>self.network</code> will be missing.</p>

sirius27: <p>I tried what you suggested, and it’s indeed the case. Thanks a lot!</p>

trangham283: <p>I’m late to the party, but does this mean using <code>list</code> instead of <code>ModuleList</code> will be wrong? i.e. will the weights of my networks inside the list still be updated during training?</p>

InnovArul: <p>Yes the weights of the modules inside the python list  will not be updated in training, unless you manually add them to the list of parameters passed to the optimizer. Moreover,  even if you do that,  when you want to save the model parameters using <code>model.state_dict()</code>, the parameters of modules inside the python list won’t be saved.</p>

Abhishek_Kishore: <p>In Training a classifier example of pytorch 60 min blitz tutorial the modules are not added in Modulelist still the optimizer is able to access the parameters. Why is that?</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/is-it-mandatory-to-add-modules-to-modulelist-to-access-its-parameters/81622/2">here</a> with a potential answer.</p>
Normalize a vector to [0,1]

Shani_Gamrian: <p>How to normalize a vector so all it’s values would be between 0 and 1 ([0,1])?</p>

jpeg729: <p>This is one way, but I doubt it is what you wanted as you weren’t very specific.</p>
<pre><code class="lang-auto">min_v = torch.min(vector)
range_v = torch.max(vector) - min_v
if range_v &gt; 0:
    normalised = (vector - min) / range_v
else:
    normalised = torch.zeros(vector.size())</code></pre>

Shani_Gamrian: <p>I want to make the following division:<code>tensor_vec / tensor_vec.sum()</code> but when I do this I get:</p>
<pre><code class="lang-auto">RuntimeError: inconsistent tensor size at /opt/conda/conda-bld/pytorch_1501972792122/work/pytorch-0.1.12/torch/lib/TH/generic/THTensorMath.c:87
</code></pre>

jpeg729: <p>It looks like you have pytorch 0.1.12 installed. It may be time to upgrade.</p>

Shani_Gamrian: <p>You’re right, it works with PyTorch 0.3, isn’t there a way to make it work with 0.1.12?</p>

jpeg729: <p>I have no idea, I have only ever used pytorch 0.3+.</p>
<p>That said tensor_vec.sum() should output a single scalar value, so you shouldn’t get an inconsistent tensor size error unless either I have misunderstood what your code does or pytorch 0.1.12 has a bug.</p>

ptrblck: <p>Broadcasting wasn’t available in version <code>0.1.12</code>.<br>
You could try:</p>
<pre><code class="lang-auto">tensor_vec = tensor_vec / tensor_vec.sum(0).expand_as(tensor_vec)
</code></pre>

mutaku: <p>This is great! One thing I did was add something to handle tensors with negative values (e.g. one that had been 0 mean scaled at some point):</p>
<pre><code class="lang-auto"># Push positive before scaling:
tensor_vec.add(tensor_vec.min() * -1)

</code></pre>

Ambareesh_S_J: <p>Major caveat : If tensor_vec has a column with all 0’s, the operation will make that column all NaNs</p>
<p>So a checker, would be ideal:</p>
<pre><code class="lang-auto">x = x/x.sum(0).expand_as(x) 
x[torch.isnan(x)]=0
</code></pre>

kshitijpatil09: <p>How to scale [0,1] image tensors for matplotlib? I tried min-max scaling and it didn’t retain my pre-processing effect. Applying sigmoid looks like working, is this the right way to do so?</p>
What is the meaning of **?

chetan06: <p>I am currently working on wheat detection challenge and was going through a notebook. And read something that am not getting</p>
<pre><code class="lang-auto">class WheatTestDataset(Dataset):

    def __init__(self, dataframe, image_dir, transforms=None):
        super().__init__()

        self.image_ids = dataframe['image_id'].unique()
        self.df = dataframe
        self.image_dir = image_dir
        self.transforms = transforms

    def __getitem__(self, index: int):

        image_id = self.image_ids[index]
        records = self.df[self.df['image_id'] == image_id]

        image = cv2.imread(f'{self.image_dir}/{image_id}.jpg', cv2.IMREAD_COLOR)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB).astype(np.float32)
        image /= 255.0

        if self.transforms:
            sample = {
                'image': image,
            }
            sample = self.transforms(**sample)
            image = sample['image']

        return image, image_id

    def __len__(self) -&gt; int:
        return self.image_ids.shape[0]
</code></pre>
<p>In the below line why the author had written ** ?</p>
<pre><code class="lang-auto">sample = self.transforms(**sample)
</code></pre>
<p>The transform function passed to the dataset is:</p>
<pre><code class="lang-auto">def get_test_transform():
    return A.Compose([
          ToTensorV2(p=1.0)
    ])
</code></pre>

chetan06: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  and others please help.</p>

ptrblck: <p>You can pass arbitrary numbers of arguments to a function using <code>**kwargs</code> as explained <a href="https://stackoverflow.com/questions/36901/what-does-double-star-asterisk-and-star-asterisk-do-for-parameters">here</a>.</p>
After colab crashes with 12GB ram, not asking for 25GB ram.Simply restarting

Sushmitha_Katti: <p>Hey all,<br>
Usually, colab allocates us 25GB ram when we crash 12GB ram. But in my case, it is not asking or allocating 25GB ram.</p>
<p>a = []</p>
<p>while (1):</p>
<p>a.append(1)</p>
<p>I ran this program to crash. It is crashing but it says, “colab crashed after using all ram, see runtime logs” and then restarts.</p>
<p>what might be the problem here?</p>

ptrblck: <p>This seems to be a Colab issue and maybe this “hack” of running OOM to get more RAM doesn’t work anymore?</p>

Sushmitha_Katti: <p>Yeah, sometimes it works but not every time.<br>
One hack we can do is we can change the backend config of colab<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3aef0544531ffae166e076139c2c3def463791a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a3aef0544531ffae166e076139c2c3def463791a" title="colab"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3aef0544531ffae166e076139c2c3def463791a_2_657x500.jpeg" alt="colab" data-base62-sha1="nm0DczAoHJ21owGcp9WyMjdzSRk" width="657" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3aef0544531ffae166e076139c2c3def463791a_2_657x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3aef0544531ffae166e076139c2c3def463791a_2_985x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3aef0544531ffae166e076139c2c3def463791a.jpeg 2x" data-dominant-color="F2F1F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">colab</span><span class="informations">1080×821 79.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> .</p>
<p><strong>Here “hm” represents ‘high memory’. So we can add machine_shape field to get high ram</strong></p>
How to lower the dimension of the tensor in pytorch?

Sterben: <p>I have tensor <code>[1, 3, 256, 256, 3]</code>. I want to reduce it to <code>[1, 3, 256, 256]</code>. How can i do this ?<br>
Thanks!</p>

ptrblck: <p>How would you like to reduce the last dimension?<br>
In a simple way you could just call <code>x.mean(4)</code> or another arithmetic operation.</p>

Sterben: <p>I could bring the tensor to the form [1, 3, 1, 256, 256], in numpy I would be able to reduce the dimension of np.squeeze and add another axis to the 0 position, but can I do it in pytorch?</p>

ptrblck: <p>I’m not sure, which dimension you would like to squeeze or add, but PyTorch has also the method <code>squeeze()</code> and <code>unsqueeze()</code> to remove and add dimensions, respectively.</p>

solsol: <p>is there any way to reduce [1, 3, 256, 256, 3] to  [1, 3, 256, 128, 3] ?</p>

ptrblck: <p>How would you like to reduce this dimension? Would you like to calculate the sum (or mean) of two neighboring values or just slice the tensor?</p>

solsol: <p>I solved it by using mean.<br>
[1, 3, 256, 256, 3].view(1, 3, 2, 256*128, 3)<br>
and then apply mean on third dimension.</p>

mathematics: <p>I have lowered torch dimension of shape<code> torch.zeros([16, 3, 32, 32])</code> into<code> [32,32,3]</code> numpy array by</p>
<pre><code class="lang-auto">    img = image.squeeze(0).detach().cpu().numpy()
    print(img.shape) #(16, 3, 32, 32)
    img = img[0, :,:,:].transpose(1,2,0)
    print(img.shape # (32, 32, 3)
</code></pre>
Sorting 4D tensor (B, C,w,h) along C axis

Abe: <p>Let’s say we have 4D tensor (B, C,w,h). I want to sort this tensor based on the average value of the channels avg_pool(wxh). The average of wxh used to sort the tensor along the C axis. Would you suggest the efficient implementation of this without needing a loop?</p>
<p>For example, using torch.sort(tensor, dim=1) affects the rows of the 2D tensor (wxh). But I don’t want to swap any rows or columns, I only need to sort the whole 2D matrix based on their averaged_pooled values.</p>