akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
LR parameter in optimizer larger than max_lr in OneCycleLr

cthnguyen: <p>Hello,</p>
<p>I made a little mistake in my code (I think at least …), where I put the following :</p>
<pre><code class="lang-auto"># Optimizer
wd = 0.01
optimizer = optim.Adam(model.parameters(), lr=10e-4, weight_decay=wd)

# LR scheduler
scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-4, steps_per_epoch=len(train_loader), epochs=epochs, pct_start=0.3) # only available on pytorch v1.3.0
</code></pre>
<p>Notice that in the optimizer I put a learning rate of 10e-4, whereas I put the max_lr on OneCycleLR to 1e-4. So in this situation what does it do? I am a little bit confuse, could anyone light me up please?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p><code>max_lr</code> from the scheduler should limit the learning rate from the optimizer.<br>
You could print out the learning rate for all <code>optimizer.param_groups</code> to check this effect.</p>

cthnguyen: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> ! I’m still new to Pytorch so I didn’t thought of checking it. You are right, the <code>max_lr</code> limit the learning rate.</p>
Reviewing statistical results for Convolutional seq2seq model

Aiman_Mutasem-bellh: <p>Dear friends,</p>
<p>I’m a little worried about my statistical results after training model. I’m working on some NLP task and the test set contain 50k word. the final result is :</p>
<pre><code>Test Loss: 0.000 | Test accuracy: 0.998 | Test precision: 0.891 | Train recall: 0.891
Test F1: 0.891
</code></pre>
<p>I have checked the code, every thing is working well, but I think this is little high score. I calculated the loss like :</p>
<pre><code class="lang-auto">        for i, batch in enumerate(iterator):

            src = batch.src
            trg = batch.trg
             
            output, _ = model(src, trg[:,:-1)
            output = output.contiguous().view(-1, output.shape[-1])
            trg = trg[:,1:].contiguous().view(-1)

            loss = criterion(output, trg)

            epoch_loss += loss.item()
</code></pre>
<p>Kindly, anyone can explain this, and how to recheck it?</p>

ptrblck: <p>I’m not sure, how you’re calculating the metrics, but if you have some doubts about these high scores, I would recommend to check for data leaks.<br>
Make sure you are not using the test data for training in any sense (e.g. early stopping, hyper parameter search etc.).</p>

Aiman_Mutasem-bellh: <p>Thank you Mr. <a class="mention" href="/u/ptrblck">@ptrblck</a>, I will check it again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why is the memory of GPU enlarge every epoch

sofzh: <p>I trained a network, but I find the memory it occupies in GPU enlarges every epoch.<br>
In my program, every epoch includes validate and train procedure. It first validates and then trains.<br>
When the program runs, the memory it occupies enlarges by epoch until the GPU-util is all 100%. It stop to preceed, but the program is still on with memory occupied in GPUs.<br>
How does it go?<br>
Can anyone helps?<br>
Thank u very much!</p>

JuanFMontesinos: <p>Can you post the training code?</p>
Parallelize histogram calculation?

dianlujitao: <p>It’s easy to calculate histogram with <code>torch.histc</code>, but it’s designed to calculate for only one tensor.</p>
<p>To be precise, I have a large tensor <code>(N, m)</code> of N length m sub-tensors, or more flexible indexing <code>(N1, N2, ..., Ni, m1, m2, ..., mj)</code> of shape $\prod_j m_j$ tensors, and I need to calculate histograms of the separate $\prod_i N_i$ sub-tensors with the same number of bins, i.e. the output shape should be <code>(N1, N2, ..., Ni, bins)</code>.</p>
<p>Since <code>torch.histc</code> doesn’t provide <code>start_dim</code> or similar options, is it possible to do this in pytorch w/o nested for loops which is extremely slow?</p>
<p>Thanks in advance!</p>
Awkward behavior of loss

Anuj_Chopra: <p>I have written my own implementation of GRU. Which is</p>
<pre><code class="lang-auto">class new_meaning(NN.Module):
    def __init__(self,input_size, hidden_size, output_size):
        super(new_meaning,self).__init__()
        self.hn = NN.Linear(hidden_size, hidden_size)
        self.inp = NN.Linear(input_size, input_size)
        self.reset_gate = NN.Linear(input_size + hidden_size, hidden_size )
        self.update_gate = NN.Linear(input_size + hidden_size, hidden_size )
        self.tan_h = NN.Linear(input_size + hidden_size, hidden_size )
        self.linear = NN.Linear(hidden_size, hidden_size)
        self.conclude = NN.Linear(hidden_size, vector_size)

    def forward(self,inputs, first_hid, lengths = -1):
        this_hid = first_hid
        gru_out = []
        for i in range(21):
            this_hid = self.gru_forward(inputs[:,i,:] ,this_hid)
            gru_out.append(this_hid)
        gru_out = torch.stack(gru_out, dim = 1)
        last_ones = torch.stack([gru_out[i,j,:] for (i,j) in lengths])
        final_out = self.linear(last_ones)
        conclusion = self.conclude(final_out)
        return conclusion
        
    def gru_forward(self, inputs, hidden):
        concat = torch.cat((inputs,hidden), dim = 1)
        reset = torch.sigmoid(self.reset_gate(concat))
        update = torch.sigmoid(self.update_gate(concat))
        temp_hid = reset*self.hn(hidden)
        new_hid = torch.cat((temp_hid, self.inp(inputs)), dim = 1)
        h_tilda = torch.tanh(self.tan_h(new_hid))
        next_hid_1 = (1-update)*hidden
        next_hid_2 = update*h_tilda
        return next_hid_1 + next_hid_2

    def init_hidden(self, batch_size = 1):
        return torch.zeros(batch_size,self.hidden_size) 
</code></pre>
<p>The problem I am facing is that it is not converging while using default GRU of Pytorch is converging. Also the losses initially seem to converge and then start behaving erratically.<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/3079e4547eadb53780c14745cbd02e8871069406.png" alt="losses_gru" data-base62-sha1="6UQ4oasgATr85utrVZivUqo5eGa" width="380" height="248"><br>
I have capped the losses to 0.05. And the data is randomly drawn from csv file. The spike in losses however is always around same iteration.</p>
Turning images into vectors using resnet

Jordan_Howell: <p>Hello,</p>
<p>I would like to take my images and turn them into vectors consisting of 1000 features.  I was thinking of using resnet50 to do this.  Can I run a model through without a loss criterion?  Is that the proper way to do what I intend to do?</p>
<p>Is there a good way to append those vectors to a series or numpy array?</p>

zimmer550: <p>I don’t know for what exact task you want to extract a feature vector for but, in theory, you can obviously use a pre-trained resnet to extract a feature vector of any size from an image. Keep in mind that such pre-trained resnets were specifically trained for image classification and so they will extract features with the sole purpose of image classification. However, you can easily circumvent this by doing transfer learning and re-training the last few fully connected layers according to your task.</p>

Jordan_Howell: <p>Thank you. I would like to join these features with tabular data to predict an ultimate binary outcome, in the same light as the Uber Ludwig model.  Would I just run my model without calculating a loss?</p>

zimmer550: <p>Up to you and your understanding of the problem. Run some experiments with and without loss and compare the results</p>

Jordan_Howell: <p>Fair enough. Thank you.</p>
Which class is the first?

Jordan_Howell: <p>When I run my data through a reset model for a binary classification, it returns the probability for each prediction (as the final node is using a softmax).  Which is the first number of the binary probability returned, the 0 or 1 class?</p>

ptrblck: <p>Based on your question I assume you are using two output neurons for the binary classification task.<br>
If that’s the case, your target basically defines which output neuron represents which class.<br>
E.g. if your target is <code>0</code> for the current sample, <code>output[0]</code> will represent the logit (or probability) for this class.</p>
<p>I’m not sure, which loss function you are using, but note that <code>nn.CrossEntropyLoss</code> expects logits (no softmax at the end), while <code>nn.NLLLoss</code> expects log probailities (<code>log_softmax</code> at the end).</p>
<p>Alternatively, you could also use a single output neuron and apply <code>nn.BCE(WithLogits)Loss</code> as the loss function.</p>

Jordan_Howell: <p>So I’m wrong to have a LogSoftmax at the end and use CrossEntropyLoss?</p>

ptrblck: <p>Yes, since internally <code>nn.CrossEntropyLoss</code> will apply <code>F.log_softmax</code> and <code>nn.NLLLoss</code> (<a href="https://github.com/pytorch/pytorch/blob/974702fba0a5309821d7c961edbd2cb16939de14/torch/nn/functional.py#L1967" rel="nofollow noopener">line of code</a>),<br>
so you would have to chose between:</p>
<ul>
<li>raw logits (no activation function) + <code>nn.CrossEntropyLoss</code>
</li>
<li>
<code>F.log_softmax</code> + <code>nn.NLLLoss</code>
</li>
</ul>
How do I make a custom layer that will be used only during prediction stage?

jsn: <p>During the prediction of a trained neural network, I want to add new custom final layer and get its output. How do I do that?</p>
<p>I created a custom layer with <code>self.training=False</code> and added it as the last layer of my network. But, that layer is still called during the forward pass during training. How do I do this right?</p>

JuanFMontesinos: <p>During the forward you can use that condition:<br>
<code>if self.training do_this else dont</code> self.training is a default flag in nn.Module.</p>

jsn: <p><a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> Thanks. I misunderstood what <code>self.training</code> was for. Yes, that seems to work. But, if I’m training like this it won’t work :</p>
<pre><code class="lang-auto">            output = model(x_train)
            loss_value = loss(output, y_train)

            # backprop
            model.zero_grad()
            loss_value.backward()
            model.update_parameters(learning_rate)
</code></pre>

JuanFMontesinos: <p>What do you mean by “like this”<br>
If you set model.eval() it won’t call it, else it will.</p>
Cuda error: GPU not found

sungengyi: <p>I’m running pytorch with Anaconda Spyder with cuda. This is a windows 10 computer with a GeForce 1050 ti locally but when  i run <code>torch.cuda.device_count()</code> console always returns 0. I installed various of cudas like <code>conda install -c anaconda cudatoolkit</code> and other possible versions. However I’m getting the same error always. Is there any possibly solutions I can look into?<br>
Thank you !<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a31b838b5261ecbef726fe92828f09f92e9c79cc.png" alt="Annotation%202019-11-04%20100531" data-base62-sha1="ngUM6gs1X8wHooHwfsNiy1jbZeY" width="677" height="264"></p>

sungengyi: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73e0cbc48c335dc2e8e408b0f9fa12badfcd359a.png" data-download-href="https://discuss.pytorch.org/uploads/default/73e0cbc48c335dc2e8e408b0f9fa12badfcd359a" title="cudas"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73e0cbc48c335dc2e8e408b0f9fa12badfcd359a.png" alt="cudas" data-base62-sha1="gx6wZW38tf6VGpr54QNYGdoUKKu" width="634" height="500" data-dominant-color="F4F9F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cudas</span><span class="informations">796×627 31.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Here’s the error trace! The code works fine on Kaggle notebook using GPU, so I’m noy sure why I can’t use my local GPU.</p>

Nikronic: <p>Hi,</p>
<p>The PyTorch version you have downloaded is incompatible with GPU. Uninstall current version fully and install it again using available commands in PyTorch’s website.<br>
But if you are sure about the package, update your Nvidia and cuda driver . You can still check consistent versions in the install page.</p>

sungengyi: <p>Thank you! I will try that !</p>

sungengyi: <p>I have a GeForce 1050 ti, which pytorch/cuda version should i use?</p>

Nikronic: <p>I do not have Nvidia GPU on my own to make sure about solutions.<br>
Could you try this:</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
</code></pre>
<p>But before this make sure you have uninstalled any other previous versions.</p>
Use `torchvision.models` pretrained models but input of smaller sizes

Guanqun_Yang: <h1>Problem</h1>
<p>The models provided in <code>torchvision.models</code> were trained on ImageNet so the size of image is <code>(3, 224, 224)</code>. However, I have a small dataset (a custom dataset with about 20 thousand images) which is of smaller size.</p>
<p>I am wondering if there is anyway I could still use the pre-trained weights with those smaller images. There are two directions in my mind.</p>
<ul>
<li>Modify the first layer and retrain that layer from scratch so that the the sizes of later layers are compatible with pre-trained weights.</li>
<li>Somehow augment the original images (perhaps some strategic padding) so that the size is made <code>(224, 224)</code>.</li>
</ul>
<p>However, both of them seems to be a long shot when moving into action.</p>
<p>Therefore, does anyone have encountered similar circumstances? Thank you in advance!</p>

albanD: <p>Hi,</p>
<p>I think the second one using classical resizing methods (with bilinear sampling for example) can be a good approach if the images in your new dataset are not too far from 224x224. Doc <a href="https://pytorch.org/docs/stable/torchvision/transforms.html?highlight=resize#torchvision.transforms.Resize" rel="nofollow noopener">here</a>.</p>
Torchvision : Normalise all image of a batch

Ierezell: <p>Hi,</p>
<p>I’m training a model to reproduce images (in [0, 1]) but they are normalized (for vgg).</p>
<p>Should the output of my network be in [0,1] and then normalized as the ground truth ?<br>
If so how can I apply torchvision.transforms.Normalize() on a whole batch of images ?</p>
<p>Else I just shouldn’t have activation at the output ?<br>
Normalisation brings [0,1] to [-2.117904 2.64] so no sigmoid or relu posible.</p>
<p>Thanks a lot for your help.</p>
Choice of mean and variance when normalizing the dataset

Guanqun_Yang: <h2>Problem</h2>
<p>Normalizing dataset is a common component of machine learning before doing any downstream task. However, when I learn the <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">tutorial</a> in PyTorch, the author used mysterious mean and variance value for different channels of RGB images, i.e. 0.485, 0.456 and 0.406 for mean and 0.229, 0.224, 0.225 for variance (see code below).</p>
<p>I am not sure how those somehow magic numbers come about. Do they come from some statistics of the training dataset which is not disclosed in tutorial. Or maybe they are heuristics that prove to be effective in computer vision literature?</p>
<pre><code class="lang-python"># Data augmentation and normalization for training
# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

</code></pre>

JuanFMontesinos: <p>They are image net statistics. You ideally should compute statistics for each dataset you use.</p>

Guanqun_Yang: <p>Thank you for your timely response!</p>
<p>I found the origin of those magic numbers in <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="noopener nofollow ugc"><code>torchvision.models</code> doc</a>, which, as you said, should be statistics of ImageNet.</p>
<blockquote>
<p>All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using  <code>mean = [0.485, 0.456, 0.406]</code>  and  <code>std = [0.229, 0.224, 0.225]</code> …</p>
</blockquote>
<p>However, it also said as long as users would like to use those pretrained models, they <strong>must</strong> normalize their images in this way. I think this makes sense since those models were trained on data scaled according to those specified <code>mean</code> and <code>std</code>.</p>
nn.BCEWithLogitsLoss() can't accept one-hot target

sr_frost: <p>Hi,<br>
My training loop looks something like this</p>
<pre><code class="lang-auto">loss_fn = nn.BCEWithLogitsLoss()
    for epoch in range(1, num_epochs+1):
        model.train()
        for X, y in train_loader:
            X, y = X.to(device), y.to(device)
            y_hot = F.one_hot(y, num_classes)
            output = model(X)
            optimizer.zero_grad()
            loss = loss_fn(output, y_hot)
            loss.backward()
            optimizer.step()
</code></pre>
<p>I am transforming the target to be a one-hot vector as expected by the BCEWithLogitsLoss().<br>
But, I am still getting the following error, pointing at the loss function:</p>
<pre><code class="lang-auto">RuntimeError: result type Float can't be cast to the desired output type Long
</code></pre>
<p>I can’t really understand the error message. What am I doing wrong?</p>

albanD: <p>I think the problem is that <code>y_hot</code> is of type long and the BCE loss expects a Tensor of type float.<br>
Can you try converting y_hot with <code>y_hot = y_hot.type_as(output)</code>.</p>

sr_frost: <p>Thanks. It worked.</p>
<p>But, since it’s pretty common for BCELoss to have one-hot target, wouldn’t it be better if it could also handle long type?</p>

albanD: <p>Hi,</p>
<p>The general policy we have in pytorch is that we avoid doing any copy that are hidden from the user. (cpu/gpu or different types).<br>
Even though you could argue that this can make the user’s code more verbose. It usually allows the user to have a better control on what happens in its code.</p>
Batch-norm with Variable batch size (where sometimes the batch can be of size 1 too))

zimmer550: <p>So, my question is basically a follow-p of this:</p><aside class="quote quote-modified" data-post="1" data-topic="59946">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/bbe5ce/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/variable-batch-size-during-training/59946">Variable batch size during training</a> 
  </div>
  <blockquote>
    Hi. I am training an RNN encoder-decoder to make predictions of the trajectory of agents. I am using the KITTI dataset. The training dataset consists of 21 image folders and each folder has a variable number of images. So, this means that the batch size changes during training, for example: 
A batch size of one, when training an encoder, means one sequence. Keeping that in mind, let’s say I take as input 5 images and, while decoding, also works on 5 images. So, that is a total of 10 images. So, …
  </blockquote>
</aside>

<p>Is batch-norm suitable for my situation where the batch size can fluctuate between 1 and 10? I know that there is no batchnorm for batch size of 1. So, is that a hard no for my situation or is there something I could do?</p>

SimonW: <p>with that low batch size, you are probably better just not use batch norm at all.</p>
Why init weight function disappear after put net to cuda

mysql403: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f74a33c751d3a106e2521dffaa4ba4f6701d2a30.png" data-download-href="https://discuss.pytorch.org/uploads/default/f74a33c751d3a106e2521dffaa4ba4f6701d2a30" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f74a33c751d3a106e2521dffaa4ba4f6701d2a30_2_690x105.png" alt="image" data-base62-sha1="zhCUAQbQMeSlLsyNGqV5ay8dsoo" width="690" height="105" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f74a33c751d3a106e2521dffaa4ba4f6701d2a30_2_690x105.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f74a33c751d3a106e2521dffaa4ba4f6701d2a30_2_1035x157.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f74a33c751d3a106e2521dffaa4ba4f6701d2a30_2_1380x210.png 2x" data-dominant-color="484445"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1711×262 69.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>code is here</p>
<pre><code class="lang-auto">    elif args.model == 'efficientnetb4':
        net = smp.Unet('efficientnet-b4', classes=4, activation=None)
        print('111:', dir(net))
        net.initialize()
        print('load smp efficientnet-b4')

    #print('net:', net)

    log_dir = os.path.join(args.out_dir, 'log')
    if not os.path.exists(log_dir):
        os.makedirs(log_dir)
    writer = SummaryWriter(log_dir)
    writer.add_graph(net, torch.rand(1, n_channels, args.imgh, args.imgw))

    if args.cuda:
        net = torch.nn.DataParallel(net)
        net = net.cuda()

    print('222:', dir(net))
</code></pre>
<pre><code class="lang-auto">class Model(nn.Module):

    def __init__(self):
        super().__init__()

    def initialize(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_uniform_(m.weight, mode='fan_in', nonlinearity='relu')
                if m.bias is not None:
                    nn.init.constant_(m.bias, 0)
            elif isinstance(m, nn.BatchNorm2d):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)

</code></pre>
<p>The initialize is defined in model class, and Unet is inherited from EncoderDecoder, and EncoderDecoder is inherited from model class. My question is why the initialize function is disappear after put net to cuda multi-gpu?</p>

JuanFMontesinos: <p>Because DataParallel wraps Model. It doesn’t modify the object itself.<br>
You can call model.module.initialize.<br>
Namely, inside a DataParallel object your model is instantiated at object.module</p>

mysql403: <p>Thanks, nice answer, got it</p>
Use backward twice, and apply retain_graph=True for the first backward, but the evaluation result is always the same

Zichao_Li: <p>Use backward twice, and apply retain_graph=True for the first backward, but the evaluation result is always the same.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ec1fe0774c5fa1a9f6c770da3faa8218b341cad.png" data-download-href="https://discuss.pytorch.org/uploads/default/3ec1fe0774c5fa1a9f6c770da3faa8218b341cad" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ec1fe0774c5fa1a9f6c770da3faa8218b341cad.png" alt="image" data-base62-sha1="8XbefboKz1lHSbAhhmdASlJn2nr" width="474" height="500" data-dominant-color="3C3F2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">564×594 39.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>Could you  give more information please about your code and what do these prints correspond to?</p>

Zichao_Li: <p>Of course. In one epoch, I want to have the two backward of the network, In the first backward, I can get the gradient of every layer which i want to use to update the parameters. And I have the second background with a different cost function.</p>

albanD: <p>Could you give a small code sample (30-40 lines) that reproduces what you see? So that I can test it on my machine?</p>
For every element in one tensor, find the index in another tensor

as1949078: <p>For every element in one tensor, I want to find the index in another tensor. Currently, I am using the following method:</p>
<pre><code class="lang-auto">x = torch.LongTensor([1,2,3,4])
y = torch.LongTensor([2,3,4,1])

# y is the query
def retrieve_indices(x, y):
    return torch.nonzero(y[:, None] == x[None, :])[:, 1]

print(retrieve_indices(x,y))  # tensor([1, 2, 3, 0])
</code></pre>
<p>For example, the first element 1 in the resuslt indicates the index of y[0] in x is 1.</p>
<p>This method is based on one implementation for Numpy mentioned in <a href="https://stackoverflow.com/a/40449296" rel="nofollow noopener">Stackoverflow</a>. But this method is memory intensive.</p>
<p>Another answer in the same Stackoverflow post has a better solution for Numpy:</p>
<pre><code class="lang-auto">x = np.array([1,2,3,4])
y = np.array([2,3,4,1])

index = np.argsort(x) 

sorted_x = x[index] 

sorted_index = np.searchsorted(sorted_x, y)

yindex = np.take(index, sorted_index)
print(yindex)
</code></pre>
<p>But I could not find PyTorch equivalent of the Numpy method <code>searchsorted</code>. Someone has implemented <a href="https://github.com/aliutkus/torchsearchsorted" rel="nofollow noopener">searchsorted for PyTorch</a>. However my program is required to rely on official PyTorch and Numpy only.</p>
<p>Is there a better way to achieve this task? Or the <code>retrieve_indices</code> method is the best solution for PyTorch so far (Note that <code>retrieve_indices</code> is only for comparing two tensors. )?  Let’s assume every element in y is in x. Besides, x and y have the same shape: (batch_size, number of elements).</p>

SimonW: <pre><code class="lang-auto">y.argsort().argsort()[x.argsort()]
</code></pre>

as1949078: <p><a class="mention" href="/u/simonw">@SimonW</a> Thanks. Unfortunately, it does not work if the inputs are:</p>
<pre><code class="lang-auto">x = torch.LongTensor([1,5,3,4])
y = torch.LongTensor([5,3,4,1])
</code></pre>
<p>The correct output is <code>tensor([1, 2, 3, 0])</code>, but your method outputs <code>tensor([3, 2, 0, 1])</code>.</p>

SimonW: <p>Oh sorry, I looked at the requirement wrong. It should be <code>x.argsort()[y.argsort().argsort()]</code></p>
RuntimeError: ONNX export failed: Couldn't export operator aten::norm

Zicheng_Ren: <p>Hi, I ran into this issue while exporting to onnx model. Has anyone run into this issue before?</p>
<p>/home/zichengr/.local/lib/python3.5/site-packages/torch/onnx/utils.py:365: UserWarning: ONNX export failed on ATen operator norm because torch.onnx.symbolic.norm does not exist<br>
.format(op_name, op_name))</p>
<p>RuntimeError: ONNX export failed: Couldn’t export operator aten::norm</p>

cfer8395: <p>I had the same problem, and couldn’t find any answer to it. So I checked out the torch.onnx docs, and saw that onnx already has an operator for L-normalization (LpNormalization). Then I edited the <strong>torch/onnx/symbolic.py</strong> file and added a mapping from tensor norm to that operator:</p>
<pre><code class="lang-auto">def norm(g, self, p, dim, keepdim):
    return g.op("LpNormalization", self, p_i=p, axis_i=dim)
</code></pre>
<p>Additionally, I had to replace this:<br>
<code>x = F.normalize(x, p=2, dim=1)</code></p>
<p>with this:<br>
<code>x = x / (x.norm(p=2, dim=1, keepdim=True) + 1e-12).expand_as(x)</code></p>
<p>to avoid having to export <strong>clamp_min</strong> as well.<br>
That seemed to work for me, hope it helps!</p>

Zicheng_Ren: <p>Amazing job! It worked for me too. Thanks!</p>

Zicheng_Ren: <p>Hi,<br>
Can you tell me the location of the LpNormalization ? I don’t seem to find it.</p>
<p>Thanks.</p>

PrinceP: <p>where is the torch/onxx/symbolic.py in torch python library</p>

jiangx25: <p>RuntimeError: ONNX export failed: Couldn’t export operator aten::softmax</p>

stoneyang: <p>Hi, <a class="mention" href="/u/cfer8395">@cfer8395</a> could share further info? Like, the version of pytorch? Thanks!</p>
How to export variable of functional during torch->onnx?

stoneyang: <p>As the title, I’ve trained an embedding function (e.g. inception-v1) with the last layer of an fc (e.g., 2048-dim vector as its output). In deployment, I want to append an L2-normalize operation after the fc layer and export the appended model (inception-v1 + fc2048 + l2norm) to onnx format? Without considering deployment in the form of trt, I know it is just adding a functional in the forward function of the model class. But in exporting to onnx format, it seems that the output name of a module is needed.</p>

stoneyang: <p>I’ve read <a href="https://discuss.pytorch.org/t/whats-the-difference-between-nn-relu-vs-f-relu/27599/2">this thread</a>, and the problem seems to become to find a torch module equivalent of <code>nn.functional.normalize()</code>, correct me if I was wrong.</p>

stoneyang: <p>And <a href="https://discuss.pytorch.org/t/runtimeerror-onnx-export-failed-couldnt-export-operator-aten-norm/18634">this thread</a> seems useful. But I still can’t find the so call <code>LpNormalization</code> mentioned by cfer8395.</p>
Variable batch size during training

zimmer550: <p>Hi. I am training an RNN encoder-decoder to make predictions of the trajectory of agents. I am using the KITTI dataset. The training dataset consists of 21 image folders and each folder has a variable number of images. So, this means that the batch size changes during training, for example:</p>
<p>A batch size of one, when training an encoder, means one sequence. Keeping that in mind, let’s say I take as input 5 images and, while decoding, also works on 5 images. So, that is a total of 10 images. So, a single sequence consists of 10 images, hence, a batch size of one is the same as 10 images in this case. Now, I have a folder of 30 images, so I can work on 3 sequences, hence my batch size is 3 for this folder. However, the next folder has 50 images, so that means the batch size becomes 5.</p>
<p>So, I guess my question is whether this is okay or not? Is there any harm while training like this?</p>

JuanFMontesinos: <p>Well, from the backprop perspective and theoretical perspective there is no problem with increasing the batch size. However there are some practical issues:</p>
<ul>
<li>Normalization depends on batch statistics. The bigger the batch size is, the more general statistics are supposed to be. You should wonder if normalization will properly generalize.</li>
<li>Additionally, there exist a relationship (empirically shown) between batch size and learning rate <a href="https://miguel-data-sc.github.io/2017-11-05-first/" rel="nofollow noopener">Explanation</a> in a log-scale. If your batch size increases a lot your training may require LR adjustments. In the same way you can find this paper <a href="https://arxiv.org/abs/1711.00489" rel="nofollow noopener">Don’t Decay the Learning Rate, Increase the Batch Size</a> which also gets into this problem.</li>
</ul>

zimmer550: <p>Yes but the batch size is variable, so while training it may increase (I start using images from a folder that has 100 images) or decrease (I once again come across a folder with 30 images). Is such fluctuation okay? Or is it only a problem when the batch size keeps on increasing while training?</p>

JuanFMontesinos: <p>Well, I guess what you expect is a binary answer, yes or no. I unfortunately cannot provide that. What I’m trying to say is something like, variable batch size may be stable up to some point. As it is dataset-dependent and architecture-dependent I cannot infer what is going to happen in your case. If such fluctuation is not very big it should be ok. But it is for sure a factor to keep an eye on if you training doesn’t work as expected.</p>
Is model in my main function updated when I call model.backward() inside another function?

coyote: <p>Hi,</p>
<p>I create my model within my main function. Then I send the model to training function and update model there. In that case, should I return the model back from the training function to use the trained model inside my  main function ?</p>
<p>Here is a small snippet to describe what I’ve just asked:</p>
<blockquote>
<p>def main():<br>
model = BertForSequenceClassification(‘bert-base-cased’)<br>
tokenizer = BertTokenizer(‘bert-base-cased’)<br>
data  = get_some_data()<br>
train(model,tokenizer,data)<br>
<span class="hashtag-raw">#So</span> at this point when I use model to make predictions, is this a trained model or<br>
<span class="hashtag-raw">#was</span> train model  just lost since I did not return it form train funciton ? (i.e. model = train(model…))</p>
</blockquote>
<pre><code class="lang-auto">def train(model,data):
    optimizer_grouped_parameters = [
        {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': args['weight_decay']},
        {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
        ]

    optimizer = AdamW(optimizer_grouped_parameters,lr=some_lr)
    scheduler = WarmupLinearSchedule(optimizer, warmup_steps=args['warmup_steps'], t_total=t_total)
   model.train()
   for x,y in data:
      outputs = model(x)
      loss = outputs[0]
      loss.backward()
</code></pre>

JuanFMontesinos: <p>If I’m not wrong from a python perspective you don’t need to return it but I would consider it a good practice.<br>
If you call train(model,data) what do you loose for doing model=train(model,data). It’s more clean, more concise and more readable. You neither need to pass model to the function as it is an outer variable but you still pass it. Furthermore that behavior depends on which object are you passing to the function, if you pass an integer it would behave the other way around.</p>
Put a tensor into a gathered tensor (inverse permutation)

deniz: <pre><code>&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.arange(0, 3).view(1,3,1).expand(2,3,4) + 1
&gt;&gt;&gt; permutation_indexes = torch.tensor([
&gt;&gt;&gt;     [[0,0,1,1], 
&gt;&gt;&gt;      [1,2,0,2], 
&gt;&gt;&gt;      [2,1,2,0]], 
&gt;&gt;&gt;     [[2,2,0,0], 
&gt;&gt;&gt;      [0,1,1,2], 
&gt;&gt;&gt;      [1,0,2,1]]
&gt;&gt;&gt; ])
&gt;&gt;&gt; torch.all((permutation_indexes + 1) == x.gather(dim=1, index=permutation_indexes))
tensor(True)
</code></pre>
<p>As you can see in the code snippet above, we can gather elements from a tensor ‘x’ with an index tensor ‘permutation_indexes’. However, I need to also be able to put another tensor ‘y’ into that 'gather’ed tensor locations:</p>
<pre><code>&gt;&gt;&gt; y = torch.arange(0, 3).view(1,3,1).expand_as(x)
&gt;&gt;&gt; print(y)
tensor([[[0, 0, 0, 0],
         [1, 1, 1, 1],
         [2, 2, 2, 2]],
        [[0, 0, 0, 0],
         [1, 1, 1, 1],
         [2, 2, 2, 2]]])
&gt;&gt;&gt; inverse_permutation_indexes = x.gather_put(dim=1, index=permutation_indexes, tensor=y)
&gt;&gt;&gt; print(inverse_permutation_indexes)
tensor([[[0, 0, 1, 2],
         [1, 2, 0, 0],
         [2, 1, 2, 1]],
        [[1, 2, 0, 0],
         [2, 1, 1, 2],
         [0, 0, 2, 1]]])
&gt;&gt;&gt; torch.all(x == x.gather(dim=1, index=permutation_indexes).gather(dim=1, index=inverse_permutation_indexes))
tensor(True)
</code></pre>
<p>Obviously there is no function in PyTorch named ‘gather_put’ as above. How can I do the above without using for loops for all other dimensions (dimension 0 and 2)?</p>
<p>NOTE: This question sprung up from implementing <a href="https://www.geeksforgeeks.org/inverse-permutation/" rel="nofollow noopener">inverse permutation</a> idea. If you follow the link, the idea is pretty simple for a 1-dimensional tensor (vector or list). But it is nearly impossible to do with a tensor of arbitrary dimensionality (2D, 3D and higher) without using serialized (for loop) code.</p>

tom: <p>It seems rather bold to say that it’s nearly impossible to do without for loops.</p>
<p>For example, I can replace the line defining <code>inverse_permutation_indexes</code> with</p>
<pre><code class="lang-auto">inverse_permutation_indexes = torch.empty_like(permutation_indexes).scatter_(dim=1, index=permutation_indexes, src=y)
</code></pre>
<p>and get the computation to evaluate to the same. One could achieve a similar thing using advanced indexing.</p>
<p>Best regards</p>
<p>Thomas</p>

deniz: <p>Please excuse my bold statement about performing above operation being nearly impossible, as it seemed so before reading your excellent answer. I had been looking for an answer non-stop until now. Thank you!</p>
Distributed Pytorch with Existing MPI Processes

Vibhatha_Abeykoon: <p>Hi All,</p>
<p>I have already running MPI program in the distributed model. I want to use those processes and run a PyTorch model distributedly. Is there a straightforward way to do this?</p>
<p>For instance, MPI_INIT() is already called from my programme, and I have a world_size of 4.<br>
Is it possible to start a PyTorch distributed programme from this model?</p>
<p>Adding more information:</p>
<p>How to use the following option with Pytorch DistributedDataParallel model?</p>
<p>store(Store, optional): Key/value store accessible to all workers, used<br>
to exchange connection/address information.<br>
Mutually exclusive with <code>init_method</code>.</p>
<p>I am trying to see how to map my existing MPI processes to launch a distributed data-parallel model in Pytorch using existing MPI instances.</p>
<p>Is there a way such that Pytorch distributed mode can consume existing MPI processes?</p>
Output of CNN used for regression converges to mean of training data

etiwal: <p>Hi everyone!<br>
I used a CNN to do regression on a trainings set to estimate the parameters of a line (mx+b). For this purpose I generated Images with Matlab which plots one line (consisting of randomly initialized values for m and b in the range of [-10,10]) per picture.</p>
<p>While training, my outputs converge to 0 for m and b since this is the mean value of both my inputs.</p>
<p>sample image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7b264df10f4166aa2f55dbd5dcd0892c9cb0242.jpeg" alt="image3" data-base62-sha1="zle8Qkequef8o4aS38DNUMls2lQ" width="227" height="227"></p>
<p>The CNN used looks like this:</p>
<pre><code class="lang-auto">    def __init__(self):
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 6, kernel_size=2, stride=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(6, 16, kernel_size=3, stride=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(16 * 55 * 55, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 2)


    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc1(out)
        out = self.fc2(out)
        out = self.fc3(out)
        return out
</code></pre>
<p>Is the CNN used for this task correct? How should I improve my model to do better in regression?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> I read in one of your replies (May 18) that you had a similar issue?</p>

mailcorahul: <p>Can you post your input image here? Is it a black line drawn on a white bg(or vice-versa)?<br>
Also from my understanding of your problem formulation, you’re trying to estimate m and b values given an input image. If that’s the case, this doesn’t seem to be a correct/intuitive function mapping.</p>

etiwal: <p>I added the input image to original post.</p>
<p>What do you mean by incorrect function mapping? In my point of view, at least for the parameter m there should only exist one solution. Also for b the solution should be unique as long as the coordinate system is the same for every image (which is the case in my set).</p>

etiwal: <p>I found out that I forgot to use an activation function (now relu()) for the last two fully connected layers. Now it works out quite well! I am still thankful for any other tips or comments on how to improve regression performance using a CNN!</p>
How can I make imageFolder to return a dataset which contains datasets also

WormWakedEarlyEatenB: <p>Here is our data’s address.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2587a295a7acce20f26d249d7993752956d8794.png" alt="address" data-base62-sha1="u0NPtHrbRlAlR8a9VHErrQY8OPO" width="555" height="198"><br>
I want to get a set of volumes with labels not just some labeled images, like a list contains all the volumes and images in each volume.<br>
How can I make it?</p>

ptrblck: <p>Do you want to load all images from the volume at once and return them?<br>
I think the best approach would be to create a custom <code>Dataset</code> and implement the desired logic there.<br>
Basically you could pass the volume paths to <code>__init__</code>, and load all images corresponding to a volume in <code>__getitem__</code>.<br>
Have a look at the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data Loading Tutorial</a> for more information.</p>

WormWakedEarlyEatenB: <p>thx, you really help me a lot. I override ImageFolder’s function <em>init</em> and <em>getitem</em> and achieve the goal.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f789047b915e3ba915b142907795fb3d14e00a6f.png" data-download-href="https://discuss.pytorch.org/uploads/default/f789047b915e3ba915b142907795fb3d14e00a6f" title="volumeSet"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f789047b915e3ba915b142907795fb3d14e00a6f.png" alt="volumeSet" data-base62-sha1="zjNuCprGKzbihkM3OX1H3B3ckQn" width="297" height="499" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">volumeSet</span><span class="informations">493×829 23.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But there are some new questions. After changing above part without changing dataloader, pycharm always warned me that ‘‘list index out of range’’. And if I set batch_size, I couldn’t get the expected output, such as the expected one is batch_size x 100x3x224x224, while what I got is like 100 x batch_size x 3x224x224.<br>
So should I override dataloader? Could you give me some advece?</p>
Error while using GPU

XYC-EXP: <h2>My network: (cuda = 9, pytorch = 1.1.0)</h2>
<pre><code>input tensor -&gt; GRU(it works) -&gt; FullConnect(that was when pytorch crashed) --x--&gt; output
</code></pre>
<h3>Traceback (most recent call last):</h3>
<pre><code class="lang-auto">  File "main.py", line 42, in &lt;module&gt;
    mod_train.train(network, device, dataloader, optimizer, loss_fn, i)
  File "/root/code/GPU/mod_train.py", line 20, in train
    outputs = model(inputs)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/code/GPU/mod_network.py", line 47, in forward
    input = self.fc_0(input[:, -1, :].cuda())
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 92, in forward
    return F.linear(input, self.weight, self.bias)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1406, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653215914/work/aten/src/THC/THCBlas.cu:259
</code></pre>
<hr>
<h5>(Is there any doc explaining special requirements for GPU training?)</h5>
<h3>How to solve this problem ?</h3>

albanD: <p>Hi,</p>
<p>Could you give more precision on the size of the input Tensor for your linear layer?<br>
Also can you make a small code sample that reproduces the issue please?</p>

XYC-EXP: <h3>nn.Linear always crashs</h3>
<pre><code class="lang-auto">m = torch.nn.Linear(2, 2)
m.cuda()
input = torch.randn(10, 2)
input = input.cuda()
m(input)
</code></pre>

albanD: <p>Hi,</p>
<p>Are you using google colab by any chance? In that case, this is a known issue and it is tracked here: <a href="https://github.com/pytorch/pytorch/issues/29096" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/29096</a><br>
If not, how did you install pytorch and cuda on your machine if not?</p>

XYC-EXP: <h3>Thanks.</h3>
<hr>
<p>At last I found out that I made a stupid mistake.<br>
The server had both CUDA 10 and CUDA 9 on it.<br>
And I forgot to switch to the correct version.</p>
<hr>
<h2>ln -snf /usr/local/cuda-9.0/ /usr/local/cuda</h2>
Get number of dimensions of tensor using C++ frontend

yewang: <p>With the python frontend <code>tensor.size()</code> already returns a list of sizes, but in the C++ frontend I have to specify the dimension <code>tensor.size(dim)</code>. How do I know the number of available dimensions in the first place?</p>

LeviViana: <p>This way -&gt; <code>tensor.dim()</code></p>
How to build caffe2 with ONNX opset version greater than 9?

zetyquickly: <p>Hello,<br>
I’ve currently worked with freshly merged feature <a href="https://github.com/pytorch/vision/pull/1401" rel="nofollow noopener">pytorch/vision#1401</a> and won’t able to find a way to make Caffe2 work with ONNX operation set 10?</p>
<p>Is there a way to build a Caffe2 from source with this opset?</p>

tom: <p>I don’t know but I would venture that arguably the go-to library for ONNX model serving is <a href="https://github.com/microsoft/onnxruntime" rel="nofollow noopener">ONNXRuntime</a> these days.</p>
<p>See also <a href="https://discuss.pytorch.org/t/caffe2-vs-libtorch-realistically/50526/4" class="inline-onebox">Caffe2 vs libtorch, realistically</a> .</p>
<p>Best regards</p>
<p>Thomas</p>

jspisak: <p>Indeed. We’ve removed tutorials around this as the Caffe2 path really isn’t being maintained. Assuming you are focused on using ONNX (and not deploying via Torchscript), to leverage the latest opset, it makes more sense to export an ONNX model from PyTorch and then deploy in ONN Runtime as <a class="mention" href="/u/tom">@tom</a> suggests. Here is a tutorial that is maintained by the MSFT team: <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html</a></p>

zetyquickly: <p>Thanks a lot, I’ve got it. Caffe2 right now is totally out maintenance</p>

zetyquickly: <p>Thank you too. Let me clarify why I was eager to find a way to make Caffe2 work with recent opset. I’ve recently worked with your repo with AICamera that manages PyTorch v1.0.0  to work with Android devices using Caffe2 library. So I know that right now it is TorchScript available in mobile and onnx-runtime doesn’t work with mobile. So I wondered how to deploy on mobile ONNX model with recent opset using Caffe2 backend.<br>
P.S. The last statement is not a question just a theme to discuss</p>