akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Wrong value recorded in register_hooks

csailnadi: <p>I am registering hooks to update convolutional layer in this way</p>
<pre><code class="lang-auto">for i, conv_layer in enumerate(model.convos):
       conv_layer.weight.retain_grad()
       print("conv_layer.weight.shape ", conv_layer.weight.shape)
       print("Ms[i].shape ", Ms[i].shape)
       h = conv_layer.weight.register_hook(lambda grad: convo_grad_mult(grad, Ms[i]))
</code></pre>
<p>The output of print statements are</p>
<pre><code class="lang-auto">conv_layer.weight.shape  torch.Size([17, 3, 3, 3])
Ms[i].shape  torch.Size([3, 3])
conv_layer.weight.shape  torch.Size([35, 17, 3, 3])
Ms[i].shape  torch.Size([17, 17])
conv_layer.weight.shape  torch.Size([9, 35, 3, 3])
Ms[i].shape  torch.Size([35, 35])
</code></pre>
<p>Then during the hook registration within function <code>convo_grad_mult</code> I print the sizes of arguments:</p>
<pre><code class="lang-auto">&gt; M  torch.Size([35, 35])
&gt; grad  torch.Size([35, 81])
&gt; M  torch.Size([35, 35])  **#here I am getting the error since I expect M of size [17, 17]**
&gt; grad  torch.Size([17, 315])
</code></pre>
<p>Why am I receiving the same M again while it is clear that I should have received M of size [17, 17]?</p>
<p><a href="https://codeshare.io/arA97q" rel="nofollow noopener">Here</a> is full training loop.</p>
<p><strong>EDIT:</strong><br>
<strong>Much simpler setting to reproduce the issue</strong></p>
<pre><code class="lang-auto">def dummy(arg):
    print(arg)
</code></pre>
<p>…</p>
<pre><code class="lang-auto">for i, conv_layer in enumerate(model.convos):
      print("i={}".format(i))
      print("conv_layer.weight.shape ", conv_layer.weight.shape)
      h = conv_layer.weight.register_hook(lambda grad: dummy(i))
      hooks.append(h)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">i=0
conv_layer.weight.shape  torch.Size([17, 3, 3, 3])
i=1
conv_layer.weight.shape  torch.Size([35, 17, 3, 3])
i=2
conv_layer.weight.shape  torch.Size([9, 35, 3, 3])
</code></pre>
<p>after calling  backward() I get</p>
<pre><code class="lang-auto">2
2
2
</code></pre>
<p>Why does it only register last one?<br>
Clearly I register hooks for different convo layers</p>
<p>If I do</p>
<pre><code class="lang-auto">h1 = model.convos[0].weight.register_hook(lambda grad: dummy(0))
h2 = model.convos[1].weight.register_hook(lambda grad: dummy(1))
h3 = model.convos[2].weight.register_hook(lambda grad: dummy(2))
</code></pre>
<p>Then I get correct output</p>
<pre><code class="lang-auto">2
1
0
</code></pre>
<p>Why it does not work with a loop?</p>

ptrblck: <p>Python uses lexical scoping so that the closure will use the name and scope of the passed variable <code>i</code>.<br>
To fix this issue, you could pass <code>i</code> to the lambda as:</p>
<pre><code class="lang-python">conv_layer.weight.register_hook(lambda grad, i=i: dummy(i))
</code></pre>
Difference between generate data on GPU and data on CPU then move it to GPU?

linyu: <p>Is there any difference between following code:</p>
<pre><code class="lang-auto">x = torch.randn(2, 2, device=torch.device("cpu"), requires_grad=Ture).cuda()
</code></pre>
<pre><code class="lang-auto">x = torch.randn(2, 2, device=torch.device("cuda"), requires_grad=Ture)
</code></pre>
<p>when I use the first one, I can not get the grad of x by loss.backward(), and the second one is work.</p>

ptrblck: <p>The <code>.cuda()</code> call returns a non-leaf variable, which won’t be optimized.<br>
<a class="mention" href="/u/alband">@albanD</a> explained it very well in <a href="https://discuss.pytorch.org/t/valueerror-cant-optimize-a-non-leaf-tensor/21751/2?u=ptrblck">this post</a>.</p>

linyu: <p>Thank you for your reply!</p>

dio_din: <p>So, should I init the parameters of a <code>Module</code> before or after moving it to GPU?</p>

ptrblck: <p>If you are using <code>torch.nn.init</code> methods to initialize your parameters, it shouldn’t matter, since they are manipulating the tensors in-place.</p>

dio_din: <p>Will the parameters becoming <strong>non-leaf tensors</strong> which wouldn’t got optimized as you said if I  transfer <code>Module</code> to GPU?</p>

ptrblck: <p>This won’t be the case, if you use <code>torch.nn.init</code> methods or manipulate them in-place with a custom method. Are you seeing any issues using this approach?</p>

dio_din: <p>No, I just want to make clear what <code>.cuda()</code> would affect the optimization mechanism.</p>

ptrblck: <p><code>.cuda()</code> is not an in-place method, but a differentiable operation, such that calling it on an <code>nn.Parameter</code> will create a non-leaf variable as explained in the linked post.</p>

dio_din: <p>Thanks for the explanation, I’m clear now.</p>
How to model 2 sets of input in RNN

ashutoshnayakIE: <p>Hi,<br>
I have two sets of input. I want one set to initialize the hidden state of LSTM RNN and use these hidden states as initial points for the RNN. How can I model that? I have attached the format below.<br>
Any links to examples would be very helpful.<br>
Thank you.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c35f170ec8d1c77aa9ff1ce517d4d903f4ca97cf.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c35f170ec8d1c77aa9ff1ce517d4d903f4ca97cf" title="Run"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c35f170ec8d1c77aa9ff1ce517d4d903f4ca97cf_2_375x375.jpeg" alt="Run" data-base62-sha1="rSkPQCVhSIWLlVywcZ4rHCYYvoz" width="375" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c35f170ec8d1c77aa9ff1ce517d4d903f4ca97cf_2_375x375.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c35f170ec8d1c77aa9ff1ce517d4d903f4ca97cf_2_562x562.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c35f170ec8d1c77aa9ff1ce517d4d903f4ca97cf_2_750x750.jpeg 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Run</span><span class="informations">2048×2048 347 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Pytorch index for 3D-tensor

H_ng_Le: <p>Hello, I have a 3D tensor A of size (x,y,z) and an 1D vector B of size (y), in which each (index,value) in B correspond to a pair of index of y-dimension and z-dimension. The result slice tensor will have dimension (x,y) or (x,y,1). Is there a fast way in Pytorch so that I can get the index that can be used to slice the tensor? Thanks a lot!</p>

ptrblck: <p>Could you post a dummy example using some random input tensors, as I’m unsure how to interpret the given shapes?</p>
Problems loading a custom LSTM Module (torch 1.4.0)

Arshad_Javeed: <p>I’m facing problems loading a custom LSTM Module. The model works as expected during the training phase, but when I save the model and load it back, I get very different and very inaccurate results.</p>
<p>I have also verified the weights of all the layers of the model before and after loading, and they are exactly similar, but the results are significantly inaccurate after loading.</p>
<p>I have tried all the methods described in the official doc (<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a>)<br>
None of the methods seem to work.</p>
<p>Can someone please help me regarding the issue.</p>
<p>Thanks.</p>

ptrblck: <p>Could you post an executable code snippet to reproduce this error, please?</p>
Any body know about confusion matrix calculation for cnn model

Muhammad_Izaz: <p>losses=[]<br>
accuracy=[]<br>
for epoch in range(epochs):<br>
epoch_loss = 0.0<br>
epoch_accuracy = 0.0</p>
<pre><code>for features, labels in train_data:
    outputs = model(features)
    loss = criterion(outputs, labels)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    epoch_loss += loss.item() # here we are accumulating loss for each training batch.
    epoch_accuracy += (outputs.argmax(1) == labels).float().mean()

losses.append(epoch_loss/batches)
accuracy.append(epoch_accuracy/batches)   
print(f'Epoch: {epoch} -&gt; Loss: {(epoch_loss/batches):.8f}, Accuracy: {(epoch_accuracy/batches):.8f}')</code></pre>

ptrblck: <p>You could use the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html">sklearn implementation</a> or the code from <a href="https://discuss.pytorch.org/t/how-to-get-the-sensitivity-and-specificity-of-a-dataset/39373/6">this post</a>.</p>
Evaluation metric for GANs

therohanpandey: <p>Does anyone know how to implement an evaluation metric such as the Inception Score? I have been able to generate new samples and save them. I now want to evaluate the same.</p>

ptrblck: <p>There seem to be a few implementation on GitHub such as <a href="https://github.com/sbarratt/inception-score-pytorch">this one here</a>.</p>

CMP_OF_UNIVERSE: <p>Sir i am going to build a AI chatbot using python<br>
Here i am facing a problem</p>

ptrblck: <p>What kind of problem are you facing?<br>
If it’s not related to this topic, please feel free to create a new topic.</p>
Using nn.Module apart from the model for training

braindotai: <p>Let’s say I wanna build a custom metric, so in that case, I could just simply just write a function and use accordingly in my training loop.<br>
Can I rather define metric as a class inherited from <code>nn.Module</code> and there define the forward method for the metric definition?<br>
I know I can, but is it beneficial in any way (if it is) or just a function definition is fine.</p>

ptrblck: <p>If your metric is stateless (i.e. you don’t want to store arguments or other parameters), then you could just define a method. Otherwise, a custom class derived from <code>nn.Module</code> could store some arguments e.g. <code>reductions</code> as an internal attribute.<br>
Also, have a look at <a href="https://discuss.pytorch.org/t/beginner-should-relu-sigmoid-be-called-in-the-init-method/18689/6">this post</a> I’ve written some time ago where I discuss the advantages of both approaches.</p>
How to use Transforms

Aliman: <p>I am sorry if I keep asking really basic questions but I was wondering how to use transforms. Can I just call them on two tensors? I have a tensor X with images and honestly didn’t know how to use transforms on them.<br>
Best</p>

charan_Vjy: <p>Do check up the examples given on this page: <a href="https://pytorch.org/docs/stable/torchvision/transforms.html" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/transforms.html</a></p>
Custom data loader: apply transforms only when called?

murphinator: <p>My first time trying to implement a custom data loader. I was just wondering if there are issues with applying transforms to the input data only when PyTorch’s data generator object is ready to call the image? Or do all of the transformations have to be applied at once, and then stored? Or are the transforms only applied to the batch_size so memory doesn’t become an issue?</p>
<pre><code class="lang-auto">class CustomDataLoader(Dataset):
    
    def __init__(self, img_paths, transforms=None):
        self.img_paths = img_paths
        self.to_image = []
        self.len = len(self.img_paths)
        
    def __len__(self):
        return len(self.img_paths)
    
    def __getitem__(self, idx):
        self.img_paths_temp = cv2.imread(self.img_paths[idx])
        
        if transforms is not None:
            self.img_paths_temp = transforms(self.img_paths_temp)
        
        return self.img_paths_temp
</code></pre>

boto: <p>Hi!</p>
<p>It shouldn’t make any difference regarding the result.<br>
In your case, the transformation is only applied when creating the batch, as it executes <code>__getitem__</code> <code>batch_size</code> times, as far as I know.<br>
I wouldn’t change it, but you can experiment with different configurations.</p>
Extracting layers of a subclassed pretrained model in PyTorch

AlJazari: <p>After loading a pre-trained model, I am able to extract its weights and biases in addition to other information, such as layer names–if exist, but not information included in the forward function. Also printing the model architecture, <code>print(model)</code> prints the layers in the order they were provided inside <code>__init__</code> function of the <code>Class(nn.Module)</code> but not their actual order.</p>
<p>Is there a way to access the information that was provided in the <code>forward</code> function but not modeled as layers, such as  <code>F.relu</code> ?</p>
<p>Here’s a code snippet:<br>
import torch<br>
import torch.nn.functional as F</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc2 = nn.Linear(128, 10)
        self.dropout2 = nn.Dropout2d(0.5)
        self.fc1 = nn.Linear(9216, 128)

    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dropout2(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output

model = Net()
print(model)
</code></pre>
<p>And the output is:</p>
<pre><code class="lang-auto">Net(
  (fc2): Linear(in_features=128, out_features=10, bias=True)
  (dropout2): Dropout2d(p=0.5, inplace=False)
  (fc1): Linear(in_features=9216, out_features=128, bias=True)
)
</code></pre>
<p>I would like to (1) print the layers in order and (2) learn about the ReLU and where they were applied (this applied to other functions, such as maxpool).</p>
<p>My current solution is to build the model using Sequential, but what about models that are already trained but do not have implementation details (model code not available)?</p>
<p>Thank you!</p>
Register_hook does not work

csailnadi: <p>I am wondering what I am doing wrong with register_hook since it does not seem to register a hook.<br>
My goal is to modify grad matrix before weights are updated. For debugging purposes I just assigned zero matrix for a grad variable, but network trains perfectly.<br>
So clearly hooks don’t work, but I cannot figure why. Here is my train function:</p>
<pre><code class="lang-auto">def train_cnn():

    model = Net()
    model.cuda()

    criterion = torch.nn.CrossEntropyLoss()

    model.train()
    optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=momentum, weight_decay=weight_decay)

    for e in range(epochs):
        agg_loss = 0

        for data in trainloader:
            x, y = data
            x = x.cuda()
            y = y.cuda()

            outputs = model(x)
            loss = criterion(outputs, y)
            optimizer.zero_grad()
            loss.backward()

            with torch.no_grad():
                 hooks = []
                 for conv_layer in model.convos:
                     conv_layer.weight.retain_grad()
                     h = conv_layer.weight.register_hook(lambda grad: torch.zeros(size=grad))
                     hooks.append(h)

                 for fc_layer in model.linears:
                     fc_layer.weight.retain_grad()
                     h = fc_layer.weight.register_hook(lambda grad: torch.zeros(size=grad))
                     hooks.append(h)

            optimizer.step()
            for h in hooks:
                 h.remove()
</code></pre>
<p>Thank you!</p>

ptrblck: <p>Disregard this post, as it was a wrong suggestion.</p>

csailnadi: <p>ohh, I see… I thought I could just create a function with return value something like<br>
lambda grad: func(grad) to update weights, because I need to perform actions on the gradient that are not attribute functions. Is this possible with register hooks?<br>
Function would be a series of different matrix manipulations.</p>

ptrblck: <p>No, sorry. I’m wrong and your code snippet should be the right way and also seems to work:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
model.weight.register_hook(lambda grad: torch.ones_like(grad) * 1000)
model(torch.randn(1, 1)).backward()
print(model.weight.grad)
&gt; tensor([[1000.]])
</code></pre>
<p>I’ll edit my previous post.</p>

csailnadi: <p>wait, then what is my problem? It definitely did not work for me using my training code, it just trained as usual, while it was supposed to fail the training…<br>
Is calling <code>optimizer.backward()</code> then right way of doing it?</p>

ptrblck: <p>I think your code registers the hooks too late (after the <code>backward</code> call).<br>
Note that the hook will be called during the gradient calculation.</p>
<p>You could register the hooks during setup and use the standard training loop without removing and adding the hooks again.</p>
Batch normalization axis

peter: <p>Is there a way to specify an alternative axis on which to perform BatchNormalizaiton. I want to apply it to a RNN which has the shape:</p>
<pre><code>     Batch x Steps x Features.
</code></pre>
<p>So the 1D batch normalization should be applied to the last dimension (Features) and not Steps (which is the default behavior since it is correct for convolutional networks).</p>
<p>But I could not find a way to achieve this, it seems that this logic is implemented in the native Torch part since even the functional API doesn’t allow for an axis to be specified.</p>
<p>So any tricks are welcome. (the only thing I could come up with is moving the axis around before and after the batch normalization, but that seems to be neither an elegant nor performant solution to the problem).</p>

Cristina_Segalin: <p>The only way around this I found so far is to transpose the 1,2 axis, do the batch norm and transpose them back.<br>
I was also working on the same issue.</p>

Cristina_Segalin: <p>However, I jut found out that using nn.LayerNorm([steps,features]) gives better performances for my model. Using the transpose trick was leading to overfit</p>

peter: <p>Same trick as I ended up using (had no issues with overfitting, but I guess that depends on the use-case). In the mean time I also opened an issue on github to make it possible to normalize on any axis.</p>
Training and test discrepancy (libtorch)

DeuS_CaNoN: <p>So I’ve been playing around with pytorch/libtorch C++ to do something similar to AlphaGo but with Gomoku instead and have almost everything working.<br>
What I’ve noticed is that what I feed into my model for training doesn’t seem to get the same result when I test it again.</p>
<p>If someone wants any part of the source code I’ll post it but I’ll try and summarize first.<br>
My model takes in a tensor of shape (x, 4, 15, 15) (where x is the batch size)<br>
it outputs a simple tanh float number between -1.0 and 1.0</p>
<p>As a test I gave it 2 boards and told it that board 1 should be 1.0 and board 2 should be -1.0.<br>
As it trains with an Adam optimizer I record down what the forward outputs, and the last output is something like<br>
0.9894<br>
-0.8495</p>
<p>However, if I feed board 1 by itself into my model as a shape (1, 4, 15, 15) the model spits out 0.5648.<br>
I then send in both board 1 and board 2 as a shape (2, 4, 15, 15) and only then would it output my recorded 0.98/-0.84 above.</p>
<p>Is this expected behaviour? Am I training my model wrong?</p>
<p>P.S. I train my model by manually creating a shape (x, 4, 15, 15) inputTensor and a (x, 1) valueAnswerTensor with the following training loop.</p>
<pre><code class="lang-auto">torch::Tensor valueTensor = m_pNetworkGpu-&gt;forward(inputTensor.to(torch::kCuda));
torch::Tensor valueLoss = (valueTensor - valueAnswerTensor.to(torch::kCUDA)).pow(2).mean();
std::cout &lt;&lt; valueTensor;
valueLoss.backward();
adamOptimizer.step();
</code></pre>
<p>For a game neural network to work I need to send individual boards to the network to evaluate, but I’m under the impression you want to send batches for training. But the results above seems to point that you have to train using 1 example at a time?</p>

DeuS_CaNoN: <p>For any future developers that come upon this, I figured out why, I had some dropout based layers in my code, which means the network being in train/eval mode mattered.</p>
<p>I was able to fix this by calling<br>
model.train() in the beginning of my training function and model.eval() when calling just trying to get an accurate evaluation of a boardstate.</p>
One-hot encoding problem

Bollo7: <p>Hello,</p>
<p>I am building a model for multiclass prediction for targets [-1,0,1] and currently getting an error from loss function that</p>
<pre><code class="lang-auto">Target -1 is out of bounds.
</code></pre>
<p>Well, it is a good idea to encode classes with <code>torch.nn.functional.one_hot</code> , but I can’t encode negative values, as I know. Is there any other alternatives or solutions of how to encode or directly feed my targets?</p>

AladdinPerzon: <p>Couldn’t you have targets [0,1,2] if the -1 is causing the issue? Could you share some code snippet to how it looks like now</p>
Why we extend Dataset in our custom class dataset

rebeen: <p>what will happen if I will not extend the Dataset(I know this is an abstract class ) but what is the usefulness of this extending?<br>
import numpy as np</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset
 
class ExampleDataset(Dataset):
    def __init__(self, data):
        self.data = data
 
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        return self.data[idx]


sample_data = np.arange(0, 10)
print('The whole data: ', sample_data)
dataset = ExampleDataset(sample_data)
print('Number of samples in the data: ', len(dataset))
print(dataset[2])
print(dataset[0:5])
</code></pre>

JuanFMontesinos: <p>It’s a way to check it’s an instance of a BaseClass. If you read the base class it’s nothing but a scheme.</p>

rebeen: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="2" data-topic="75024">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>instance of a BaseClass</p>
</blockquote>
</aside>
<p>what will happen if we will not extend the Baseclass?</p>

JuanFMontesinos: <p>Well I would say (my memory is finite) nothing happens but in the worst case just an exception like dataloader requires dataset class or something like that. For sure you will find an exception if you use ConcatDataset over non inherited classes.</p>

rebeen: <h2>Do we need to use the Dataset class?</h2>
<p>Well, we are able to run deep learning models  <em><strong>without</strong></em>  the  <code>Dataset</code>  class; but loading a dataset is generally memory-intensive — so it’s highly recommended to use the  <code>Dataset</code>  class.</p>
Calculating Loss using target network in DQN (libtorch)

Loris_Donovan: <p>I encountered an exception in KernelFuntion.h file when trying to pass the output from the target Network to calculate loss, and i would like to know what i am missing here, like can’t i just pass the value in the target argument of the loss function? like this:</p>
<pre><code class="lang-auto">torch::Tensor targetQvalues = targetNet-&gt;forward(input);

policyNet-&gt;zero_grad();
torch::Tensor loss = torch::mse_loss(currentQValues, targetQvalues);
</code></pre>
Evaluation during training harming training

wenlongzhao094: <p>I am finetuning the huggingface implementation of bert on glue tasks. I did two experiments. In the first one, I finetune the model for 3 epochs and then evaluate. In the second, I implemented early stopping: I evaluate on the validation set at the end of each epoch to decide whether to stop training. I print the training loss every 500 steps. In the first epoch, the loss from two experiments are exactly the same as the training proceeds. Right after the first evaluation in the second experiment, the model in the second experiment starts to perform worse, resulting in higher training loss and worse validation performance. Does that have anything to do with Batch Norm? Is there a way to implement early stopping without harming performance?</p>
<p>The train and evaluate functions are here:<br>
<a href="https://github.com/huggingface/transformers/blob/master/examples/run_glue.py" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/huggingface/transformers/blob/master/examples/run_glue.py</a></p>
<p>In the second experiment I added the following at the end of each training epoch.</p>
<blockquote>
<pre><code>    if args.local_rank in [-1, 0] and args.early_stopping &gt; 0:
        logs = {}
        if args.local_rank == -1:  # Only evaluate when single GPU otherwise metrics may not average well
            results = evaluate(args, model, tokenizer)
            for key, value in results.items():
                eval_key = "eval_{}".format(key)
                logs[eval_key] = value

            if "f1" in results:
                metric = "f1"
            elif "acc" in results:
                metric = "acc"
            elif "mcc" in results:
                metric = "mcc"
            else:
                metric = "corr"

            if results[metric] &gt; best_eval_result:
                best_eval_result = results[metric]
                num_degrading_epoch = 0

                if not os.path.exists(args.output_dir):
                    os.makedirs(args.output_dir)

                logger.info("Saving model checkpoint to %s", args.output_dir)
                # Save a trained model, configuration and tokenizer using `save_pretrained()`.
                # They can then be reloaded using `from_pretrained()`
                model_to_save = (
                    model.module if hasattr(model, "module") else model
                )  # Take care of distributed/parallel training
                model_to_save.save_pretrained(args.output_dir)
                tokenizer.save_pretrained(args.output_dir)
                torch.save(args, os.path.join(args.output_dir, "training_args.bin"))
                torch.save(optimizer.state_dict(), os.path.join(args.output_dir, "optimizer.pt"))
                torch.save(scheduler.state_dict(), os.path.join(args.output_dir, "scheduler.pt"))
            else:
                num_degrading_epoch += 1
</code></pre>
</blockquote>

boto: <p>Hi!</p>
<p>What does your training + validation loop look like?</p>
_ValueError: optimizer got an empty parameter list

zahra: <p>Hi,<br>
I wrote this code and I got this error:</p>
<pre><code class="lang-auto">ValueError: optimizer got an empty parameter list
</code></pre>
<pre><code class="lang-auto">class Conv2d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, bias=True):
        super(Conv2d, self).__init__()
        w = torch.empty(kernel_size, kernel_size)
        self.filter = nn.Parameter(nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu'))
        self.bias = nn.Parameter(torch.zeros(out_channels))
        pass

    def forward(self, x):
        out = F.conv2d(x, self.filter, self.bias, stride=self.stride, padding=self.padding)
        return out
</code></pre>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, dropout=None):
        super(Model, self).__init__()
        # in_channels, out_channels, kernel_size, stride=1, padding=0, bias=True:
        conv1 = Conv2d(8, 16, 3, stride=1, padding=1)

    def forward(self, x):
        out = self.conv1(x)
        return out
</code></pre>
<pre><code class="lang-auto">model = Model()
model.to(device)
for param in model.parameters():
    print(param)
learning_rate = 5e-4
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)
</code></pre>
<p>Would you have any advice?</p>
<p>Thanks before all</p>

boto: <p>Hi!</p>
<p>Your <code>forward(self, x)</code> can’t access your <code>conv1</code> as you didn’t initialize it. To do so, you have to call <code>self.conv1 = Conv2d(...)</code> in the <code>___init___</code> method of your model. See here:</p>
<pre><code class="lang-auto">    def __init__(self, dropout=None):
        super(Model, self).__init__()
        # in_channels, out_channels, kernel_size, stride=1, padding=0, bias=True:
        self.conv1 = Conv2d(8, 16, 3, stride=1, padding=1) # Changes made here

    def forward(self, x):
        out = self.conv1(x)
        return out
</code></pre>
Use torch.optim to optimize a stats learning objective

kaiseryet: <p>Hey guys, so I am trying to optimize a stats learning model but not nn object, or put it in this way: can pytorch optimize any objective function with respect to its parameters? e.g.</p>
<pre><code class="lang-auto">x = torch.randn(500)
obj = x.T@x + torch.log(torch.abs(x)).sum + x[0]
</code></pre>
<p>Now can I optimize <code>obj</code> with respect to <code>x</code>? Bother giving an example? Thanks.</p>
Is it possible to name param_group?

Paritosh: <p>I am using <code>optimizer.add_param_group()</code> to add parameters to be optimized as and when needed. During training I would like to change the learning rates of different parameter groups in different ways.</p>
<p>One way to do this is to remember the order in which they were added and use that ordering/ID to modify the learning rates.</p>
<p>I was wondering if I can name the param_groups so that I can modify learning rates by accessing them through their names?</p>

Intel_Novel: <p>Very few people know this.</p>

baxincs: <p>Yes, you can. As defined in the Optimizer class:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/85c51a8c10984e4ebdbca731ca0ebd30e5e8af4d/torch/optim/optimizer.py#L26" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/85c51a8c10984e4ebdbca731ca0ebd30e5e8af4d/torch/optim/optimizer.py#L26" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/85c51a8c10984e4ebdbca731ca0ebd30e5e8af4d/torch/optim/optimizer.py#L26</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="16" style="counter-reset: li-counter 15 ;">
<li>
</li>
<li>class Optimizer(object):</li>
<li>    r"""Base class for all optimizers.</li>
<li>
</li>
<li>    .. warning::</li>
<li>        Parameters need to be specified as collections that have a deterministic</li>
<li>        ordering that is consistent between runs. Examples of objects that don't</li>
<li>        satisfy those properties are sets and iterators over values of dictionaries.</li>
<li>
</li>
<li>    Arguments:</li>
<li class="selected">        params (iterable): an iterable of :class:`torch.Tensor` s or</li>
<li>            :class:`dict` s. Specifies what Tensors should be optimized.</li>
<li>        defaults: (dict): a dict containing default values of optimization</li>
<li>            options (used when a parameter group doesn't specify them).</li>
<li>    """</li>
<li>
</li>
<li>    def __init__(self, params, defaults):</li>
<li>        torch._C._log_api_usage_once("python.optimizer")</li>
<li>        self.defaults = defaults</li>
<li>
</li>
<li>        if isinstance(params, torch.Tensor):</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>You can use a dictionary to define a parameter group with the syntax:<br>
param_groups = [{‘params’: COMPONENT_1_PARAMTERS, ‘lr’: COMPONENT_1_LR, <strong>‘name’: ‘GROUP_1’</strong>},<br>
{‘params’: COMPONENT_2_PARAMTERS, ‘lr’: COMPONENT_2_LR, <strong>‘name’: ‘GROUP_2’</strong>},<br>
…<br>
]</p>
<p>opt.Adam(param_groups, lr=INITIAL_LR)</p>
Speeding up mask creation

Philipp_Hager: <p>Hey everybody!</p>
<p>I got a newbie question: I am creating an attention mask for an NLP project. I essentially need a quadratic mask of (words x words) for each document (all documents have the same number of words) but with a twist: I want to have n different masks per word category. So essentially, I have a boolean mask of shape (Batch x Categories x Words x Words) at hand and want to efficiently select only those words which belong to a specific category. I have a minimal solution to my problem, but it uses 2 nested loops, which is very slow and I wondered if some of you more versed at Tensor broadcasting could help me out.</p>
<p>This is a minimal example with 2 documents, containing 4 words belonging to 3 categories.</p>
<pre><code class="lang-auto"># Batch x Category Mapping per Word. Essentially, the first word of document 0 belongs to category 1 etc.
categories = torch.LongTensor([
    [1, 0, 1, 2],
    [0, 2, 0, 0],
])

# Values to be masked: Batch x Categories X Word X Word
values = torch.rand(2, 3, 4, 4)

# Mask: Batch x Categories X Word X Word
mask = torch.zeros(2, 3, 4, 4)
num_documents = mask.size(0)
num_categories = mask.size(1)
num_tokens = mask.size(2)

for document in torch.arange(num_documents):
    for category in torch.arange(num_categories):
        # Filter only values of tokens in category.
        token_in_category = (categories[document] == category)
        mask[document, category, token_in_category] = token_in_category.float()

mask
</code></pre>
<p>Desired Output:</p>
<pre><code class="lang-auto">tensor([[[[0., 0., 0., 0.],
          [0., 1., 0., 0.],
          [0., 0., 0., 0.],
          [0., 0., 0., 0.]],

         [[1., 0., 1., 0.],
          [0., 0., 0., 0.],
          [1., 0., 1., 0.],
          [0., 0., 0., 0.]],

         [[0., 0., 0., 0.],
          [0., 0., 0., 0.],
          [0., 0., 0., 0.],
          [0., 0., 0., 1.]]],


        [[[1., 0., 1., 1.],
          [0., 0., 0., 0.],
          [1., 0., 1., 1.],
          [1., 0., 1., 1.]],

         [[0., 0., 0., 0.],
          [0., 0., 0., 0.],
          [0., 0., 0., 0.],
          [0., 0., 0., 0.]],

         [[0., 0., 0., 0.],
          [0., 1., 0., 0.],
          [0., 0., 0., 0.],
          [0., 0., 0., 0.]]]])
</code></pre>
<p>Thanks in advance, appreciate any help <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
Factorization of a weight matrix as products of low-rank matrices

Fawaz_Sammani: <p>Hi. I have a weight matrix (call it <code>Q</code>). I’d like to factorize it as product of low-rank matrices to reduce model parameters. So i’d like to have <code>q(left)q(right) = Q</code>, where <code>q(left)</code> has a shape of <code>(mxk)</code> and <code>q(right)</code> has a shape of <code>(kxn)</code>, where <code>k &lt;min(m,n)</code>. Is there any function in Pytorch to do this, or can I implement it like this:</p>
<pre><code class="lang-auto">m = 512 
k = 85  # if set to 85: (512 * 85) + (85 * 512) &lt;&lt; (512 * 512)
n = 512  

q_left = nn.Parameter(torch.FloatTensor(m, k).uniform_(-math.sqrt(1/n)/2, math.sqrt(1/n)/2))
q_right = nn.Parameter(torch.FloatTensor(k, n).uniform_(-math.sqrt(1/n)/2, math.sqrt(1/n)/2))
q = torch.matmul(q_left, q_right)
</code></pre>

KFrank: <p>Hi Fawaz!</p>
<aside class="quote no-group" data-username="Fawaz_Sammani" data-post="1" data-topic="76278" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fawaz_sammani/48/10755_2.png" class="avatar"> Fawaz_Sammani:</div>
<blockquote>
<pre><code class="lang-auto">q = torch.matmul(q_left, q_right)
</code></pre>
</blockquote>
</aside>
<p>This approach will work, but it has the disadvantage that even though<br>
you don’t train the full set of the <code>m*n</code> elements of <code>q</code>, you do create<br>
and work with them.  It’s simpler and cheaper to never explicitly create<br>
<code>q</code> and, instead, apply <code>q_right</code> and <code>q_left</code> successively to the data<br>
tensor flowing through your network.</p>
<p>Something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">module = torch.nn.Sequential (
    # linear layer 1
    # non-linear activation 1
    # ...
    torch.nn.Linear (n, k, bias = False),  # q_right, no bias between q_right and q_left
    # no activation between q_right and q_left
    torch.nn.Linear (k, m),  # q_left, bias if you want it
    # non-linear activation after "q layer"
    # ...
)
</code></pre>
<p>This – on the forward pass – is mathematically the same as having a<br>
single linear layer for <code>q</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">    # ...
    torch.nn.Linear (n, m),  # the full q matrix
    # non-linear activation after "q layer"
    # ...
</code></pre>
<p>where you have somehow set <code>q = q_left * q_right</code>.</p>
<p>(As an aside, another technique people sometimes use is to first train<br>
the full <code>m*n</code> <code>q</code> layer-- or take it from a pre-trained network – and then<br>
factor it into <code>q_left * q_right</code>, and then further train or fine tune.)</p>
<p>Best.</p>
<p>K. Frank</p>

Fawaz_Sammani: <p>Thank you for your detailed reply <a class="mention" href="/u/kfrank">@KFrank</a>. I highly appreciate the answer you gave.</p>
Kernel crash while normalizing

Angry_potato: <p>Using Torchvision.transform.Normalize crashes my kernel.<br>
I am using jupyter notebook<br>
GPU:920M<br>
CPU:i5 7th gen<br>
Some say that it is due to outdated hardware. Is it true?</p>

KarthikR: <p>You were able to isolate the crash to the ‘Normalize’ piece or are there other transforms as well? Can you share piece of code around this normalize part?</p>
<p>I have worked on Jupyter notebooks in a system with given GPU and CPU but not seen crashes…</p>

Angry_potato: <p>Before normalise i just read images using opencv. There are no other transforms that i used.</p>

KarthikR: <p>Did you try the same code in another system? Does it work without crash?<br>
Did you reduce the number of images you read at a time and try normalizing?</p>

Angry_potato: <p>The issue is not occuring now,I guess i made some type of error while using the function</p>

KarthikR: <p>good to know, thanks for the update.</p>
Deserialization error

gsouza7: <p>I saved the object of my trainer class using the torch save method. However, when loading the saved file, the depickle process can’t find “__iterator”, which is a private method inside my “Trainer” class. Because of that, the whole loading process fails. Any idea how to solve or bypass it? I just need to recover the data saved into the attributes of that class.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File ".conda/envs/py37/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "&lt;ipython-input-1-a5666d77c70f&gt;", line 1, in &lt;module&gt;
    torch.load("snapshots/model.pth", map_location='cpu')
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 702, in _legacy_load
    result = unpickler.load()
AttributeError: 'Trainer' object has no attribute '__iterator'
</code></pre>

ptrblck: <p>Did you store the <code>Trainer</code> instance directly? Also, could you post its definition?</p>
<p>As explained in the <a href="https://pytorch.org/docs/stable/notes/serialization.html">serialization docs</a> it’s recommended to store the <code>state_dict</code>s instead of the objects, as changes in the source files might yield to such errors and you would also have to make sure to recreate the same folder structure.</p>

gsouza7: <p>Unfortunately, I saved the entire object, and that is what caused the whole problem. Here there is a piece of code that will generate the problem I’m facing right now.</p>
<pre><code class="lang-auto">import torch

class Test:
	def __init__(self):
		self.a = min
		self.b = max
		self.c = self.__private  # buggy

	def __private(self):
		return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")
</code></pre>
<p>However, if you remove the private attribute from the method, you won’t get any error.</p>
<pre><code class="lang-auto">import torch

class Test:
	def __init__(self):
		self.a = min
		self.b = max
		self.c = self.private  # not buggy

	def private(self):
		return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")
</code></pre>
Blob does not exist (caffe2 model exported from pth)

rizal_alfarizi: <p>Hi i’ve done training and export my object detector model to onnx format, but after run prediciton i got an error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "run_ssd_live_caffe2.py", line 85, in &lt;module&gt;
    confidences, boxes = predictor.run({'0': image})
RuntimeError: [enforce fail at ..\caffe2\predictor\predictor.cc:12] blob. Blob does not exist: 0
</code></pre>
<p>this is my code inside opencv loop</p>
<pre><code class="lang-auto">    image = cv2.cvtColor(orig_image, cv2.COLOR_BGR2RGB)
    image = cv2.resize(image, (300, 300))
    image = image.astype(np.float32)
    image = (image - 127) / 128
    image = np.transpose(image, [2, 0, 1])
    image = np.expand_dims(image, axis=0)
    timer.start()
    confidences, boxes = predictor.run({'0': image})
</code></pre>
Calling model.train()

Faria_Huq: <p>Hi, I was confused about whether it is necessary to call model.train() after loading a model if I call optimizer.zero_grad() and optimizer.step() for each batch? I wanted to understand the functionality of model.train().</p>

AladdinPerzon: <p>The role of <code>model.train()</code> and <code>optimizer.zero_grad()</code>, <code>optimizer.step()</code> are a bit different. Some layers behave differently during training vs testing the model. For example dropout and batchnorm layers. That’s why we are careful using .train(), .eval(), as the model performance could worsen considerably if we don’t.</p>
<p>This is a bit different than <code>optimizer.step()</code> which does a gradient descent step based on the gradients computed during the backpropogation for the specific minibatch. After you’ve done <code>optimizer.step()</code> you don’t want these to impact the gradients for the next batch so you reset them by doing <code>optimizer.zero_grad()</code>.</p>

Faria_Huq: <p>thanks for clarifying!</p>
RGB to grayscale shape mismatch

Aliman: <p>I am augmenting my images and part of this process involves using an rgb2gray filter as below:</p>
<pre><code class="lang-auto">def rgb2gray(rgb):
  return np.dot(rgb[...,:3], [0.2989, 0.5870, 0.1140])
</code></pre>
<p>The problem is the original image has shape (64,64,3) while the new image has shape (64,64).This is sadly problematic given that the size of the neural net is 64<em>64</em>3 and I cannot change it.Is there anyway to work around this problem?</p>

ebarsoum: <p>You can duplicate the gray image on each channel.</p>

AladdinPerzon: <p>I think you could also use <code>transforms.Grayscale(num_output_channels=3)</code> and it would do what <a class="mention" href="/u/ebarsoum">@ebarsoum</a> said and duplicate it on each channel so r == g == b.</p>
Model freeze explanation (transfer learning)

rizal_alfarizi: <p>Hi, i have a question, because i’m new in DNN, so i read term about  transfer learning, there are stragtegies to do a transfer learning, like we freeze our base net layer or just classification layer, i want to know if there are simple explanation behind this (would appreciate with math explanation), what happend to the frozen layer while training?<br>
i read github code to freeze some layer like this:</p>
<pre><code class="lang-auto">def freeze_net_layers(net):
    for param in net.parameters():
        param.requires_grad = False
</code></pre>
<p>what is requres_grad? what happen if we set the value true?</p>

AladdinPerzon: <p>In a model you only define the forward propogation and Pytorch handles the backpropogation automatically for you with <code>loss.backwards()</code>. When you’re setting <code>param.requires_grad = False</code> you’re telling Pytorch not to do the backpropogation for those layers.</p>
<p>Essentially what happened to the frozen layer during training is you’re doing the forward propogation part, but Pytorch no longer does the backpropogation part and the parameters aren’t updated. They are “frozen”, usually you would freeze a certain amount of layers and perhaps train the last 3-4 layers. This is also called fine tuning.</p>
Name 'base' is not defined

Michael-J98: <p>I opened the DQN tutorial in colab. And when I ran the following code in the “Input extraction” cell:<br>
“”"<br>
plt.figure()</p>
<p>plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),</p>
<pre><code>       interpolation='none')
</code></pre>
<p>plt.title(‘Example extracted screen’)<br>
“”".<br>
I got this error. Anyone know how to fix it?</p>

ptrblck: <p>It’s a known issue tracked <a href="https://github.com/pytorch/tutorials/issues/915">here</a>.</p>

Michael-J98: <p>Thanks. And <a href="https://colab.research.google.com/drive/18LdlDDT87eb8cCTHZsXyS9ksQPzL3i6H" rel="nofollow noopener">this</a> might be helpful.</p>
nn.Softmax returns very small value

Seannyyy: <p>I’m trying to replicate the GNNExplainer.<br>
When evaluating a trained GNN model inside the GNNExplainer with a subset of the original training dataset, some the of node returns prediction as shown below. As i try to do a softmax on it, it returns a list of very small values, which I won’t be able to use to calculate the loss…</p>
<p>model_ypred: tensor([-2.0499, -0.7565,  5.0231, -1.4513], grad_fn=)<br>
model_ypred(softmax): tensor([8.4303e-04, 3.0732e-03, 9.9455e-01, 1.5340e-03],<br>
grad_fn=)</p>
<p>Sorry I’m very new to GNNs and softmax function… I would appreciate if anyone can give me some advice on the softmax function. Or are the prediction values returned incorrectly?</p>

KFrank: <p>Hello Sean!</p>
<aside class="quote no-group quote-modified" data-username="Seannyyy" data-post="1" data-topic="76186" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seannyyy/48/22593_2.png" class="avatar"> Seannyyy:</div>
<blockquote>
<p>model_ypred: tensor([-2.0499, -0.7565,  5.0231, -1.4513], grad_fn=)<br>
model_ypred(softmax): tensor([8.4303e-04, 3.0732e-03, 9.9455e-01, 1.5340e-03],<br>
grad_fn=)</p>
<p>Or are the prediction values returned incorrectly?</p>
</blockquote>
</aside>
<p>These <code>Softmax</code> values are correct.</p>
<p><code>Softmax</code> converts raw-score <em>logits</em> that run from <code>-inf</code> to <code>inf</code><br>
into <em>probabilities</em> that run from <code>0</code> to <code>1</code>.  Note that your<br>
“model_ypred(softmax)” sum to 1 as proper probabilities should.</p>
<p>The <code>[2]</code> element of “model_ypred(softmax)” is <code>99.46%</code>, very nearly<br>
<code>100%</code>, and corresponds to the largest (and only non-negative) value<br>
in “model_ypred: tensor”.</p>
<p>Whether the results of <code>Softmax</code> should be understood as “large” or<br>
“small” depends on what you do with them.</p>
<p>The specific formula is here:  <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Softmax" rel="noopener nofollow ugc">Softmax</a></p>
<p>Best.</p>
<p>K. Frank</p>

Seannyyy: <p>AHHH… There’s actually no error… 9.9455e-01 is 99.45% which is a the correct value.<br>
My mistake. Thank you Frank.</p>
How to feed ranked multiple labels for a single image to a supervised training pipeline

spaul13: <p>In supervised learning for image classification, we feed the training images with its labels.  <strong>Is it possible for each image, can I provide a series of label ranked in the order of relatedness instead of just one label?</strong></p>
<p>I am using PyTorch for this supervised training. Right now, under each label, I have my training images for that label. Can anyone plz help me saying how can I implement ranked multi-label image training in PyTorch?</p>