akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cleanest way to get grads of grads

Gabriele_Orlando: <p>Hello,<br>
I would need to get the gradient of gradients and I wanted to know what’s the safest way to achieve this.<br>
Basically what I want to do is to optimize the parameters of a model to generate outputs that are in a relative minimum with respect to the parameters themselves. In order to do so, I was planning to optimize a loss that takes as input the grad and the grad of the grad (so minimizing something like -∇∇f / |∇f| ).<br>
Thanks in advance for your help.</p>
torchvision.transforms.ColorJitter: hue != 0 produces error when used on tensor

spadel: <p>I want to make some transformations outside the Dataloader with <code>torchvision.transforms.</code> <code>ColorJitter</code>. However, when I use <code>hue!= 0</code>, I receive:</p>
<pre><code class="lang-auto">a = torch.randn(3, 128, 128)
a_T = torchvision.transforms.ColorJitter(brightness=0.5, contrast=0.5, saturation=0.5, hue=0)(a)    # works
a_t = torchvision.transforms.ColorJitter(brightness=0.5, contrast=0.5, saturation=0.5, hue=0.5)(a)  # error
</code></pre>
<p>TypeError: img should be PIL Image. Got &lt;class ‘torch.Tensor’&gt;</p>
<p>With hue=0 it works, so apparently there seems to be a bug. Can someone confirm this or explain how to work around that issue?</p>

ptrblck: <p>The code works using <code>torchvision==0.8.0.dev20201006</code> and it seems the functionality was added in <a href="https://github.com/pytorch/vision/pull/2566">this PR</a>.<br>
You could update to the latest nightly binary, build from source, or wait for the next stable release. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to freeze part of model in the custom defined nn.Module class

seyeeet: <p>I have a Class like this, I want to freeze Net1 and Net3 during my training.<br>
Would it be possible?</p>
<pre><code class="lang-auto">class MyNetwork(nn.Module):
    def __init__(self, a, b, c, d):
        super(MyNetwork, self).__init__()
        self.a = a
        self.b = b
        self.c = c
        self.d = d              
        self.Net1 = nn.Linear(a, b)
        self.Net2 = nn.Linear(b, c)
        self.Net3 = nn.Linear(c, d)        

    def func1(self, x):
        h_x = self.Net1(x)
        h_x = self.Net2(h_x)
        return h_x
    
    def func2(self, x):        
        h_x2 = self.Net3(x)        
        return h_x2
    
    def forward(self, x):
        y = self.func1(x)
        z = self.func2(y)        
        return z
</code></pre>
<p>I was thinking maybe I can do something like this, but I feel like this is wrong,</p>
<pre><code class="lang-auto">class MyNetwork(nn.Module):
    def __init__(self, a, b, c, d):
        super(MyNetwork, self).__init__()
        self.a = a
        self.b = b
        self.c = c
        self.d = d              
        self.Net1 = nn.Linear(a, b)
        self.Net2 = nn.Linear(b, c)
        self.Net3 = nn.Linear(c, d)        

    def func1(self, x):
        h_x = self.Net1(x).eval()
        h_x = self.Net2(h_x)
        return h_x
    
    def func2(self, x):        
        h_x2  = self.Net3(x).eval()
        return h_x2
    
    def forward(self, x):
        y = self.func1(x)
        z = self.func2(y)        
        return z
</code></pre>

ptrblck: <p><code>model.eval()</code> does not freeze the parameters, but changes the behavior of some modules.<br>
E.g. dropout will be disabled and the running stats in batchnorm layers will be used.<br>
To freeze trainable parameters, you would have to set their <code>.requires_grad</code> attribute to <code>False</code>.<br>
E.g. this would freeze all parameters of <code>Net1</code> and <code>Net2</code>:</p>
<pre><code class="lang-python">model = MyNetwork(a, b, c, d)
for param in model.Net1.parameters():
    param.requires_grad = False
for param in model.Net2.parameters():
    param.requires_grad = False
</code></pre>
How to reset my early stop patience counter when scheduler step() changed the learning rate?

fred: <p>I implemented early stop as a separated class but now i want to reset patience counter when the scheduler changes the learning rate. Is there a way to detect the scheduler step change?</p>

ptrblck: <p>The number of “bad epochs” will already be reset as seen <a href="https://github.com/pytorch/pytorch/blob/1b31ed3ad60fef347ef8043d72209d370d56251b/torch/optim/lr_scheduler.py#L641-L644">here</a> so if you haven’t changed the <code>cooldown</code> the scheduler should again apply its patience schedule.</p>

fred: <p>Sorry, i was not clear enough. I need to reset the patience counter of my early stop object, but for that, i need to detect the “reset” of the scheduler patience.</p>
NaN loss and loss scale below 0.0001

karan_purohit: <p>I am finetuning wav2vec2 on my own data. I am running this on k80 and as it doesn’t support fp16. When I was training with fp16 flag got <strong>loss scale reached to 0.0001</strong></p>
<pre><code class="lang-auto">FloatingPointError: Minimum loss scale reached (0.0001). Your loss is probably exploding. Try lowering the learning rate, using gradient clipping or increasing the batch size.
</code></pre>
<p>Then I switched to FP32 but loss became <strong>nan</strong> this time:<br>
getting log:</p>
<pre><code class="lang-auto">/data/fairseq/fairseq/utils.py:306: UserWarning: amp_C fused kernels unavailable, disabling multi_tensor_l2norm; you may get better performance by installing NVIDIA's apex library
  "amp_C fused kernels unavailable, disabling multi_tensor_l2norm; "
/data/fairseq/fairseq/utils.py:306: UserWarning: amp_C fused kernels unavailable, disabling multi_tensor_l2norm; you may get better performance by installing NVIDIA's apex library
2020-10-01 10:25:50 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:25:50 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:25:50 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:25:50 | INFO | train | {"epoch": 73, "train_loss": "nan", "train_ntokens": "15558.8", "train_nsentences": "427.098", "t
rain_nll_loss": "nan", "train_wps": "2443", "train_ups": "0.16", "train_wpb": "15558.8", "train_bsz": "427.1", "train_num_updates": "
9291", "train_lr": "1e-08", "train_gnorm": "nan", "train_loss_scale": null, "train_train_wall": "867", "train_wall": "0"}
2020-10-01 10:25:50 | INFO | fairseq.trainer | begin training epoch 74
2020-10-01 10:40:41 | INFO | fairseq_cli.train | begin validation on "valid" subset
2020-10-01 10:42:40 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:42:40 | WARNING | root | NaN or Inf found in input tensor.
2020-10-01 10:42:40 | INFO | valid | {"epoch": 74, "valid_loss": "nan", "valid_ntokens": "2570.42", "valid_nsentences": "71.4286", "v
alid_nll_loss": "nan", "valid_uer": "100", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2780.3", "valid_wpb": "2570.4", 
"valid_bsz": "71.4", "valid_num_updates": "9454", "valid_best_wer": "100"}
</code></pre>
<p>Any suggestions on how to overcome such conditions.</p>

ptrblck: <p>Are you using <code>opt_level='O0'</code> in <code>apex.amp</code>? If so, could you remove <code>apex</code> and run the model in “pure” <code>float32</code> and check if you are still seeing invalid outputs?<br>
If so, could you check, if the inputs are containing valid values via <code>torch.isfinite(input).all()</code>?</p>
More than two optimizer with multiple backward

hyuntae: <p>Hello I have a question about backward with multiple loss and three optimziers.</p>
<p>My code is</p>
<pre><code class="lang-auto">first_model = Model_first()
second_model = Model_second()
thrid_model = Model_third()

first_opt = torch.optim.SGD(first_model.parameters)
second_opt = torch.optim.SGD(second_model.parameters)
third_opt = torch.optim.SGD(third_model.parameters)

first_result = first_model(data)
second_result = second_model(data)
third_result = third_model(data)

first_opt.zero_grad()
second_opt.zero_grad()

loss1 = torch.nn.MSELoss(first_result, second_result)
loss1.backward()

frist_opt.step()    # update only first model with loss1



third_opt.zero_grad()
loss2 = torch.nn.MSELoss(second_result, third_result)

second_opt.step()  # updated second model with loss1 + loss2
third_opt.step()    # update third model with loss2

</code></pre>
<p>I wonder that if those three optimizers are updated like the annotations I wrote.<br>
Especially second and third. (<strong>second_opt.step(), third_opt.step()</strong>)<br>
And could you explain how I check the answer?</p>

ptrblck: <p>The optimizers don’t get valid parameters and should thus raise an error, since you would have to call the <code>parameters</code> method:</p>
<pre><code class="lang-python">first_opt = torch.optim.SGD(first_model.parameters())
second_opt = torch.optim.SGD(second_model.parameters())
third_opt = torch.optim.SGD(third_model.parameters())
</code></pre>
<p>The loss calculation should also raise an error, since you are not creating the criterion, but try to pass the tensors to the constructor.<br>
Replace it with:</p>
<pre><code class="lang-python">criterion = torch.nn.MSELoss()
loss1 = criterion(first_result, second_result)
</code></pre>
<p>I’m also not sure why your code didn’t raise any errors.</p>

hyuntae: <p>Sorry for the confusion.<br>
I didn’t care about grammatical errors because I was writing simply.<br>
My real code is written with ‘model.parameters()’ and ‘troch.nn.MSELoss()(restult1, result2)’<br>
There is no error in my code, because I wrote the code like you mentioned. Sorry…</p>
<p>But I wonder that if there is no grammatical error, the backward step is really proceeded like annotations.</p>
Using classifiers in scikit-learn with PyTorch

somnath: <p>3m<br>
How can we use machine learning classifiers like SVM, Decision Tree, Random Forest, etc in PyTorch? Is there any library available?</p>

ptrblck: <p>You could apply these techniques using <a href="http://scikit-learn.org/stable/index.html">scikit-learn</a>.<br>
They work an numpy arrays, so if you would like to train an SVM on top of a CNN, you could just get the numpy array from the <code>tensor</code>:</p>
<pre><code>output = pytorch_model(x)
output = output.numpy()</code></pre>

somnath: <p>Yes, scikit-learn is an option. But it does not support GPU. Is there any way to use those classifiers on a GPU?</p>

ptrblck: <p>I think xgboost has some option to create the trees on the GPU, but I’m not sure, if a lot of these methods get a significant performance boost on the GPU.<br>
Sebastian Raschka, as an author of a nice book on this topic, (<a class="mention" href="/u/rasbt">@rasbt</a> ) will most definitely know more on this topic. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

rasbt: <p>Scikit-learn currently doesn’t have GPU support and is also not planning to add GPU support in foreseeable future. The reason is that there is no benefits to use a GPU for (most of) the algorithms that it implements. I would suspect it would even make things slower compared to efficient C++ libraries like LIBLINEAR and LIBSVM etc. For distributed parallelism (e.g., distributing the computation across multiple machines) there’s dask-ml, which has an API similar to sklearn: <a href="https://dask-ml.readthedocs.io/en/latest/" rel="nofollow noopener">https://dask-ml.readthedocs.io/en/latest/</a></p>

somnath: <p>Sorry for the late reply. Thanks for letting me know about desk-ml,</p>

faraz_moradi: <p>Hi,<br>
we are using LDA (linear discriminant analysis) in network forwardness and we net to keep the gradients but sci-kit learn library does not work with torch tensors with grad. is there any way to use this library in our code?</p>

ptrblck: <p>I don’t know how the LDA is implemented in scikit-learn, but you could either try to reimplement it using PyTorch operations and tensors so that Autograd could create the computation graph and would be able to automatically compute the gradients.<br>
If that’s not possible e.g. since a specific numpy operation is not implemented in PyTorch, you would have to derive the backward function manually and could implement a custom <code>autograd.Function</code> as described in <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">this tutorial</a>.</p>
Making custom loss cuda compatible

SpandanMadan: <p>Hi,</p>
<p>I am using a custom loss function in PyTorch. When using a standard loss function like CrossEntropyLoss() I can simply do a .cuda(). How do I take my custom loss on cuda() ?</p>
<p>The tensors and model are both on cuda, and nvidia-smi shows memory being used too. But I’m not sure about the importance of criterion.cuda() and how to do that with my own loss function.</p>
<p>For reference, here’s the loss -</p>
<pre><code>class SpandanLoss(torch.nn.Module):
'''
custom loss for our model
'''

def __init__(self):
    super(SpandanLoss, self).__init__()
    
def forward(self, model_output,margin = 0.8):
    mis_classifs = 0
    classifs=[]
    total_loss = 0
    for i in range(0,len(model_output),2):
        true_val = model_output[i].data
        false_val = model_output[i+1].data
        total_loss += torch.max(torch.zeros(1).cuda(),margin - true_val + false_val)
                                
    return Variable(total_loss,requires_grad = True)
</code></pre>
<p>Thanks!</p>

SpandanMadan: <p>Still waiting on someone to reply on this! Any help appreciated!</p>

smth: <p>your custom loss seems to already be cuda ready. In general, you dont need to call <code>.cuda()</code> on a model or loss unless it has some learnable <code>nn.Parameter</code> parameters (for example in a ResNet or RNN).</p>

SpandanMadan: <p>Thanks! and if it does have learnable parameters? Because for a custom function, I can’t do a .cuda()</p>

raningtky: <p>I change every Variable in the custom loss function into cuda. This method is slow but it works for me. I hope somebody else can come up with a fast way.</p>

mathematics: <p>Creating <code>device</code> argument in <code>__init__</code> method and passing<code>.to(device)</code> to all tensors and problem solved in my case.</p>
Only 100 predictions outputs for "TorchVision Object Detection Finetuning Tutorial""

Emil_Hovad: <p>When I follow the tutorial “<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a>” and I make the number of predictions it is only 100? is that due to the setting in pytorch/torchvision, it is not the model?<br>
img, _ =dataset_test[1]# dataset[0]</p>
<h1>put the model in evaluation mode</h1>
<p>model.eval()</p>
<p>with torch.no_grad():<br>
prediction = model([img.to(device)])</p>
<p>prediction[0][“scores”]<br>
<span class="hashtag">#this</span> only gives a 100, which is not enough for the application I will use it for.</p>

Emil_Hovad: <p>all the 100 predictions have a confidence threshold of above 0.5, so it might be a thresh hold setting?</p>

ptrblck: <p>That might be the case and you could try to change these threshold with <a href="https://github.com/pytorch/vision/blob/6e639d3e49371a509235201cb3f335b1c5cac0e3/torchvision/models/detection/faster_rcnn.py#L67-L88">these arguments</a>.</p>

Emil_Hovad: <p>Thanks a lot, this was set to 100 by default, which might make sense in most cases.<br>
box_detections_per_img (int): maximum number of detections per image, for all classes.<br>
Initialize the model with, say 500 in this way<br>
model(box_detections_per_img=500)</p>
<p>kind regards Emil</p>
PyTorch C++ torch::var(tensor)

donJuan: <pre><code class="lang-auto">torch::Tensor out = torch::randn({100, 12}).toType(torch::kFloat32);
std::cout &lt;&lt; torch::mean(out, 0).sizes() &lt;&lt; "\n";
std::cout &lt;&lt; torch::var(out, 0).sizes() &lt;&lt; "\n";

[12]
[]

</code></pre>
<p>Anyone know whats wrong?</p>

albanD: <p>Hi,</p>
<p>I guess your compiler is confusing <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1aee550fa3eda9d4958b05ec93026d946f.html?highlight=var">this</a> overload and <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1a2d4a5a2ffc3a0677dd7eca056026b8f6.html?highlight=var">this</a> overload.<br>
I guess that passing all the arguments would solve this: <code>torch::var(out, 0, true, false)</code></p>

donJuan: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="98432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>torch::var</code></p>
</blockquote>
</aside>
<p>thanks that solve it</p>
Loss does not decrease when the model is called twice

hoangcuong2011: <p>Hello everyone, I have a model, say, net, where its forward function contains an if else statement inside. It is something like this:</p>
<pre><code class="lang-auto">def forward(self, x):
   if first case:
       self.module_1() #run this module only
   else:
       self.module_2() #run this module only
</code></pre>
<p>The special thing is that I need to call the forward function of net two times. Moreover, the later one uses the first one’s output as the input. It is something like this:</p>
<pre><code class="lang-auto">output = net(input for first case)
second_output = net(output from the first call as the input)
</code></pre>
<p>While the code is running OK, what I found strange with this is that when I run training the loss does not decrease. This means something is wrong but I don’t know what it is and why.</p>
<p>Any explanation? Thanks.</p>

hoangcuong2011: <p>Actually I found the real reason behind this. In between these two lines:</p>
<blockquote>
<p>output = net(input for first case)<br>
second_output = net(output from the first call as the input)</p>
</blockquote>
<p>I actually have several extra simple operations over the output tensor in the middle. Strangely enough, with that back-propagation does not work properly. My solution is to wrap these extra operations into a nn.module. With this it works normally again.</p>
<p>That is it. I hope my experience is useful to someone.</p>
Single dataloader on datasets with different transforms

l_s: <p>I have 2 datasets, in which each has a different sequence of transforms applied e.g., the first transform differs in whether the data needs to be converted to a PILImage. Transforms are applied in their <strong>getitem</strong> function definitions.</p>
<p>I am able to construct a data loader on each data separately and train properly.</p>
<p>I am trying to construct a single data loader with the 2 datasets merged and interleaved randomly.</p>
<p>I used a ConcatDataset to combine the 2 datasets, and passed this into the dataset loader (where the instances get shuffled), but get the following error after 1 epoch appears to complete successfully. I’m not sure why this error is occurring since the dataloader works when constructed using each dataset separately.</p>
<p>AttributeError: Caught AttributeError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/home/.conda/envs/di_env/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py”, line 178, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/home//.conda/envs/di_env/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 47, in fetch<br>
return self.collate_fn(data)<br>
File “/home//.conda/envs/di_env/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py”, line 79, in default_collate<br>
return [default_collate(samples) for samples in transposed]<br>
File “/home//.conda/envs/di_env/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py”, line 79, in <br>
return [default_collate(samples) for samples in transposed]<br>
File “/home//.conda/envs/di_env/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py”, line 52, in default_collate<br>
numel = sum([x.numel() for x in batch])<br>
File “/home//.conda/envs/di_env/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py”, line 52, in <br>
numel = sum([x.numel() for x in batch])<br>
AttributeError: ‘int’ object has no attribute ‘numel’</p>

ptrblck: <p>One of your datasets seem to return an <code>int</code> where a tensor is expected.<br>
You could add print statements into the <code>Dataset.__getitem__</code> methods, check the type of all returned objects, and make sure they are tensors.</p>
Specific sequence of learning rates

tiger17: <p>What is the best way to specify a learning rate per batch iteration? Let’s say I have (an independently constructed)  list [a1,a2,…,a_k]. What I need is for the first batch to have lr = a1, for the second lr = a2 and so on. Epochs are not that important (i.e. that batch = N is the same as batch = 0 is not important).</p>
<p>Thanks!</p>

ptrblck: <p>You could use e.g. <code>LambdaLR</code> and call <code>scheduler.step()</code> in each iteration as shown in this small code snippet:</p>
<pre><code class="lang-python">class MyLRS(object):
    def __init__(self, lrs):
        self.lrs = lrs
        
    def get_lr(self, epoch):
        return self.lrs[epoch]

lrs = MyLRS(lrs=torch.linspace(1.0, 0.01, 100))
model = nn.Linear(1, 1)
optimizer = torch.optim.Adam(model.parameters(), lr=1.)

for epoch in range(2):
    scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lrs.get_lr, last_epoch=-1, verbose=True)
    for batch_idx in range(99):
        scheduler.step()
        print(scheduler.get_last_lr())
        print(optimizer.param_groups[0]['lr'])
</code></pre>
<p>Note that you don’t necessarily need to write a custom class for it and could also use a <code>lambda</code> function directly, if your code allows it.</p>
<p>Alternatively you could manually try to change the learning rate in the optimizer or use another scheduler, if your learning rates follow a specific pattern such as a multiplicative decay.</p>
Sharing Information between DDP Processes

zphang: <p>Is there a good way to share information between DDP processes? Even if it’s just something from process-0 to the other processes.</p>
<p>My use-case is that I need to coordinate some data-loading/sampling across my different processes, and it would be good if I could, for example, determine what to sample in process-0 and distribute that information to the other processes.</p>

seungjun: <p>You can use <a href="https://pytorch.org/docs/stable/distributed.html#point-to-point-communication" rel="noopener nofollow ugc">point-to-point communication</a> or <a href="https://pytorch.org/docs/stable/distributed.html#synchronous-and-asynchronous-collective-operations" rel="noopener nofollow ugc">collective operations</a>.</p>
<p>For example, a tensor could be sent by <code>torch.distributed.send</code> and received by <code>torch.distributed.recv</code> functions by specifying the target ranks.</p>
<p>You can choose to <code>broadcast</code> or <code>reduce</code> if you wish. I usually use <code>torch.disbtibuted.all_reduce</code> function to collect loss information between processes.<br>
Example <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/loss/__init__.py#L259" rel="noopener nofollow ugc">here</a>.</p>
<p>If you use nccl backend, you can only use <code>CudaTensor</code> for communication.</p>

rvarm1: <p>In addition to the above response, standard <code>multiprocessing</code> communication methods such as <code>Queue</code> or <code>mp.Manager</code> should work as well: <a href="https://docs.python.org/3/library/multiprocessing.html" rel="noopener nofollow ugc">https://docs.python.org/3/library/multiprocessing.html</a>, assuming your processes are on the same machine.</p>
<p>If your particular use case is around data sampling, you could also look into <code>DistributedSampler</code>, which will automatically partition data out to DDP ranks for you: <a href="https://github.com/pytorch/pytorch/blob/c1e6592964261d2856c84e166a0989684f946697/torch/utils/data/distributed.py#L12" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/c1e6592964261d2856c84e166a0989684f946697/torch/utils/data/distributed.py#L12</a></p>
<p>Finally, we have also have APIs such as <code>all_gather_object</code> and <code>broadcast_object_list</code> which can be used to communicate general picklable python objects across ranks: <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/distributed_c10d.py#L1279" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/distributed/distributed_c10d.py#L1279</a>. These APIs are quite new and subject to significant changes, but may be useful for your use case.</p>
Katz's backoff implementation

aclifton314: <p>I’ve been staring at this <a href="https://en.wikipedia.org/wiki/Katz%27s_back-off_model" rel="noopener nofollow ugc">wikipedia article</a> on Katz’s backoff model for quite some time. I’m interested in trying to implement it into my pytorch model as a loss function. I have no sample code for the loss unfortunately. To make things easier, I could set <code>d = 0</code> and start with a trigram model.</p>
<p>I was wondering if anyone out there had implemented something like this? I was thinking for a given sentence I would calculate the katz probability for the trigrams in the sentence then take the negative log of their sum as the loss. Has anyone done something like this or something similar?</p>
<p>Here’s a simplification of my thought process:</p>
<pre><code class="lang-python">x = torch.tensor(input_text) #here the input text is a training example.
new_text = model(x) #new_text is generated text by the model
loss = katz_loss(new_text) #the katz backoff probability is calculated for each trigram in new_text. The final loss will be the negative log of their sum.
loss.backward()
</code></pre>
<p>Thanks in advance for your help!</p>
Question regarding PyTorch's Backwards Compatibility guarantee between releases

vdantu: <p>Does PyTorch guarantee backwards compatibility between minor releases (ie., 1.X-&gt;1.Y for all Y &gt; X) for models/operator behaviors? In other words, would all the models defined and saved using an older 1.X version be loadable and runnable, with same behavior, using latest 1.Y version? Or, is this best effort which implies that some models will fail with upgrade of PyTorch version from 1.X-&gt;1.Y</p>
<p>Similarly, does PyTorch JIT guarantee backwards-compatibility for traced/scripted models between releases?</p>
<p>I didn’t find any documentation on backwards compatibility guarantee and I see backwards incompatible changes being noted in the release notes of each version release. Could someone help answer this or point to a document?</p>

eellison: <p>Yes, we do preserve backwards compatibility</p>
Confusion with __len__ for custom dataloading

peepeepoopoo: <p>I’m currently trying to load datasets that have separate sub-folders( as video sequences), and the number of images under each sub-folder varies. I assume that _<em>len</em>_ is for the length of the entire datasets. But since the images under training are separated in sequences under different folders, would this effect how the _<em>len</em>_ works in PyTorch?</p>

ptrblck: <p>It depends how you are loading each sample.<br>
The <code>__len__</code> method returns the length of the complete <code>Dataset</code> and thus also defines the used indices (passed into <code>__getitem__</code>).<br>
E.g. if you are loading single images via the passed <code>index</code> in <code>__getitem__</code>, the length should define the total number of images in your dataset.</p>
Create new convolution filter

khalfinv: <p>Hi,<br>
I want to implement a new convolution filter, the Edge-to-Edge Filter represented in this article : <a href="https://kawahara.ca/convolutional-neural-networks-for-adjacency-matrices/" rel="noopener nofollow ugc">https://kawahara.ca/convolutional-neural-networks-for-adjacency-matrices/</a>.<br>
But I have no idea how to do this, that I will able to use it as a layer in my model like I would use the Conv2d.</p>
<p>I would love guidance on how to do this.</p>
<p>Thank you in advance !</p>
Expected input[1, 3, 224, 224] to have 4 channels, but got 3 channels instead

shivsd: <p>Trying to convert a pytorch model to onnx, i have create a dummy_input of shape (1, 3, 224, 224) and while running the below code i get following error</p>
<p><strong>RuntimeError: Given groups=1, weight of size 32 4 3 3, expected input[1, 3, 224, 224] to have 4 channels, but got 3 channels instead</strong></p>
<p>dummy_input.shape = (1, 3, 224, 224)</p>
<p>Here is the code :</p>
<pre><code class="lang-auto">    input_shape = (3, 224, 224)
    model_onnx_path = "torch_model.onnx"
    model.train(False)

    # Export the model to an ONNX file
    dummy_input = Variable(torch.randn(1, *input_shape))
    print('dummy_input shape : ', str(dummy_input.shape))
    output = torch.onnx.export(model,
                              dummy_input,
                              model_onnx_path,
                              verbose=False)
    print("Export of torch_model.onnx complete!")
</code></pre>
<p>Is there anything wrong in this code, please correct me.</p>

ptrblck: <p>A conv layer (probably the first one) expects 4 input channels, while your data only contains 3 channels.<br>
Change the data shape to <code>[batch_size, 4, 224, 224]</code> or the <code>in_channels</code> to <code>3</code> in the corresponding conv layer.</p>

shivsd: <p>Thanks it worked with 4 channel.</p>
Backpropagation for sparse matrix seems problematic (colab provided)

Jason_Bian: <p>Hi, everyone,</p>
<p>I find when I index the sparse matrix, it performs well in the forward, but something wrong in the backpropagation. I guess the  in the sparse matrix setting is problematic. The gradients for the sparse matrix are empty.</p>
<p>To reproduce, I put up a toy case in colab here: <a href="https://colab.research.google.com/drive/1RgVT2heWNOZZi2YZeSi07Agq0k0nQl_I?usp=sharing" rel="noopener nofollow ugc">colab toy case</a><br>
But the key part is:</p>
<pre><code class="lang-python">#construct sparse mat
ind = torch.LongTensor([[0, 1, 1,3],
                          [2, 1, 2,3],
                    ])
vals = torch.FloatTensor([3, 4, 5,9]).requires_grad_(True)
sp1 = torch.sparse.FloatTensor(ind, vals, torch.Size([5,5])).to(device)

sp1 = sp1.detach().requires_grad_(True)
losses = []
for i in range(sp1.shape[0]):
    loss = sp1[i].to_dense().sum()
    losses.append(loss)
l = sum(losses)

# check the gradients of sp1
sp1.register_hook(lambda grad: print("==grad== sp1 :\n", grad)) # sp1's gradients are all zeros, which is problematic

l.backward()
</code></pre>
<p>printed output:</p>
<pre><code class="lang-python">==grad== sp1 :
 tensor(indices=tensor([], size=(2, 0)),
       values=tensor([], size=(0,)),
       device='cuda:0', size=(5, 5), nnz=0, layout=torch.sparse_coo)
</code></pre>
<p>Thanks for your help! If you need more info let me know.</p>
Pytorch loss error help please

PARDUS: <pre><code class="lang-auto">for epoch in range(0, 300):
    out1 = net(X)
    optimizer.zero_grad()
    lossf.backward()
    optimizer.step()
    print(lossf)
</code></pre>
<p>“Found dtype Long but expected Float” help!</p>

neem-41: <p>Hi <a class="mention" href="/u/pardus">@PARDUS</a>, Most likely your tensors are Long type but should be Float type. Could you share part of the code that has loss, optimizer, network initialization and your input for the network.</p>

PARDUS: <pre><code class="lang-auto">loss = nn.BCELoss()
optim = torch.optim.Adam(net.parameters())
</code></pre>
<p>(X: tensor([44., 76., 47.,  …, 21., 68., 46.]), y: tensor([0, 1, 0,  …, 1, 0, 1]))</p>

PARDUS: <pre><code class="lang-auto">lossf = loss(out, y)
</code></pre>

neem-41: <p>Try changing the input tensor X to type float. Something like this:</p>
<pre><code class="lang-python">x = torch.tensor([44.,76.,57.,...,21.,68.,46.], dtype=torch.float32)
</code></pre>
<p>Could you also copy and paste what the full error - that has information on which line of code is giving - is? (full traceback of error is better)</p>
Getting memory at inference

i-cant-code: <p>Hello,</p>
<p>I am trying to figure out how much RAM is used while running a trained model on a single input like:</p>
<p>model = <span class="hashtag">#model</span> definition and loading trained weights<br>
x = image <span class="hashtag">#load</span> image<br>
y = model(x) <span class="hashtag">#run</span> model on image</p>
<p>I want to basically get the memory usage of this script</p>

tom: <p>On Linux, I’d probably use <code>/usr/bin/time -v my_script.py</code> (you need the path for time because you don’t want the builtin function. This will report the <code>Maximum resident set size (kbytes):</code>.</p>

i-cant-code: <p>is this approach recommended over using pytorch functions like those described here?:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/cuda.html#memory-management" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/cuda.html#memory-management" target="_blank" rel="noopener nofollow ugc">torch.cuda — PyTorch 1.6.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


i-cant-code: <p>specifically I am trying to find out how much memory is used on the CPU and GPU for this program</p>
Tracking the parameter gradients of different inputs

cpsanders: <p>Hello, I’ve defined a simple network given by:</p>
<pre><code class="lang-auto">class RBM_Network(nn.Module):

    def __init__(self, n_visible, n_hidden):
        super(RBM_Network, self).__init__()

        self.n_visible = n_visible
        self.n_hidden = n_hidden

        # define network layers 
        self.hidden = nn.Linear(n_visible, n_hidden)
        self.output = nn.Linear(n_hidden, 1)
        self.sigmoid = nn.Sigmoid()


    # forward function is our feed forward algorithm
    def forward(self, s):

        s = self.hidden(s)
        s = self.sigmoid(s)
        s = self.output(s)
        s = self.sigmoid(s)

        return s
</code></pre>
<p>The network’s input is a spin state given by either [0,0], [0,1], [1,0], or [1,1]. It outputs a single value representing a wavefunction coefficient. Based on the way I’ve implemented the network so far, it has 9 parameters. When passing one of these states through the network and runing <code>.backward()</code> on the output, I get a unique set of parameter gradients for the given state (which makes sense). I’m trying to store each of these unique parameter gradients in a list for each spin state. This seems like it should be relatively easy, but it appears the the parameter gradients are being overwritten when I try and append them to a simple list, <code>param_grads</code>. My implementation code is given below. I’m expecting <code>param_grads</code> to be a list of 4 sub-lists (one sub-list for each spin state), each sub-list containing 9 unique tensors that store the gradients of the network output wrt each input. However, <code>param_grads</code> just ends up being a list of 4 identical sub-lists (the parameter gradients given by the most recent state passed through the network). I appreciate any help here, I’m relatively new to Pytorch and might be missing something simple.</p>
<pre><code class="lang-auto">import torch
import numpy as np 
from RBM_Network import RBM_Network

inputs = 2
hidden_nodes = 2
spin1 = torch.tensor([0,0], dtype=torch.float)
spin2 = torch.tensor([0,1], dtype=torch.float)
spin3 = torch.tensor([1,0], dtype=torch.float)
spin4 = torch.tensor([1,1], dtype=torch.float)
spins = [spin1, spin2, spin3, spin4]

# initialize network
network = RBM_Network(inputs, hidden_nodes)
param_grads = []
psi_omega = []

# pass spin states through network to obtain coefficients for psi_omega
# track parameter gradients for each spin state in param_grads
for spin in spins:
    
    network.zero_grad()
    psi = network(spin)
    psi.backward()
    psi_omega.append(psi)
    
    current_grads = []
    for param in network.parameters():
        current_grads.append(param.grad)
        
    param_grads.append(current_grads)
    print(param_grads)
    print("\n")
</code></pre>

googlebot: <p>these tensors are reused, so you need to do param.grad.clone()</p>
IOU Loss function implementation in Pytorch

amitkayal: <p>Hello All,</p>
<p>I am trying to implement IOU as loss function for my semantic segmentation problem which has multiple classed. I am looking for pytorch implementation and found the post <a href="https://discuss.pytorch.org/t/understanding-different-metrics-implementations-iou/85817" class="inline-onebox">Understanding different Metrics implementations (IoU)</a> but it does not support multiple class. Can anyone point me towards IOU Pytorch implementation implementation as loss function? Thanks</p>
Saved model for inference

Kenzo: <p>Hi all,</p>
<p>I am new to pyTorch and curious to know how can I use the saved model during training for inference (Testing), especially for the GAN saved model.</p>
<p>I am following this repository [<a href="https://github.com/eriklindernoren/PyTorch-GAN" rel="nofollow noopener">https://github.com/eriklindernoren/PyTorch-GAN</a>]. In the all GAN models which implemented here the coder saved the whole models and now I want to use the saved model (for example, Vanilla GAN model) for inference but don’t have Idea how to do that,</p>
<p>Any help will be appreciated,<br>
Thanks in advance!</p>

sun: <p><a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a></p>

Kenzo: <p>I already know how to load the saved model with torch.load() and my problem is not loading the model. For example, in CycleGAN one domain is translating to other domain (apple2orange). I trained my model based on training data and saved my model parameters and everything. Now I want to use the saved model and give a single image (apple image which network never has seen) to saved model and get the output in other domain.</p>

bobobo-prog: <p>Hey did you figure out how to do it ?? im stuck on the same thing please help</p>
Difference between different checkpoint methods

gnadaf: <p>What is the difference between <strong>torch.save()</strong> and <strong>torch.utils.checkpoint.checkpoint()</strong></p>

albanD: <p>Hi,</p>
<p>The first one can be used to save some data in a file on disk.<br>
The second one is used to perform a training trick to reduce the memory usage.<br>
You can check the doc for each for more details.</p>
Error of backward with two optimizers

marka17: <p>Hi!<br>
I have faced a frequent problem:</p>
<blockquote>
<p>one of the variables needed for gradient computation has been modified by an inplace operation:</p>
</blockquote>
<p>but in an unusual case. The code below reproduces the error. Error occur in <strong>output_2.backward()</strong></p>
<p>So, what’s wrong?</p>
<pre><code class="lang-auto">import torch
from torch import nn

model_1 = nn.Linear(1, 1, bias=False)
model_2 = nn.Linear(1, 1, bias=False)

opt_1 = optim.Adam(model_1.parameters())
opt_2 = optim.Adam(model_2.parameters())

input = torch.Tensor([1.])

output_1 = model_1(input)

opt_1.zero_grad()
output_1.backward(retain_graph=True)
opt_1.step()

output_2 = model_2(output_1)

opt_2.zero_grad()
output_2.backward()
opt_2.step()
</code></pre>

albanD: <p>Hi,</p>
<p>The problem comes from the fact that the backward for <code>output_2</code> tries to propagate all the way to <code>model_1</code>. But the parameters for <code>model_1</code> were modified when you did <code>opt_1.step()</code> hence the error that you see.<br>
If you just don’t want this backward to go all the way to <code>model_1</code>, you can do <code>output_2 = model_2(output_1.detach())</code> to break the graph.</p>
About converting loss.data.cpu().numpy() from pytorch to libtorch

Chen0729: <p>Hi, All, I have met an issue about converting the following from pytorch to libtorch (C++)</p>
<p>In pytorch it is:<br>
return loss.data.cpu().numpy()</p>
<p>I write in C++ as:<br>
return loss.data.cpu().numpy();</p>
<p>I get error no member “numpy()” for at::Tensor</p>
<p>Any suggestions ?<br>
Much appreciated</p>

ptrblck: <p>You cannot transform the tensor to a numpy array, since numpy doesn’t have a C++ API.</p>

Chen0729: <p>Thanks.</p>
<p>So I will just convert it as: return loss.data().cpu();</p>
<p>Are they the same in this case ?</p>
<p>Best,<br>
Xu</p>

a_d: <p>Yes, but with the catch that you would still be returning a tensor.</p>
Mismatch of kernel size

111353: <p>This code work well, no problem.<br>
However I have a question because The size of the image is not divisible by the kernel.<br>
(Conv2d in pytorch on colab.)</p>
<p>The code is here <a href="https://github.com/YutaroOgawa/Qiita/blob/master/MNIST_IIC.ipynb" rel="noopener nofollow ugc">https://github.com/YutaroOgawa/Qiita/blob/master/MNIST_IIC.ipynb</a></p>
<p>I could see the size,<br>
torch.Size([512, 1, 28, 28])<br>
This code was added by myself.<br>
[for data, target in train_loader:<br>
print(data.size())]</p>
<p>And the first conv2d layer is<br>
self.conv1 = nn.Conv2d(1, 128, 5, 2, bias=False)</p>
<p>Therefore, the output pixels are calculated by<br>
output pixel = (pixel - kernel size) / stride + 1</p>
<p>but (28 - 5) / 2 become odd and could not divisible.<br>
I couldn’t understand this.</p>
<p>Thank you for reading.</p>

KarthikR: <p>The value is rounded as we see below:</p>
<p>import torch<br>
import torch.nn as nn</p>
<p>input = torch.randn(512, 1, 28, 28)<br>
print(input.shape)</p>
<p>conv1 = nn.Conv2d(1, 128, 5, 2, bias=False)<br>
print(conv1(input).shape)</p>

111353: <aside class="quote no-group" data-username="KarthikR" data-post="2" data-topic="98389">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/eada6e/48.png" class="avatar"> KarthikR:</div>
<blockquote>
<p>import torch<br>
import torch.nn as nn</p>
<p>input = torch.randn(512, 1, 28, 28)<br>
print(input.shape)</p>
<p>conv1 = nn.Conv2d(1, 128, 5, 2, bias=False)<br>
print(conv1(input).shape)</p>
</blockquote>
</aside>
<p>Thank you so much.<br>
I could understand.<br>
Also, Thank you for telling sample code, it was very useful for me.</p>
<p>Thank you.</p>
How to fix model's parameter

Kyle: <p>I’m learning doubleDQN, now I want to calculate a Variable by using a model, and then backward a loss calculated from the Variable, but I do not want to optimize the model’s parameters, How to do that?</p>

Abhishek_Pal: <p>you just want to run one iteration?</p>

Cadene: <p>Two different solutions you can try.</p>
<ol>
<li>You can specify to not process the gradient on a Variable with : <code>variable.requires_grad = False</code><br>
Then use your optimizer as:</li>
</ol>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), learning_rate)
</code></pre>
<ol start="2">
<li>Process the gradient on all your Variables and choose which one you want to update with your optimizer.</li>
</ol>
<pre><code class="lang-auto">optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>

HaFred: <aside class="quote no-group" data-username="Cadene" data-post="3" data-topic="3885">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cadene/48/7036_2.png" class="avatar"> Cadene:</div>
<blockquote>
<p><code>requires_grad</code></p>
</blockquote>
</aside>
<p>Hi Cadene,<br>
May I know what is the  <code>lambda p: p</code> here and how does it work? Shouldn’t it be <code>lambda variable: variable.requires_grad</code>?</p>
How pytorch internally launches cuda kernels

softplus: <p>Nvidia GPUs are only able to launch a limited number of threads (ex. 1024 for 1080ti) in parallel. I was wondering how pytorch adjusts grid and block size to deal with this limitation when input size is beyond the maximum parallel capability. For example, when input size is (128, 3, 270, 360) (in NCHW), even if it can fit into GPU memory, it cannot be processed in parallel altogether. I can think of splitting the large input into multiple chunks so that each chunk can be executed in parallel in GPU, and execute all of input chunks iteratively. But i believe doing this optimally maximizing GPU utilization is not that simple. How is pytorch dealing with this issue? It would be very much appreciated if any of you guys can point out source codes that i can refer to.</p>

albanD: <p>Hi,</p>
<p>You can simply have your kernel work on more than one datapoint in a for-loop.<br>
Like <a href="https://github.com/pytorch/pytorch/blob/2254e5d9766f5c9ef52db2e6a8b6002ff0604d05/aten/src/ATen/native/cuda/DistanceKernel.cu#L196">this one</a> for example.</p>
<p>But I am not a specialist in that domain and there are a lot of subtle things to know to be able to do that most efficiently. I would advise you to check a cuda class online to learn how to design such algorithms efficiently.</p>

googlebot: <p>From what I’ve seen, in pytorch “elementwise” kernels just use hardcoded 512 threads per block (and data is processed like a flat 1d view). And kernels with special dimensions (“vector kernels”) are mostly dispatched to libraries like cudnn, cublas, so these select block sizes somehow…</p>

googlebot: <p>Perhaps I’ve seen 512 somewhere else, and it is actually 256 threads/block. Basically, it is hardcoded here: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Loops.cuh#L6" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Loops.cuh#L6</a></p>
<p>And example of “vector kernel” block size selection logic (not pretty): <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Reduce.cuh" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Reduce.cuh</a></p>

softplus: <p>Thank you very much. It is extremely helpful.</p>
Sample code runs on 3.7 but not 3.6

MikeToolin: <p>I am new to Pytorch.  I have been running the sample code for reinforcment learning (<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a>) on a different data set that has 5 output classes.  It runs fine on my Python 3.7 environment, but gives a runtime error when running on Python 3.6</p>
<p>THCudaCheck FAIL file=C:/cb/pytorch_1000000000000/work/aten/src\THC/generic/THCTensorMath.cu line=29 error=710 : device-side assert triggered<br>
C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: block: [0,0,0], thread: [0,0,0] Assertion <code>t &gt;= 0 &amp;&amp; t &lt; n_classes</code> failed.</p>
<p>I’ve been comparing my environment for days and can’t see any tangible differences.  Any pointers in the right directions would be appreciated.</p>

ptrblck: <p>This error is raised, if you pass invalid target indices to <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code>, which both expect the target tensor to contain values in the range <code>[0, nb_classes-1]</code>.<br>
I don’t think the Python version changes this error, so you would have to check the target values instead.</p>

MikeToolin: <p>Thanks for the quick response. I’ll look at it, but it is the exact same code from the PyTorch website.  All I am changing is the environment in PyCharm and re-running the same code. I’ll let you know my resolution.</p>

MikeToolin: <p>I downloaded the Notebook with the sample data and it ran fine in 3.6.  When I pointed the Notebook at my data, it failed again. I found where the number of output classes was hardcoded, changed it to the correct number of classes and it ran fine.  So problem resolved, but I still don’t know why the wrong number of classes ran on 3.7.  Not going to worry about it, thank you for your help.</p>