akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What does "model.training" in the transfer learning tutorial mean?

username: zy_D
cooked: <p>Hi everyone,</p>
<p>I’m a beginner of Pytorch and I’m reading the transfer learning for computer vision tutorial on the pytorch website. (Link below) I failed to understand the code in the visualization part “was_training = model.training”. Is it defined in the ResNet init function? Where can I check it? Thank you in advance!</p>
<p>Tutorial link: <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31be892b601c4bb8ad4af76355d376b0fa7dc347.png" data-download-href="https://discuss.pytorch.org/uploads/default/31be892b601c4bb8ad4af76355d376b0fa7dc347" title=")C@IC(CFT@3U0)EPFEERPWE"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31be892b601c4bb8ad4af76355d376b0fa7dc347.png" alt=")C@IC(CFT@3U0)EPFEERPWE" data-base62-sha1="763CgmpDo8umb09mtkVV28D9Y4T" width="690" height="472" data-dominant-color="EEEFF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">)C@IC(CFT@3U0)EPFEERPWE</span><span class="informations">1021×699 41.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: zy_D
cooked: <p>Can someone help me?</p>

username: ptrblck
cooked: <p>All <code>nn.Module</code>s have an internal <code>training</code> attribute, which is changed by calling <code>model.train()</code> and <code>model.eval()</code> to switch the behavior of the model, if necessary.<br>
E.g. it is used in dropout layers to disable them during evaluation.</p>
<p>The <code>was_training</code> variable stores the current training state of the model, calls <code>model.eval()</code>, and resets the state at the end using <code>model.train(training=was_training)</code>.</p>

username: zy_D
cooked: <p>Thank you so much for the excellent explanation!</p>
How to match parameters and optimizer state?

username: 11118
cooked: <p>Hi there, I want to initialize a 24-layers transformer from a pre-trained 12-layers one (model + optimizer).</p>
<p>For example, I use the parameters of the layer[0] to initialize layer[13], and so on. As of now, everything is going well, but when I tried to initialize the optimizer in the same way, I found that the key of the optimizer state is not the name of the parameter, but some numbers like ‘140453283719280’, How do I match these numbers with the parameter names (And further, how can I extend them)?</p>
How to implment Group Lasso in PyTorch

username: DeepakVK
cooked: <p>I am trying to implement Group Lasso on weight matrices of a neural network in PyTorch.</p>
<p>I have written the code to implement Group Lasso but am unsure if this is correct, confirmation or correction of my code will be very helpful.</p>
<pre><code class="lang-auto">def gl_norm(model, gl_lambda, num_blk):
    gl_reg = torch.tensor(0., dtype=torch.float32).cuda()
    for key in model:
        for param in model[key].parameters():
            dim = param.size()
            if dim.__len__() &gt; 1 and not model[key].skip_regularization:
                div1 = list(torch.chunk(param,int(num_blk),1))
                all_blks = []
                for div2 in div1:
                    temp = list(torch.chunk(div2,int(num_blk),0))
                    for blk in temp:
                        all_blks.append(blk)
                for l2_param in all_blks:
                    gl_reg += torch.norm(l2_param, 2)
    return gl_reg * float(gl_lambda)
</code></pre>
<p>I expect the torch.chunk function to break up the weight matrix into small blocks which then go through L2 norm for the block and L1 norm between all the blocks.</p>

username: Gaoyuan_Yang
cooked: <p>Hi, I am also interested in this question, do you need to implement proximal gradient descent algorithm here since it is a non-differentiable regularizer.</p>
Does setting ```torch.backends.cudnn``` also helps improve performance on CPU?

username: mohit117
cooked: <p>Hello,<br>
I know setting <code>torch.backends.cudnn</code> for fixed input size improves performance for GPU inference. But if i want to speed up inference just on CPU does this help (for fixed input size)?</p>

username: ptrblck
cooked: <p>No, cudnn is a library on top of CUDA and works only on GPUs.<br>
For CPU performance improvement, you could use e.g. MKL.</p>
Custom Dataset error: Lagged timeseries data

username: NIkolaStaykov
cooked: <p>I am trying to implement a custom dataset for analyzing time series, but get the following error</p>
<pre><code class="lang-auto">class Data(Dataset):
    def __init__(self, data, labels, window):
        self.data = data
        self.labels = labels
        self.window = window
    def __len__(self):
        return len(self.labels)-self.window
    def __getitem__(self, index):
        item = self.data[index:index + self.window]
        label = self.labels[index+self.window] 
        return item, label
</code></pre>
<pre><code class="lang-auto">        8     def __getitem__(self, index):
----&gt;   9         item = self.data[index:index + self.window]
       10         label = self.labels[index+self.window]
       11         return item, label

TypeError: unsupported operand type(s) for +: 'slice' and 'int'
</code></pre>
Is it possible to use pre-trained image multi-label classification model

username: Tuna
cooked: <p>I knew that nowadays, people usually use a pre-trained model to fine-tune and train their own models. There are few famous models that have been trained with the ImageNet dataset is Resnet, Inception, or VGG.</p>
<p>You may all know that the ImageNet dataset comes along with a Multi-class classification problem so that we usually use <code>softmax</code> with <code>CrossEntropy</code> to train the model. I usually use pre-trained Resnet50 or VGG19 in <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">TorchVision models</a></p>
<p>I just wonder if we have a large dataset of Multi-Label instead (suppose 10M images) of Multi-class classification with more than 10k labels (10x larger than Imagenet labels) so that I can build a CNN model using <code>sigmoid</code> with <code>BinaryCrossEntroy</code> to train our model with that dataset.</p>
<p>Is that a good idea if we can use that new checkpoint model for transfer learning purposes? If yes, when we should use that scenario when we shouldn’t? or Why not?</p>

username: ptrblck
cooked: <p>Using a pretrained model shouldn’t hurt your training, as you could see it as a “good initialization” of the parameters in comparison to a random initialization.</p>
<p>From my experience, the model loss might blow up and you would need to retrain the model, but this shouldn’t be worse than training from scratch.</p>
<p>An advantage and also a shortcoming from using a pretrained model is the fixed architecture.<br>
While it should be no problem to change e.g. the final classifier, I’m not sure which approach would work best e.g. for transforming a 2D model into a 3D one.</p>
After building from source, how do I use it?

username: prismspecs
cooked: <p>I successfully built it from source, so that I can take advantage of my older GPU the gtx 760, and now I’d like to follow along with the examples but I’m unsure of how to use it once I’ve built it. I was specifically hoping to use it inside of conda, if that’s helpful.</p>

username: ptrblck
cooked: <p>If you’ve built PyTorch from source and the installation was successful, you should be able to execute <code>python</code> and <code>import torch</code> in it (or use your favorite IDE to write PyTorch code). <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Let me know, if you get stuck somewhere.</p>
Should I use softmax activation before crossentropy loss backward?ack

username: acmilannesta
cooked: <p>I know <code>nn.CrossEntropyLoss()</code> incoporates softmax. So my model output did not add softmax layer. But when I do the <code>loss.backward()</code>, should I change the outputs using softmax like <code>outputs = torch.softmax(outputs, dim=1).cpu().detach().numpy()</code>?</p>

username: ptrblck
cooked: <p>As you said, you should not apply <code>softmax</code> on the outputs before passing it to <code>nn.CrossEntropyLoss</code>.</p>
<aside class="quote no-group" data-username="acmilannesta" data-post="1" data-topic="82062">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/acmilannesta/48/24428_2.png" class="avatar"> acmilannesta:</div>
<blockquote>
<p>But when I do the <code>loss.backward()</code> , should I change the outputs using softmax like <code>outputs = torch.softmax(outputs, dim=1).cpu().detach().numpy()</code> ?</p>
</blockquote>
</aside>
<p>If you’ve calculated the <code>loss</code> using the raw logits and would now want to get the probabilities for debugging purposes etc., then you could of course apply the <code>softmax</code> afterwards.</p>
Dataparallel causing shape conflicts

username: ilyes
cooked: <p>Hello guys,</p>
<p>Let’s say I have the following forward function in a given defined module</p>
<pre><code class="lang-auto">class Classifier(nn.Module):

    def __init__(self,args*):
        super(Classifier, self).__init__()
        # defining some layers

    def forward(self, feature_batch, tensor_A, *args):
        # some operations between feature_batch and tensor_A
        return results
</code></pre>
<p>This module is a part of my global model. and tensor_A is a learnable parameter passed from another module.</p>
<pre><code class="lang-auto"># feature_batch.shape : (32, 128)
# tensor_A.shape : (500, 128) # is supposed to be like this
# feature_batch_expanded.shape : (32, 500, 128)
# tensor_A_expended.shape : (32, 500, 128)# is supposed to be like this
</code></pre>
<p>I have 2 GPUs, I am doing dataparallelization before startning the training:</p>
<pre><code class="lang-auto">global_model = nn.DataParallel(global_model).to(self.device)
</code></pre>
<p>The problem I am having is shapes mismatch during the operation because tensor_A is not what I am waiting for:</p>
<pre><code class="lang-auto"># tensor_A.shape : (250, 128)
</code></pre>
<p>when I print the shapes inside the forward, this is what i get:</p>
<pre><code class="lang-auto"># feature_batch.shape : (32, 128)
# tensor_A.shape : (250, 128)
# feature_batch.shape : (32, 128)
# tensor_A.shape : (250, 128)
</code></pre>
<p>the forward pass is called two times before starting the operations.<br>
my questions are:<br>
1 - why feature_batch is not being split on two GPUs? I mean when I print the shape I dont have two times (16, 128). means the feature_batch is passed only to one GPU<br>
2 - How can I stop pytorch from thinking that 500 is the batch size, and so stopping it from splitting the tensor_A. is there an effective solution to this without having gradient problems.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>This code snippet does the expected splitting:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        pass

    def forward(self, feature_batch, tensorA):
        print(feature_batch.shape, feature_batch.device)
        print(tensorA.shape, tensorA.device)
        return feature_batch

model = MyModel().cuda()
model = nn.DataParallel(model, device_ids=[0, 1])
feature_batch = torch.randn(32, 128).cuda()
tensorA = torch.randn(500, 128).cuda()
out = model(feature_batch, tensorA)
&gt; torch.Size([16, 128]) cuda:0
torch.Size([250, 128]) cuda:0
torch.Size([16, 128]) cuda:1
torch.Size([250, 128]) cuda:1
</code></pre>
<p>Are you sure you haven’t increased the batch size?</p>
<p>Anyway, if you want to pass the same <code>tensorA</code> to all replica, you could use:</p>
<pre><code class="lang-python">tensorA = tensorA.unsqueeze(0).expand(2, -1, -1)
out = model(feature_batch, tensorA)
</code></pre>
Loading a dynamically quantized Transformers model

username: Pramodith
cooked: <p>Hi,</p>
<p>I’ve trained a custom transformer model and followed <a href="https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html#serialize-the-quantized-model" rel="nofollow noopener">this</a>  to save a quantized model.</p>
<p>However when I try to load the model using</p>
<pre><code class="lang-auto">model.load_state_dict(torch.load('path'))
</code></pre>
<p>I receive the following error:</p>
<p>Missing key(s) in state_dict: “xxxxx.weight”,<br>
Unexpected key(s) in state_dict: “xxxx.scale, xxxx.zero_point, …”</p>
<p>It looks like the names of the original parameters of the model have been changed. Can anyone help with how I can resolve this error?</p>

username: supriyar
cooked: <p>While loading the model is the <code>model</code> now a quantized model? If you convert the model to quantized model and then load the quantized state_dict it should work.</p>

username: Pramodith
cooked: <p>I’m not sure that I understand, assuming class A inherits from nn.Module and corresponds to the architecture of my dnn.</p>
<p><code>model = A()</code><br>
is essentially all I do. Do I need to do anything to quantize it?</p>

username: Pramodith
cooked: <p>Ok I think I get it now, I have to do something like this after</p>
<p>model = A</p>
<pre><code class="lang-auto">quantized_model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)
</code></pre>
State_dict optimizer params

username: MGud
cooked: <p>Hi there,</p>
<p>when saving an optimizier during training with optimizer.state_dict()<br>
and later loading it via torch.load() I get a dictionary containing “state” and “param_groups”<br>
as described in the documentation <a href="https://pytorch.org/docs/stable/optim.html?highlight=state_dict#torch.optim.Optimizer.load_state_dict" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html?highlight=state_dict#torch.optim.Optimizer.load_state_dict</a>.</p>
<p>In “params_group” I have an entry for “params”: what exactly are these values?<br>
Are these the last gradient values that have been used to update my NN?</p>
<p>Thanks in advance!</p>

username: Alpha
cooked: <p>see answer <a href="https://discuss.pytorch.org/t/optimizer-params-not-in-the-optimizer-state-dict/42957/2">here</a></p>
<p>These numbers represent the ids of the parameters you’ve passed to the optimizer.</p>
<p>Thank <a class="mention" href="/u/ptrblck">@ptrblck</a>  very much!</p>
Getting batchnorm layer statistics

username: Frida
cooked: <p>Hi!<br>
How can I extract the mean and variance of a given batch in batchnorm layer, is it the running_mean\running_var just without model.evel()<br>
Thanks a lot!</p>

username: Kushaj
cooked: <p>Running mean would contain the exponential average over previous batches. To get the mean and variance of the current batch you have to compute them using separate functions.</p>

username: Frida
cooked: <p>What do you mean by separate functions</p>

username: Kushaj
cooked: <p>Use separate functions to compute mean like torch.mean().</p>

username: Frida
cooked: <p>but how can I extract the inputs to the batchnorm</p>

username: Eta_C
cooked: <p>What do you mean?</p>
<p>If you want to get the current <code>running_mean</code> and <code>running_var</code> in a pretrained model, use <code>model.{your_batch_norm_module}.running_mean</code> and <code>model.{your_batch_norm_module}.running_var</code>.</p>
<p>If you want to get the <code>running_mean</code> and <code>running_var</code> in a pretrained model after forward <code>x</code>, use <code>torch.nn.functional.batch_norm</code> or just forward that layer.</p>
<p>If you donot have a pretrained model, and want to get the <code>running_mean</code> and <code>running_var</code>, init <code>running_mean</code> to <code>0</code> and <code>running_var</code> to <code>1</code> then use <code>torch.nn.functional.batch_norm</code> or <code>torch.batch_norm</code>.</p>

username: Frida
cooked: <p>I want the mean and the variance of a batch as the normbatch layer calculate</p>

username: Kushaj
cooked: <p>Use torch.mean() and torch.std(). There is also torch.normalize() function that you can look into.</p>

username: Frida
cooked: <p>thanks a lot but how can I get the tensor of the batchnorm, X</p>

username: meryem_janati
cooked: <p>Hi, have you found a way to do it?</p>
Generate unique triplets from source and labels tensor list

username: John_J_Watson
cooked: <p>I was wondering if there was an external lib or a built-in to generate unique triplets from two lists. So, I have two lists like so (the elements of the source list are PyTorch tensors - I have abstracted it out here as  just numbers):</p>
<pre><code class="lang-auto">source_list=[1,2,3,4,5,6,7,8..] # size N
labels_list=[1,1,0,2,1,3,6..] # size N
</code></pre>
<p>and I would like triplets of form:</p>
<pre><code class="lang-auto">&lt;anchor&gt; &lt;postive&gt; &lt;negative&gt;
</code></pre>
<p>where,  <code>&lt;anchor&gt;</code>  can be any element in the  <code>source_list</code> ,  <code>postive</code>  means that this element should come from the same label as the anchor(so, 1,2 and 1,5 could be seen as postive pairs in the triplet) and  <code>negative'</code> s mean that they should be from a different label (so, 1,3 and 1,4 would be considered a negative pair of the triplet). So, some examples of the correct triplet would be:</p>
<pre><code class="lang-auto">1, 2, 3
1, 2, 4
1, 2, 6
</code></pre>
<p>From the outset it seems like a  <code>N choose k</code>  problem, but I am not sure how to approach this with minimal computational cost other than doing loops with each element in python.</p>
Converting model from Keras to Pytorch (Problem with BatchNorm)

username: David_Hresko
cooked: <p>Hi, I am trying to convert pix2pix GAN network from keras to pytorch. I would like to convert Keras BatchNorm layer to Pytorch BatchNorm2d, because i think that the reason why my discriminator loss is falling faster than in keras.</p>
<p>This is original Keras implementation: <strong>BatchNormalization(momentum=0.8)</strong> and I also would like to set up init weight and bias as it is in Keras.</p>
<p>Thanks for help.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The batchnorm in pytorch takes a  momentum parameter as well so you can do the same thing (doc <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm2d">here</a>).<br>
For the initialization, you can initialize the <code>.weight</code> and <code>.bias</code> of the batchnorm as you want.</p>

username: David_Hresko
cooked: <p>this one is clear for me, but i am not sure about the others, which are set as default in keras version</p>
Prevent exceptions from cholesky

username: Anders_G
cooked: <p>I want to do batched cholesky decompostion but some matrixes are not positive definite, due to round-off. So pytorch raises an exception, failing the entire batch. Is there a way to just get the information that a particular element in the batch failed, like for lu decomposition? Repeating the analysis until all failed elements are removed would cancel the utility of using GPU since it becomes too slow. As far as I can see the underlying MAGMA functions support this but it is not exposed to user:</p>
<p><a href="https://icl.cs.utk.edu/projectsfiles/magma/doxygen/group__magma__potrf__batched.html#ga3893d96e1bdf0b3bfef285ac9a48416b" class="onebox" target="_blank" rel="nofollow noopener">https://icl.cs.utk.edu/projectsfiles/magma/doxygen/group__magma__potrf__batched.html#ga3893d96e1bdf0b3bfef285ac9a48416b</a></p>

username: glaringlee
cooked: <p><a class="mention" href="/u/anders_g">@Anders_G</a><br>
If you do your batch using a loop in c++, you can do:<br>
for (…) {<br>
try {<br>
cholesky decomposition for 1 matrix<br>
}<br>
catch(…) {<br>
// catch exception but do nothing, so your code can pass through with exception<br>
}<br>
}<br>
If you do care about what exception is thrown, you can catch the actual exception.</p>

username: Anders_G
cooked: <p>I suspect this will be rather slow since it cannot make use of doing things in parallel. I don’t know how to setup torch in c++ so can someone comment on if this could be a good solution before I start to walk down that road.</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/anders_g">@Anders_G</a><br>
Let me clarify.<br>
This won’t affect parallel, you can split your batch in multiple threads or different cores. try/catch only wrapped your cholesky code, so the exception won’t stop your process. It will be a little bit slower compare to not using try/catch.</p>
<p>The only thing is that you can only process one matrix at a time, though pytorch cholesky function support batches, but it will stop when process a not-valid matrix. What you can do is to have a loop, and process one matrix each time with try/catch so it won’t break, if you want to print out error, you can catch(std::exception&amp; e) and print the error msg. You can scale the loop in multiple threads or cores for sure.</p>
Error in Tensorboard's(PyTorch) add_graph

username: paul_shuvo
cooked: <p>I’m following this Pytorch’s <a href="https://pytorch.org/docs/stable/tensorboard.html" rel="noopener nofollow ugc">Tensorboard</a> documentation.</p>
<p>I have the following code:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torchvision.models.resnet50(False)
writer.add_graph(model)
</code></pre>
<p>It throws the following error:</p>
<blockquote>
<p>_ = model(*args)  # don’t catch, just print the error message<br>
TypeError: ResNet object argument after * must be an iterable, not NoneType</p>
</blockquote>
<p>I don’t know what I’m doing wrong here!</p>

username: xvdp
cooked: <p>you need to pass a tensor , the graph gets built by passing data thru it</p>
<pre><code class="lang-python">import torchvision
import torch
from torch.utils.tensorboard import SummaryWriter
model = torchvision.models.resnet50(False)
writer = SummaryWriter(log_dir='graph')
writer.add_graph(model, torch.randn([1,3,224,224]))
writer.close()
</code></pre>
How to change the learning rate in the middle of training?

username: Janashree_Ananthan
cooked: <p>I am using reduce LR on Plateau scheduler to reduce the learning rate, i need to reset the learning rate, once it reaches 10^-6.<br>
Does adding below line helps???<br>
<code>optimizer.param_groups[0]['lr'] = 0.0001</code></p>

username: futscdav
cooked: <p>Yes, that would be the gist of manually setting LR.</p>
<pre><code class="lang-auto">for group in optimizer.param_groups:
    group['lr'] = lr
</code></pre>
For-loop indexing and value assignment

username: konstantinosKokos
cooked: <p>Hi,</p>
<p>I am trying to iterate and value-assign over a tensor from another tensor within the forward() call of my network. This component alone causes a slowdown of 5x for my forward/backward pass but I can’t figure a better way of implementing it.</p>
<p>I tried utilizing a multiprocessing Pool which seemed easy (since my piece of code contains no intra-dependencies), but torch is complaining about autograd not being able to track gradients between shared non-leaf tensors.</p>
<p>Am I missing something? Is there any other way to get some performance back?</p>
<p>code:</p>
<pre><code class="lang-auto">def recover_batch(original: FloatTensor, processed: FloatTensor, ids: Sequence[Tuple[int, range]]) -&gt; FloatTensor:
   """ original: B x M x D  -- non-leaf, req.grad
         processed: T x N x D  -- non-leaf, req.grad
         id: a pair (b, r) where b in B, r a subrange of [0, M)
         ids: sequence of T ids identifying each item in processed -- no pair of ids is overlapping!
   """
   for i, (b, r) in enumerate(ids):
       original[b, r] = processed[i, 0:len(r)]
   return original
</code></pre>

username: Awais_Jafar
cooked: <p>I would really like to know the answer to this. Facing similar issue</p>
Locking a parameter

username: seliad
cooked: <p>I have a big giant tensor shared between two processes/streams.<br>
I want to create a simple lock,<br>
it should combine both with (<code>forward()</code>, <code>backward()</code>, <code>step()</code>).<br>
For <code>forward</code> and <code>step</code> I have a simple solution: overriding the <code>toch.nn.Parameter</code> class.<br>
The problem is in backward/autograd.<br>
We have a post-backward hook but no pre-backward hook.</p>
<p>Is there any possibility doing what I want in Pytorch?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you only read it, you will be fine.<br>
If you have to write, you should make sure no reads happen at the same time but that should be much simpler to enforce.</p>
<blockquote>
<p>For <code>forward</code> and <code>step</code> I have a simple solution: overriding the <code>toch.nn.Parameter</code> class.</p>
</blockquote>
<p>I am curious how you do that?</p>

username: seliad
cooked: <p>I mean readers-writers lock. So for for disallowing reads I have to lock something when read start.</p>
<p>Its optimal to do it just when we are about to read the parameter during the backward pass<br>
(so I mentioned this “pre-backward-hook”)</p>
<p>locking at the begging of <code>autograd.backward</code> is the easy solution, but its wasteful right.<br>
can split the backward to 2 calls if the model is sequential, is that what you meant by “easy to enforce”?</p>
<p>I actually thought on overriding  <code>toch.nn.Parameter</code> class with composition, and automate it somehow (e.g with with setatter/getattr, or maybe there is more elegant pythonic solution in stackoverflow <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ) .</p>

username: albanD
cooked: <p>Most of the autograd logic (in particular, .grad field update) happens in cpp. And so out of the scope of <code>nn.Parameter()</code> <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:">  These get/setattr won’t be called when the autograd updates the field in cpp <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>If the only part that writes is the end of the backward, a solution would be to use <code>autograd.grad</code> and do the copy into the <code>.grad </code> field manually (with proper locking).</p>
<p>But I don’t think there is any way to do general locking on a Tensor. You will have to do it for each use one by one.</p>
Run n models in parallel on single GPU

username: twikwik
cooked: <p>I want to train n models (per n, I have f times t data points). I can load all data onto a single GPU. I assign the dataloader batches and each batch gets a number of minibatches. Each minibatch holds the data to train one model (one n). The data per n is rather small, but the number of models is large. The ‘problem’ that I am facing is that the batches are executed sequentially rather than in parallel on the single GPU. Is there a way to parallelize the batches on the single GPU to ensure scaling to a large number of models quicker? The bare, individual training time per model is improved by a factor of X using GPUs over CPU (not given any parallelization).</p>
<p>Thanks in advance.</p>

username: osalpekar
cooked: <p>I think you should be able to spawn multiple processes on a single GPU (using torch.multiprocessing - <a href="https://pytorch.org/docs/stable/multiprocessing.html" rel="nofollow noopener">https://pytorch.org/docs/stable/multiprocessing.html</a>), and train each model in a separate process. You may need to tune the number of processes you spawn, since performance may be degraded with too many processes due to resource contention.</p>
Use of .backward()

username: km_proton
cooked: <p>what is the use of backward i cannot understand<br>
x=torch.tensor(3.,requires_grad=True)<br>
w=torch.tensor(2.,requires_grad=True)<br>
b=torch.tensor(4.,requires_grad=True)<br>
y=w*x+b<br>
y.backward()<br>
x.grad<br>
w.grad<br>
b.grad</p>
<p>could you please explain use of .backward class also use of .grad</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>y.backward()</code> will perform backprop to compute the gradients for all the leaf Tensors used to compute <code>y</code>.<br>
The <code>.grad</code> attribute of leaf Tensors is where these computed gradients are stored.</p>
<p>The 60min Blitz on pytorch <a href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html">here</a> and the part on autograd describes this in details.</p>

username: km_proton
cooked: <p>i came after watching that lol <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"> . it is not given in laymans terms. i can understand untill above gradients but after that</p>
<blockquote>
<p>Let’s backprop now. Because  <code>out</code>  contains a single scalar,  <code>out.backward()</code>  is equivalent to  <code>out.backward(torch.tensor(1.))</code> .</p>
</blockquote>
<p>what does this mean i cannot get it .</p>

username: albanD
cooked: <p>This means it will use the backpropagation algorithm (very close to reverse mode automatic differentiation) to compute the gradient of the output with respect to the input Tensors that require gradients.</p>

username: km_proton
cooked: <p>no still i cannot get it could you please explain me in simple terms all i know is differentiation jacobians ect . could you please explain me somewhat like what happens when you call .backpropagation() in simple terms (are you telling that we are taking dy/dx and y is output here and x is input) .<br>
ok in the example above y.backward() y is output then what is input</p>

username: albanD
cooked: <p>As mentionned in <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.backward">the doc</a> <code>.backward()</code> computes the gradient for all the leafs used to compute the output. Doc for leaf Tensor is <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.is_leaf">here</a>.<br>
If you use autograd.grad doc <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.grad">here</a> you can specify explicitly which inputs you want the gradient for.<br>
Depending on what you want to do, you should use the one that fits best.</p>
<p>Backpropagation algorithm is very well explained on the wikipedia article: <a href="https://en.wikipedia.org/wiki/Backpropagation">https://en.wikipedia.org/wiki/Backpropagation</a><br>
The short answer in this case of scalar output is that it computes the gradient dy/dx (either stored in y.grad if you use .backward() or returned to your if you use autograd.grad).</p>
Plotting train and test loss

username: cevvalu
cooked: <p>Hi, trying to plot training and test loss in same graph, any advice?</p>

username: ptrblck
cooked: <p>You could use <code>matplotlib.pylot</code> to plot multiple graphs in the same plot.<br>
Alternatively, PyTorch has also TensorBoard as well as Visdom support.</p>

username: cevvalu
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> will try both. Thanks for help</p>
Does autodiff engine track tensor.data?

username: Kevin96
cooked: <p>I noticed that variable and tensor has been merged together after 0.4. But I still encounter some code accessing data field directly. So I just want to clarify a few things.</p>
<p>It seems like tensor.data is still a tensor object. Does autodiff engine track computation when operating on tensor.data directly (e.g. c = a.data + b.data)? If not, is tensor.data just another view of tensor but with requires_grad=False?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes you should stop using it altogether <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
You can find in this other thread a detailed explanation of the effects of .data: <a href="https://discuss.pytorch.org/t/using-tensor-data/79640/4" class="inline-onebox">Using tensor.data</a></p>
<p>Does that answer your question?</p>
Can I backward without forward after an optimizer.step()

username: 111319
cooked: <p>I have a simple resnet. When training, I call forward to get output and calculate the loss. Then I backward the loss to get the gradients of weight and call optimizer.step() to train the net. Here, everything is OK. But then I want to call output.backward() again to get the gradients of inter-results of resnet, such as the output feature map of the last CNN layer (This is used in Grad_CAM). Here, I faced the problem: " RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation". I guess this is because the weights of each layer have been modified by the optimizer and this is an inplace operation. So anther backward will fail.</p>
<p>I know that a simple solution is to do the forward again. But this will lead to a slow training process. Is there a better solution to this problem, such as is there a method to store the gradients of two backward separately?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Another solution is to compute these intermediate gradients before doing the step of the optimizer no?</p>
UserWarning: indexing with dtype torch.uint8 is now deprecated, please use dtype.torch.bool instead,

username: Geoffrey_Payne
cooked: <p>I am get this error;</p>
<p>UserWarning: indexing with dtype torch.uint8 is now deprecated, please use dtype.torch.bool instead.<br>
It is located in IndexingUtils.h:20<br>
It is not my code. How do I fix this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is just a warning, not an error.<br>
This happens because we used to have only <code>uint8</code> type to represent boolean and were using them as masks.<br>
Now that we introduced <code>bool</code> dtype for that purpose, we are deprecating the use of <code>uint8</code> as a bool.</p>
<p>You can remove it by changing your <code>uint8</code> Tensor to a boolean one before using it as a mask when indexing.</p>
DataLoader: one input class in each batch?

username: nickvu
cooked: <p>Is there a way to make the DataLoader produce batches containing only one class each?</p>
<p>For example, the training dataset could contain 1000 images A, 1500 images B, 1300 images C…</p>
<p>I would need the DataLoader to then yield a batch of only A samples, then another batch of B samples, etc.</p>
<p>Thank you so much for any help or tips!</p>

username: ptrblck
cooked: <p>I think the easiest way would be to implement a custom sampler and use it in the <code>DataLoader</code>.<br>
This sampler could take the batch size and create data indices for the same class for each call.<br>
Your <code>Dataset.__getitem__</code> method would therefore get a batch of indices and you would have to load the complete batch there.</p>
<p>Alternatively, you could also try to make sure that each subsequent call to the sampler only returns the “expected” value (single index), so that the <code>collate_fn</code> will create a batch of the same class.</p>

username: nickvu
cooked: <p>That makes a ton of sense - after researching a bit, I think I will go with the approach where I create a separate Dataset for each input class, wrapped in a ConcatDataset. Then the DataLoader would call on a custom sampler which would alternate between pulling from the different sub-Datasets…<br>
Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
When I was using VSCode, pylint always show some errors

username: NimenDavid
cooked: <p>When I was using VSCode with pylint, it told me</p>
<pre data-code-wrap="Python"><code class="lang-Python">device = torch.device('cpu')
</code></pre>
<p>E1101:Module ‘torch’ has no ‘device’ member</p>
<p>and so is  torch.randn</p>
<p>but it run successfully and I guess there is no error…</p>
<p>how can I fix this problem?</p>
<p>THANKS a lot!</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/55d87fd8e72a516e37c458eb45dd23bf6ff2a830.png" alt="20180822142637" data-base62-sha1="cfqqPLj6pmtil8BYIgiTPTMIzQc" width="513" height="174"></p>

username: NimenDavid
cooked: <p>I don’t why…it shows some error but work well…</p>

username: orena1
cooked: <p>Try this:<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/56844378/pylint-no-member-issue-but-code-still-works-vscode/59413950#59413950" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/8445144/vaitul-bhayani" target="_blank" rel="nofollow noopener">
    <img alt="Vaitul Bhayani" src="https://www.gravatar.com/avatar/159c84ae5abd49faf8286071c4b7d0b8?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/56844378/pylint-no-member-issue-but-code-still-works-vscode/59413950#59413950" target="_blank" rel="nofollow noopener">pylint no member issue but code still works vscode</a>
</h4>

<div class="tags">
  <strong>python, pylint</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/8445144/vaitul-bhayani" target="_blank" rel="nofollow noopener">
    Vaitul Bhayani
  </a>
  on <a href="https://stackoverflow.com/questions/56844378/pylint-no-member-issue-but-code-still-works-vscode/59413950#59413950" target="_blank" rel="nofollow noopener">04:53PM - 19 Dec 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to get sentence probability from CTCLoss?

username: Jan_Vainer
cooked: <p>I need to compare probabilities of two sentences in an ASR. I have the logprobability matrix from the accoustic model and I want to use the CTCLoss to calcuate the probabilities of both sentences.<br>
Does the CTCLoss return the negative log probability of the sentence? Or does it return pure probability of the given sentence? It would make sence to me that it would return negative log probabilities. However, I am receivin relatively small values (less than 1) for long sentences. I would expect that long senences would have very low log-likelihood and therefore very large negative log-likelihood (like thousands or so).</p>
Manipulate 3D CT DICOM image

username: eranb
cooked: <p>Hi everyone,<br>
I have a simple question,<br>
If I have 3D CT image DICOM, there is a way to slice it to 2d images (in PyTorch, I build a regular function in python)?</p>
<p>And I want to take this 3D image and to do manipulation and after that to slice it again (for data augmentation). How can I do that? what is the simple way to manipulate a 3D CT image (DICOM)?</p>

username: ptrblck
cooked: <p>It depends a bit how you are loading the data.<br>
If you can create a 3D tensor of the DICOM volume, you should be able to slice the tensor directly.<br>
However, if seems you would like to apply some data augmentation on the volume directly.<br>
If that’s the case, have a look at <a href="https://github.com/Project-MONAI/MONAI">MONAI</a> which provides some processing and augmentation techniques for medical data.</p>

username: eranb
cooked: <p>Thanks a lot for your help!</p>
Doing meaning pooling within a given range to get sentence representation

username: hongyuanluke
cooked: <p>Hi, I am wondering how I could perform mean pooling within a given range of word embeddings.</p>
<p>For example, I have a list of word embeddings with length 100. I would like to perform a mean averaging on the 0-35 words and 36-100 words individually. How could I perform this given this list and an index list [0, 35, 100]?</p>
<p>Will be appreciating for any help…</p>
<p>Thanks!</p>

username: hongyuanluke
cooked: <p>The current way I have done is to:</p>
<pre><code class="lang-auto">    def create_sentence_embedding(self, encoder_states, sentence_indices, sentence_index_num):
        sentence_indices=sentence_indices[:sentence_index_num]
        xxs=[torch.div(torch.sum(encoder_states[sentence_index:sentence_indices[i+1]], dim=0), len(encoder_states[sentence_index:sentence_indices[i+1]]))  for i, sentence_index in enumerate(sentence_indices[:-1])]
        xxs=torch.stack(xxs)
        return xxs

    def forward():
        xxs = [self.create_sentence_embedding(encoder_state, yys[i], sentence_index_num[i][0]) for i, encoder_state in enumerate(encoder_states[0])]
        xxs = rnn_utils.pad_sequence(xxs, batch_first=True)
       
       # pass xxs into a sentence transformer
        xxs, mask, seq_len = self.sentence_encoder(xxs)

       # pass xxs into a feed-forward position-wise network
        xxs = self.sentence_ordering_decoder(xxs)
</code></pre>
<p>However, after doing this, my network weight does not update… So I think there is some problem with the way I used above.</p>
Pytorch Geometric - Build a dataset from multiple graphs with common nodes

username: agma
cooked: <p>I have 2 different graphs, in the form of edge lists, that I would like to integrate into one large graph before training my model on node classification.<br>
These 2 graphs share nodes which correspond to common entities.<br>
How is it possible to integrate the graphs, while preserving information about common nodes?</p>
<p>For instance, let’s consider the following edge lists:</p>
<p>[0 - 1,<br>
5 - 4,<br>
2 -16,<br>
13 - 8]</p>
<p>AND</p>
<p>[5 -12,<br>
15 - 3,<br>
13 - 2]</p>
<p>The first graph edge list has 8 nodes and the second has 6 nodes.<br>
Providing the edge lists as such to the Data object leads to an “out of bound” error during training because the model tries to access nodes according to their index in the edge list. For instance, edge “15-3” makes the model choose the 15th and the 3rd node, however the data object has only 6 nodes, so it returns an “out of bound” error.</p>
<p>Any help or recommendation would be much appreciated! Thank you.</p>
[Caffe2] Setting XNNPACK engine for convolution operations

username: JustasZ
cooked: <p>Hello,</p>
<p>I am using Caffe2 for mobile and I am trying to test the XNNPACK engine for convolutions. Until now I have been using NNPACK and it has resulted in some performance boost compared to the default engine, though, when I try to change the engine to XNNPACK, it seems that it switches to the default engine, because the performance becomes worse. Is there a way to use XNNPACK in caffe2 (built from pytorch v1.5) and if yes, how should it be done?</p>
<p>Code:</p>
<pre><code class="lang-auto">caffe2::NetDef modelDef;
CAFFE_ENFORCE(caffe2::ReadProtoFromFile("path to model def file", &amp;modelDef));
for (int i = 0; i &lt; modelDef.op_size(); ++i)
{
	caffe2::OperatorDef *opDef = modelDef.mutable_op(i);
	if (opDef-&gt;type() == "Conv")
	{
		opDef-&gt;set_engine("NNPACK"); // Works
		opDef-&gt;set_engine("XNNPACK"); // Probably not working
	}
}
</code></pre>
<p>NOTE: The code was built with XNNPACK support.</p>
Release of Pytorch 1.6 with native CUDA amp

username: tyb_10
cooked: <p>Hi all,<br>
Is there an expected date for the release of Pytorch1.6, I am mostly interested in the forthcoming native CUDA amp support that is currently available in nightly binaries.</p>
<p>Thanks !!</p>

username: ptrblck
cooked: <p>No official date yet, but we are working hard to ship amp in 1.6 <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: tyb_10
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
awesome, a tentative one maybe ? (like in weeks, months) <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
actually i am moving my codebase to pytorch-lightening and I was reading that pytorch-lightening has already included the support of native cuda amp (whenever torch1.6) is released. I would like to use the FP16 training feature soon, otherwise i will have to think of relying on torch-nightly or apex in my project.</p>

username: ptrblck
cooked: <p>For now I would recommend to use the nightly binaries. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Creating a DimnameList from an std::vector<std::string> in C++

username: dfalbel
cooked: <p>How can I create a <code>DimnameList</code> from a vector of strings in C++?</p>
<p>I am trying something like:</p>
<pre><code class="lang-auto">torch::DimnameList cpp_torch_dimname_list (std::vector&lt;std::string&gt; x) {
 
 std::vector&lt;torch::Dimname&gt; out;
 
 for (int i = 0; i &lt; x.size(); ++i) {
   out.push_back(torch::Dimname::fromSymbol(torch::Symbol::dimname(x[i])));
 }
 
return out;
}
</code></pre>
<p>But this is not working as expected…</p>

username: sbrhss
cooked: <p>Maybe this code snippet can help you: <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
#include &lt;array&gt;
#include &lt;iostream&gt;
#include &lt;iomanip&gt;

torch::Dimname dimnameFromString(const std::string&amp; str);

int main()
{

	std::cout &lt;&lt; std::fixed &lt;&lt; std::setprecision(4);

	auto options =
	    torch::TensorOptions()
	    .dtype(torch::kFloat32);

	auto r = dimnameFromString("r"); // name of the first dimension
	auto c = dimnameFromString("c"); // name of the second dimension

	std::vector&lt;torch::Dimname&gt; names = { r, c }; // put them into a vector

	auto z_ = torch::zeros({5, 5}, names,options); // make a dummy zeros tensor
	std::array&lt;torch::Tensor, 2&gt; arrayRefz_ = {z_, z_}; // make a arrayRef

	auto cat_out_z = at::cat(arrayRefz_, c); // concatinate besed on the dimension names
	std::cout &lt;&lt; z_.names() &lt;&lt; std::endl; // print the dimension names

	std::cout &lt;&lt; "Concatinate z_ with z_: \n" &lt;&lt; cat_out_z  &lt;&lt; std::endl;
}

torch::Dimname dimnameFromString(const std::string&amp; str) {
	return torch::Dimname::fromSymbol(torch::Symbol::dimname(str));
}
</code></pre>
How to use Pytorch 1.5 without installing CUDA

username: Christian_Reyes
cooked: <p>I was wondering how we could use PyTorch with just cpus. There were previous builds which separated gpu/cpu, but using 1.5 allows for more complex indexing, so I wanted to find a way to use the most updated stable version of PyTorch without installing cuda (won’t be able to use GPUs based on hardware constraints).</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>For the CPU only version, you would have to select the <code>CUDA None</code> option on the <a href="https://pytorch.org/get-started/locally/">website</a>.</p>
<p>This command would install <code>1.5</code> without CUDA:</p>
<pre><code class="lang-auto">conda install pytorch torchvision cpuonly -c pytorch
</code></pre>

username: Christian_Reyes
cooked: <p>To clarify, I’m trying to use libtorch. Can I edit the libtorch cmake file (i.e. TorchConfig.cmake) in order to not care about CUDA installation?</p>

username: ptrblck
cooked: <p>The CPU libtorch package shouldn’t care about CUDA, right?<br>
What kind of error are you getting and how are you building your project?</p>

username: Christian_Reyes
cooked: <p>Oh my… I should go to sleep. I didn’t even see the None option for CUDA when downloading libtorch. I’m so sorry!!!</p>

username: ptrblck
cooked: <p>Haha, no worries! Let me know, if it still yields errors. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to use nn.LSTMCell to build nn.LSTM

username: wardenjohn
cooked: <p>I means, I can not find out the relationship between nn.LSTMCell and nn.LSTM.Alought I know that nn.LSTMCell is stacked to be nn.LSTM. But there is no output but just (h0,c0) in LSTMCell, well nn.LSTM has not only (h0,c0) but also output. So, is there any one can tell me how to use nn.LSTMCell to build nn.LSTM???</p>