akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Multiprocessing for-loop on CPU

Knievel: <p>I try to speed up a multidirectional RNN with <code>torch.multiprocessing</code> as I don’t get it to efficiently run on the GPU, but I have access to a lot of CPUs on a cluster. The one point where I want to apply multiprocessing is the for-loop over the different directions, as they are computed completely independent and only after all iterations are finished, the results are combined in a final layer.</p>
<p>So instead of</p>
<pre><code class="lang-auto">results = []
for d in directions:
    results.append(self.forward_direction(batch, param1, param2))
</code></pre>
<p>i tried doing</p>
<pre><code class="lang-auto">pool = mp.Pool(processes=self.num_directions)
results = pool.starmap(self.forward_direction, (batch, param1, param2))
</code></pre>
<p>This results in<br>
<code>MaybeEncodingError: Error sending result: '[tensor([...],        grad_fn=&lt;StackBackward&gt;)]'. Reason: 'RuntimeError('Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).',)'</code></p>
<p>Did not find a solution yet, is it even possible to multiprocess the for-loop without breaking autograd?</p>

albanD: <p>Hi,</p>
<p>You can look at the DistributedDataParallel module to perform distributed training.<br>
Otherwise, we very recently added to master support for autograd with the distributed package. (not the multiprocessing package, the distributed package).</p>

Brando_Miranda: <p>is that the answer even to the example I posted here: <a href="https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698/7" class="inline-onebox">How to parallelize a loop over the samples of a batch - #7 by Brando_Miranda</a> ? <a class="mention" href="/u/alband">@albanD</a></p>
<hr>
<p>related links from research:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/multiprocessing-for-loop-on-cpu/59836" class="inline-onebox">Multiprocessing for-loop on CPU</a></li>
<li><a href="https://stackoverflow.com/questions/56174874/how-to-use-multiprocessing-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">How to use multiprocessing in PyTorch? - Stack Overflow</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698/7" class="inline-onebox">How to parallelize a loop over the samples of a batch - #7 by Brando_Miranda</a></li>
<li><a href="https://stackoverflow.com/questions/66226135/how-to-parallelize-a-training-loop-ever-samples-of-a-batch-when-cpu-is-only-avai" class="inline-onebox" rel="noopener nofollow ugc">machine learning - How to parallelize a training loop ever samples of a batch when CPU is only available in pytorch? - Stack Overflow</a></li>
</ul>
Forward one object at a time or all objects in the image at a time as a target during object detection

Chame_call: <p>As I know we should label all objects we have in an image for certain category since otherwise loss function will penalize detections which are correct but not present in the target.</p>
<p>But it doesn’t work thus in practice for me.<br>
I’m using <code>torchvision.models.detection.fasterrcnn_resnet50_fpn</code> for custom object detection.<br>
I have images which contain from 1 to ~20 objects.</p>
<p>I tried two different approaches for creating <code>target</code> variable:</p>
<ol>
<li>
<code>target</code> variable consists of only one object every time:</li>
</ol>
<pre><code class="lang-auto">[{'boxes': tensor([[313,  34, 369,  62]], device='cuda:0'), 
'labels': tensor([13], device='cuda:0')}]
</code></pre>
<ol start="2">
<li>
<code>target</code> variable consists of all objects that are present in image:</li>
</ol>
<pre><code class="lang-auto">[{'boxes': tensor([[313,  34, 369,  62], [332, 244, 389, 274]], device='cuda:0'),
 'labels': tensor([13, 13], device='cuda:0')}]
</code></pre>
<p>In that specific task the smallest loss I could get is when create each individual target for each individual object (first case).</p>
<p>It doesn’t fit with theory so can someone tell me what I am doing wrong?</p>
Multiclass classification: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Ohyeahmanolito: <p>When I use cross-entropy loss function for multiclass text classification, I get the error “Dimension out of range (expected to be in range of [-1, 0], but got 1)”</p>
<p>This is my code:</p>
<pre><code class="lang-auto">def train(model, iterator):
    
    ...
    for batch in iterator:
        text, text_lengths = batch.Turn
        
        optimizer.zero_grad()
        predictions = model(text, text_lengths).squeeze(1)
        loss = criterion(predictions, batch.label)
        acc = categorical_accuracy(predictions, batch.label)

       ...
</code></pre>
<p>Dataset:</p>
<pre><code class="lang-auto">TEXT = data.Field(tokenize = 'spacy', include_lengths = True)
LABEL = data.LabelField(dtype = torch.long)
</code></pre>
<p>Forward:</p>
<pre><code class="lang-auto"> def forward(self, text, text_lengths):
      
        
        embedded = self.embedding(text)
   
        
        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths)
        
        packed_output, (hidden, cell) = self.rnn(packed_embedded)
  

        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))
        output = self.fc1(hidden)
        output = self.dropout(self.fc2(output))
              
            
        return output
</code></pre>
<p>LSTM setup for 3-class classification:</p>
<pre><code class="lang-auto">num_epochs = 25
learning_rate = 0.001
INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 128
OUTPUT_DIM = 1
N_LAYERS = 1
BIDIRECTIONAL = True
DROPOUT = 0.2
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token] # padding
</code></pre>

Dwight_Foster: <p>Can you print out the shape of your predictions and your labels?</p>
Reset pytorch sequential model during cross validation

talhaanwarch: <p>I am getting accuracy of each fold better than before. so I think my model is not resetting to initial state after a fold.<br>
Here is my code</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def__init__(self,...)
    super().__init()
    self.net=nn.Sequential(...)
def forward(self,x):
    return self.net(x)

def kfoldcv(model,data,...):
    
    kf = KFold(n_splits)
    fold=0
    train_cv=[]

    for train_index, test_index in kf.split(data.img):
        opt=torch.optim.Adam(params=model.parameters(),lr=lr)
        
        train=data.iloc[train_index,:].values
        test=data.iloc[test_index,:].values
        
        trainloader=dataloader(....)
        
        for batch in trainloader():
            out=model(batch[0])


kfoldcv(Model())
</code></pre>
<p>i tried this code from <a href="https://discuss.pytorch.org/t/how-to-reset-model-weights-to-effectively-implement-crossvalidation/53859/2">here</a></p>
<pre><code class="lang-auto">        
for name, module in model.named_children():
     print('resetting ', name)
     module.reset_parameters()
</code></pre>
<p>But it errors as follow<br>
<code>ModuleAttributeError: 'Sequential' object has no attribute 'reset_parameters' </code></p>

Dwight_Foster: <p>You could try the solution for <a href="https://discuss.pytorch.org/t/reset-model-weights/19180/4">this</a> post where you just define new random weights and apply them to your model.</p>
Can't download nightly wheels for ARM64

Fenouil: <p>Hello,</p>
<p>I can’t download nightly wheels for aarch64, I got “Acces denied”. Here the links that doesn’t work:</p>
<p>(at <a href="https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html</a>)<br>
torch-1.9.0.dev20210212-cp36-cp36m-linux_aarch64.whl<br>
torch-1.9.0.dev20210212-cp37-cp37m-linux_aarch64.whl<br>
torch-1.9.0.dev20210212-cp38-cp38-linux_aarch64.whl<br>
torchvision-0.9.0a0.dev20210212-cp36-cp36m-linux_aarch64.whl<br>
torchvision-0.9.0a0.dev20210212-cp37-cp37m-linux_aarch64.whl<br>
torchvision-0.9.0a0.dev20210212-cp38-cp38-linux_aarch64.whl</p>
<p>Thanks</p>
Image size for training in VGG

bhaswara: <p>Hi, I have dataset with the size of 720x1280. I want to train this images using VGG16 via transfer learning. Do I need to resize the images first into 224 x 224 so that I will fit into the VGG16 dimension or I don’t have to do that?</p>
<p>Thank you</p>

ptrblck: <p>Larger images should work in this model, as <code>torchvision</code> models use an adaptive pooling layer before feeding the activation to the first linear layer.<br>
However, since the model was pretrained on smaller images (<code>224x224</code>), you might want to fine-tune the conv layers so that they could extract the features from the current resolution, but it depends on your use case of course.</p>

nitaifingerhut: <p>as mentioned above VGG16 was trained on much smaller images, so it might not work as good on your data. In order to overcome this issue, you can add conv layer before feeding to VGG16, so the sizes after your conv layer will be at the same scale.</p>

bhaswara: <p>Thank you for your reply. Is it fine if I resize the images first, for example into 180x320, before I put into VGG16?</p>

nitaifingerhut: <p>you can but you may need to adapt the size of the linear layers.<br>
The issue mentioned before is more conceptual than technical- you have large images and the model was trained with different dataset, hence it may fail to perform on your images.</p>

bhaswara: <p>Thank you. So what you suggest is that I should put conv layer first which converts my dataset from 720x1280 into 224x224 then use VGG16 after that, correct?</p>

nitaifingerhut: <p>You can<br>
or you can do as you suggested<br>
sub sample your data set and feed to vgg16</p>
Learning best 'label smoothing' values

Mark_McPherson: <p>Consider a simple use of label-smoothing, on MNIST. You might use the label [0.01, 0.91, 0.01, …, 0.01] for the class ‘1’, similarly for the other 9 classes. I have an idea I would like to try out - have the network learn the best label <em>for each sample</em>. I think this would require back prop thru and updates to the actual labels. How can I set things up so that this occurs?</p>
How to import deep learning models from MATLAB to PyTorch?

Oubaida_Chouchane: <p>Hi everyone,</p>
<p>I’m trying to import a DNN trained model from MATLAB to PyTorch.</p>
<p>I’ve found solutions for the opposite case (from PyTorch to MATLAB), but no proposed solutions on how to import a trained model from MATLAB to PyTorch.</p>
<p>Any ideas, please?</p>
<p>Thanks in advance!</p>
Grad remains None for attributes, not for loose variables

Nifrec: <p>When I make a Tensor that requires_grad an attribute of an object and return a clone of it with an <code>@property</code> method, it does not obtain any gradient for backpropagation. However, the same computation does give gradients for other Tensors (<code>v</code> in the example below), eventhough they go through the same operations. Removing the <code>clone()</code> operation does work from the <code>my_parameter()</code> method does work. However, according to the documentation <code>clone()</code> should backpropagate changes to the original, and it does so for v. Could someone please hint me what I am missing?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.__my_parameter = nn.Parameter(torch.tensor([1.0], requires_grad=True))

    @property
    def my_parameter(self) -&gt; torch.Tensor:
        return self.__my_parameter.clone()

v = torch.tensor([1.0], requires_grad=True)
my_module = MyModule()
loss = torch.sum(v.clone() + my_module.my_parameter)
loss.backward()
print(v.grad, my_module.my_parameter.grad)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([1.]) None
</code></pre>

googlebot: <aside class="quote no-group" data-username="Nifrec" data-post="1" data-topic="112021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nifrec/48/34722_2.png" class="avatar"> Nifrec:</div>
<blockquote>
<p><code>my_module.my_parameter.grad</code></p>
</blockquote>
</aside>
<p>you’re calling the property getter here, my_module._my_parameter.grad would work (use single underscore to avoid name mangling, otherwise you parameter is only accessible as my_module._parameters[‘_MyModule__my_parameter’])</p>

Nifrec: <p>Thank you, single underscores do indeed seem to fix it.</p>
<p>But with double underscores the variable would still be accessible through <code>my_module._MyModule__my_parameter.grad</code>. However, in the example above, printing <code>my_module._MyModule__my_parameter.grad</code> leads to exactly the same result as just calling <code>my_module.my_parameter.grad</code>.</p>
<p>Does the name mangling impact autograd?</p>

googlebot: <p>idk, works for me:</p>
<pre><code class="lang-auto">my_module._MyModule__my_parameter.grad
Out[10]: tensor(cpu,(1,)[1.])
id(my_module._MyModule__my_parameter)
Out[11]: 158271096
id(my_module._parameters['_MyModule__my_parameter'])
Out[12]: 158271096
</code></pre>

Nifrec: <p>Indeed it does. Hmm strange, I am quite sure I got a different result a moment ago. <img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging.png?v=9" title=":man_shrugging:" class="emoji" alt=":man_shrugging:"> (Perhaps I made a typo somewhere? Or maybe Jupyter didn’t load the newest version properly?)</p>
<p>Either way, I can continue my work, thanks!</p>
Why the result is changed after model.eval()?

naoto-github: <p>I used DenseNet to classify images into some categories.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">
      <a href="https://pytorch.org/hub/pytorch_vision_densenet/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/hub/pytorch_vision_densenet/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>As you know, model.train() is called while training a model, and model.eval() is called while evaluating a model.</p>
<p>Before a test by using “evaluation data”,  I used “training data” to evaluate the model,<br>
If the mode is train(), the AAC was 96.25%, but the mode is changed to eval(),  the AAC was 83.02%.<br>
(Of course, the same model and the same training data)</p>
<p>Is there anyone who faces the same problem?<br>
What is the course of the above problem?</p>

nitaifingerhut: <p>switching to evaluation mode fixes some of the model’s parameters,<br>
for eg: batch norm uses collected statistics instead of actual batch statistics, dropouts are fixed, etc.<br>
all the above may effect the performance of your model over the data, especially since you use the training data for evaluation.</p>

naoto-github: <p>Thanks for your comment.<br>
I understand that the model’s parameters are fixed by model.eval().<br>
However, I’m not sure that such a big difference is caused.<br>
Whenever I use Densnet, should I set model.train() even for evaluation?</p>

nitaifingerhut: <p>first I suggest to evaluate the model on testset.<br>
you can try and see if there is a difference if when you evaluate you use with torch.no_grad() instead on switching to eval mode<br>
however no reason to perform inference in training mode</p>

naoto-github: <p>I will try to apply torch.no_grad() according to your advice.<br>
After the trial, I will report the result.<br>
Thanks.</p>

nitaifingerhut: <p>note that torch.no_grad() just turns off the gradients tracking while .eval() fixes parameters and randomness.</p>
Should Pytorch install in develop mode using 100% CPU?

ayrts: <p>Hello, I built Pytorch from source and installed in develop mode for making open-source contributions. However, I noticed that when I have to update my local repository to keep up-to-date with the main repository, I have to re-install Pytorch in develop mode as stated in the contributing doc by running <code>python setup.py develop</code> (Step 3).</p>
<p>Running this script takes around 40 minutes on a brand new Intel i7-10700K (8 core with hyperthreading) and uses 100% of the CPU. I am wondering if this is normal or is there a way I can reduce CPU usage while running this script? Thank you.</p>

ptrblck: <p>You could set the <code>MAX_JOBS</code> env variable to a specific number to reduce the CPU workload.</p>
Can I set num_workers in torchvision.datasets.Kinetics400?

111429: <p>Hi,<br>
I am trying to pre-compute the kinetics dataset.<br>
I try it with</p>
<pre><code class="lang-auto">dataset = torchvision.datasets.Kinetics400(
            traindir,
            frames_per_clip=250,
            step_between_clips=200,
            transform=transform_train,
            frame_rate=30,
            extensions=('avi', 'mp4', )
        )
</code></pre>
<p>But it took like ~160hours.<br>
I wonder if I can set num_workers, like dataloaders, to speed up.<br>
Thanks.</p>

ptrblck: <p>You should be able to pass this dataset to a <code>DataLoader</code> and set the number of workers there, which would use multiple processes to create the batches or are you seeing any errors with it?</p>

111429: <p>Hi, thanks for reply. No, no errors.  I understand what you mean. The problem for me is at this step for generating the <code>torchvision.datasets.Kinetics400</code> instance (before feed it to the dataloader), it took too long and exceed the time limit for my computing cluster. But I just realized I can add <code>num_workers</code>, like</p>
<pre><code class="lang-auto">dataset = torchvision.datasets.Kinetics400(
            traindir,
            frames_per_clip=250,
            step_between_clips=200,
            transform=transform_train,
            frame_rate=30,
            extensions=('avi', 'mp4', ),
            num_workers=4
        )
</code></pre>
<p>And it seems to be able to run smoothly, so I am testing if it helps.<br>
But still wonder if it make sense to add the <code>num_workers</code> argument here.</p>

ptrblck: <p>Yes, it seems you are right and don’t need to wrap this <code>Dataset</code> into a <code>DataLoader</code>.</p>
<p>While the <code>num_workers</code> argument is shown in the <a href="https://pytorch.org/vision/0.8/datasets.html#kinetics-400">docs</a>, it’s unfortunately not explained in the parameters.</p>
<p>Inside the <code>Kinetics400</code> dataset, a <code>VideoClips</code> object will be created, which accepts the <code>num_workers</code> argument as seen <a href="https://github.com/pytorch/vision/blob/61e00d586354d86af7f6b0222825c7810b7e4580/torchvision/datasets/kinetics.py#L52-L64">here</a>.<br>
Internally a <code>DataLoader</code> is created using the <code>num_workers</code> argument as seen <a href="https://github.com/pytorch/vision/blob/61e00d586354d86af7f6b0222825c7810b7e4580/torchvision/datasets/video_utils.py#L142-L147">here</a>.</p>
<p>So it seems this “Dataset” reverts the logic of passing a <code>Dataset</code> instance into a <code>DataLoader</code> and uses a <code>DataLoader</code> instead internally.<br>
I’m not sure, if this approach is used for all video datasets.</p>

111429: <p>I understand, thank you very much.</p>
Is it hard to detect an object without boundary edge?

Yolkandwhite: <p>Hi I am trying to train YOLO v1 model, which detects some kind of crack in human pelvic bone.</p>
<p>That crack thing appears on the same spot in pelvic bone.</p>
<p>I have about 50000 images and bounding box pair and trained it for 100 epochs</p>
<p>but the model performance is little disappointing.</p>
<p>So, I am assuming that if the object (crack in this case) doesn’t have exact boundary edge in the image,</p>
<p>the model won’t be train as I thought.</p>
<p>Is it right?</p>
PyTorch function similar to tf.set_shape

vishalthengane: <p>Dose PyTorch has something similar to <code>tf.set_shape()</code>. Link to the <a href="https://www.tensorflow.org/api_docs/python/tf/Tensor#set_shape" rel="noopener nofollow ugc">TF set_shape</a>.</p>

Dwight_Foster: <p>You can use the .view() function. <a href="https://pytorch.org/docs/stable/tensor_view.html" rel="noopener nofollow ugc">view</a></p>

nitaifingerhut: <p>You can use either .view() or torch.reshape</p>
Hanging on to a "sub-graph" after backward() to reduce computation time

rbuechler: <p>I am having an issue for my GAN model, when I am trying to train the discriminator. There is a portion of code that evaluates the discriminator with the real training data. This output makes up part of the loss function, but since I am doing multiple weight updates with every minibatch, this data does not change between iterations. Here is a pseudo-code snippet:</p>
<pre><code class="lang-auto">#  Part 1: Train D on real
d_real_decision = D(d_real_data)
d_real_error = torch.mean(d_real_decision)

for d_index in range(c["d_steps"]):
    D.zero_grad()

    #  Part 2: Train D on fake
    d_gen_input = Variable(torch.rand(c["samples_per_batch"], c["g_input_size"], device=device))
    d_fake_data = G(d_gen_input).detach()  
    d_fake_decision = D(d_fake_data)
    d_fake_error = torch.mean(d_fake_decision)

    # Part 3: Calculate Gradient Penalty (this is a custom function)
    grad_penalty = gradient_penalty(d_real_data, d_fake_data, c, device)

    # Part 4: Backward prop and gradient step
    total_d_error = -1 * (d_real_error - d_fake_error) + grad_penalty
    total_d_error.backward(retain_graph=False)
    d_optimizer.step()
</code></pre>
<p>As you can see, the loss term <code>total_d_error</code> includes <code>d_real_error</code>, which does not change with every iteration. I want to avoid including “Part 1” in the for loop to reduce runtime. I’ve tried:</p>
<p>(1) Making cloned versions of <code>d_real_error</code> inside the for loop. If I clone and detach, the gradients can’t backprop all the way back to the inputs of D. If I don’t detach, the “Part 1” graph is lost after the backward pass.<br>
(2) <code>retain_graph</code> True and False in the <code>backward</code> command, combined with both options above in (1). Also, if I retain the graph with neither of the options in (1), and just have Part 1 inside the for loop, it breaks because other portions of the loss function are changing and have in-place operations.</p>
<p>Long story short, I don’t want to include Part 1 in the for loop, because of high computational cost. But, I can’t figure out a way to do this properly without anything else breaking. Any help would be appreciated!</p>
Allocating with pin_memory() fails on sm_80 device

elb: <p>Hello,<br>
I have some code that has worked for a long time, that I am now running on a new A100 device, and is failing. I am using pytorch 1.7.1 and cudatoolkit11.0.</p>
<p>Any help is appreciated. Thanks!</p>
<p>(Note that a C++/CUDA program that uses cudaHostAlloc() works just fine on the A100 device)</p>
<p>relevant lines from conda list are:</p>
<pre><code class="lang-auto">cudatoolkit               11.0.221             h6bb024c_0
pytorch                   1.7.1           py3.7_cuda11.0.221_cudnn8.0.5_0    pytorch
</code></pre>
<p>Here is a MWE of the failing line on the A100 device:</p>
<pre><code class="lang-auto">Python 3.7.9 (default, Aug 31 2020, 12:42:55) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.get_device_capability()
(8, 0)
&gt;&gt;&gt; torch.cuda.get_device_properties(0)
_CudaDeviceProperties(name='A100-SXM4-40GB', major=8, minor=0, total_memory=40537MB, multi_processor_count=108)
&gt;&gt;&gt; torch.cuda.get_arch_list()
['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'compute_37']
&gt;&gt;&gt; cpu = torch.empty(1024, 3, 224, 224, dtype=torch.float32)
&gt;&gt;&gt; cpup = torch.empty(1024, 3, 224, 224, dtype=torch.float32).pin_memory()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: cuda runtime error (1) : invalid argument at /opt/conda/conda-bld/pytorch_1607370156314/work/aten/src/THC/THCCachingHostAllocator.cpp:278
</code></pre>
<p>Here is same  on a Quadro RTX8000 device:</p>
<pre><code class="lang-auto">
Python 3.7.9 (default, Aug 31 2020, 12:42:55) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.get_device_properties(0)
_CudaDeviceProperties(name='Quadro RTX 8000', major=7, minor=5, total_memory=48601MB, multi_processor_count=72)
&gt;&gt;&gt; torch.cuda.get_device_capability(0)
(7, 5)
&gt;&gt;&gt; torch.cuda.get_arch_list()
['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'compute_37']
&gt;&gt;&gt; cpu = torch.empty(1024, 3, 224, 224, dtype=torch.float32)
&gt;&gt;&gt; cpup = torch.empty(1024, 3, 224, 224, dtype=torch.float32).pin_memory()
&gt;&gt;&gt;  [ no error ]
</code></pre>

elb: <p>nevermind. Seem like there was a mis-configuration of the submission file on the server running the jobs that was the source of this error.</p>
Help with: "RuntimeError: One of the differentiated Tensors appears to not have been used in the graph" when trying to calculate attributions

ProGamerGov: <p>So, I have been trying to recreate a TensorFlow function in PyTorch, and the code below should in theory replicate the TensorFlow code. Though I appear to missing something as I get an error when testing the code. The <code>logit_activ</code> variable is the captured activations from a model’s final layer (before softmax), and the <code>layer_activ</code> variable is the capture activations from the target layer.</p>
<pre><code class="lang-auto">def fwd_gradients(logit_activ, layer_activ):
    zeros = torch.nn.Parameter(torch.zeros_like(logit_activ))
    grad_one = torch.autograd.grad(
            outputs=[logit_activ],
            inputs=[layer_activ],
            grad_outputs=[zeros],
            create_graph=True,      
        )[0]
    grad_two = torch.autograd.grad(
            outputs=[grad_one],
            inputs=zeros,
            grad_outputs=[torch.ones_like(grad_one)],
            create_graph=True,
        )[0]

    # Issue occurs somewhere after here:
    # Equivalent tensorflow code is: 
    # logit_attr = grad_two.eval({layer_name: layer_activ, d_previous: layer_zeros})

    layer_zeros = nn.Parameter(torch.zeros_like(target_activ))
    layer_zeros[..., x, y] = layer_activ[..., x, y]

    # Calc logit attribution
    logit_attr = torch.autograd.grad(
            outputs=[layer_activ],
            inputs=grad_two,
            grad_outputs=[layer_zeros],
            create_graph=True,     
        )[0]
    return logit_attr
</code></pre>
<p>The error message when running the above function looks like this:</p>
<pre><code class="lang-auto">1 frames

&lt;ipython-input-48-326f99eba6c0&gt; in fwd_gradients(logit, activ)
     21             inputs=out,
     22             grad_outputs=[layer_zeros],
---&gt; 23             create_graph=True,
     24         )[0]
     25     return out

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
    202     return Variable._execution_engine.run_backward(
    203         outputs, grad_outputs_, retain_graph, create_graph,
--&gt; 204         inputs, allow_unused)
    205 
    206 

RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
</code></pre>
<p>Any help resolving the issue would be appreciated!</p>
Access TwoMLPHead and FastRCNNPredictor separately from saved model after training

ajaychawda58: <p>I am training a FasterRCNN model using VGG16 as backbone network.</p>
<p>I save the model with torch.save(‘model.pt’)</p>
<p>Then I can load the model model = torch.load(‘model.pt’)</p>
<p>Now, I want to access some functions I added in the TwoMLPhead class and Fast RCNNPredictor class. But I am not able to access them, as I get the error. 'FasterRCNN ’ has no attribute ‘function_name’</p>
<p>How can I use the function from the class?</p>
<p>class TwoMLPHead(nn.Module):</p>
<pre><code>"""

Standard heads for FPN-based models

Arguments:

    in_channels (int): number of input channels

    representation_size (int): size of the intermediate representation

"""

def __init__(self, in_channels, representation_size):

    super(TwoMLPHead, self).__init__()

    self.fc6 = nn.Linear(in_channels, representation_size)

    self.fc7 = nn.Linear(representation_size, representation_size)

def forward(self, x):

    x = x.flatten(start_dim=1)

    x = F.relu(self.fc6(x))

    x = F.relu(self.fc7(x))

    return x

def extract(self, x, verbose=False):

    out1 = x.flatten(start_dim=1)

    out2 = F.relu(self.fc6(out1))

    out3 = F.relu(self.fc7(out2))

    if verbose == True:

        print(str(out1.shape) + ' ' + str(np.prod(out0.shape[1:])))

        print(str(out2.shape) + ' ' + str(np.prod(out1.shape[1:])))

        print(str(out3.shape) + ' ' + str(np.prod(out2.shape[1:])))

    return [out1, out2, out3]
</code></pre>
<p>extract() is added to classes. I wan to access the function from saved model.</p>
Export unsupported/compound ops to ONNX

gyenesvi: <p>I am working on a Python tool-chain that includes exporting PyTorch models to ONNX and reading them on a client side to process and execute them in a custom inference engine. I’d like to understand if it is possible to cover various use-cases that I have. I’d like to avoid solutions that require modifying the torch or onnx source code, as the toolchain will be used by other people, who will not have the modified source code, just the pip installed torch/onnx packages. My problematic use cases are:</p>
<ul>
<li>
<p>exporting unsupported PyTorch ops to ONNX, such that it can be read back on the client side: so far, I could export unsupported ops using torch.onnx.export(operator_export_type=ONNX_FALLTHROUGH) which does export unsupported ops as they are, but then the client side cannot process them as onnx shape inference (and checker) cannot be run. Is it possible to fix this somehow, by either injecting a shape inference function (could not find such on Python side, only C++ which is not what I need) or by exporting the tensor shapes into the model as well, such that shape inference is not required on the client side?</p>
</li>
<li>
<p>export python functions as a single op/function into ONNX: I saw in the ONNX IR some sort of (experimental?) support for functions, but could not find a way to export such functions from PyTorch. An example would be exporting a shuffle operation in ShuffleNet as a single op instead of a reshape-transpose-reshape sequence. Is that possible to do?</p>
</li>
</ul>
A vectorized way to find mention spans?

erolm-a: <p>Crossposted from <a href="https://discuss.pytorch.org/t/vectorizing-an-entity-memory-in-entities-as-experts/111314">here</a> but narrows down the question.</p>
<p>Suppose I have a matrix of rows of integers containing BIO outputs (assume O = 0, B = 1, I = 2).</p>
<pre><code class="lang-python">bio_outputs = torch.tensor([[0, 0, 1, 2, 2, 1, 2, 0, 1, 1],
                                #        1  1  1  2  2      3  4
                                 [1, 2, 2, 0, 0, 0, 1, 1, 0, 0]])
                                # 5  5  5             6  7

begin_positions = torch.nonzero(bio_outputs == 1)
end_positions = ...
assert(end_positions == torch.tensor([4, 6, 8, 9, 2, 6, 7])
</code></pre>
<p>The challenge is to compute <code>end_positions</code>. An end position can be either the last 2 (I) before an O (or the end of the tensor) or a new 1, or the start index itself. What I wish I could have is that end_position is a tensor of shape M (with M the number of detected 1s (Bs), in the example above 7). The reason for that is that I can perform a <code>torch.cat</code> on begin_positions to have the right start and end index for each mention.</p>
<p>I am using PyTorch 1.7.0 and cannot use experimental features like <code>vmap</code>.</p>
<p>Any ideas? Is this problems worth vectorizing?</p>
How are the batches created and sampled by Dataloader?

PhysicsIsFun: <p>Dear community,</p>
<p>if I create my batch collection of a data set using</p>
<p><code>batches = torch.utils.data.DataLoader(train_set, batch_size=bs, shuffle=True)</code></p>
<p>will any data point appear in exactly one batch? I.e. is the sum of batches equal to my entire data set?<br>
Or is each batch independently sampled from the entire data set, and it can happen that there are data points that don’t belong to a batch or that the same data point happens to be in different batches?</p>
<p>Also, if I do this:</p>
<pre><code class="lang-auto">    for epoch in range(epochs):
        for (batchidx, (features, targets)) in enumerate(batches_train):
            output = serial_net.forward(features)
</code></pre>
<p>Will the order of batches that is fed into the network be equal for each epoch?</p>
<p>Best,<br>
PiF</p>

ptrblck: <aside class="quote no-group" data-username="PhysicsIsFun" data-post="1" data-topic="111463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>will any data point appear in exactly one batch? I.e. is the sum of batches equal to my entire data set?<br>
Or is each batch independently sampled from the entire data set, and it can happen that there are data points that don’t belong to a batch or that the same data point happens to be in different batches?</p>
</blockquote>
</aside>
<p>If you are using the <code>shuffle</code> option in the <code>DataLoader</code>, a <code>RandomSampler</code> will be created as seen <a href="https://github.com/pytorch/pytorch/blob/d0fd41dcfe84a20f253ecaa0c74edbc68f92a973/torch/utils/data/dataloader.py#L267">here</a>. This sampler sets  <code>replacement</code> argument to <code>False</code> by default so that a random permutation will be applied as seen <a href="https://github.com/pytorch/pytorch/blob/d0fd41dcfe84a20f253ecaa0c74edbc68f92a973/torch/utils/data/sampler.py#L123-L124">here</a>.<br>
This makes sure that each sample is only drawn once in this setup.<br>
Note that the last batch might contain less than <code>batch_size</code> samples, of the length of the dataset (number of samples) is not evenly divisible by the <code>batch_size</code>. If you are using <code>drop_last=True</code> in the <code>DataLoader</code>, this last smaller batch will be removed and this epoch will not contain all samples from the dataset.</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="1" data-topic="111463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>Will the order of batches that is fed into the network be equal for each epoch?</p>
</blockquote>
</aside>
<p>No, it’ll be reshuffled in each epoch.</p>

PhysicsIsFun: <p>Thank you very much for clarifying!</p>
Erreur to execute after pyinstaller

aazz: <p>Hello,<br>
I have make a software to classification image.<br>
The soft run on my computer, so I have make an exe with pyinstaller to my client.<br>
My client have some error. I post to screen shot of this errors.<br>
Could you help me please ?<br>
Thank you</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://prntscr.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://prnt.sc/zlv5jx" target="_blank" rel="noopener nofollow ugc">Lightshot</a>
  </header>
  <article class="onebox-body">
    <img src="https://image.prntscr.com/image/5pUVZmOxRA_AKa6LiMXzdg.png" class="thumbnail onebox-avatar" width="16" height="16">

<h3><a href="https://prnt.sc/zlv5jx" target="_blank" rel="noopener nofollow ugc">Screenshot</a></h3>

<p>Captured with Lightshot</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://prntscr.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://prnt.sc/zlumup" target="_blank" rel="noopener nofollow ugc">Lightshot</a>
  </header>
  <article class="onebox-body">
    <img src="https://image.prntscr.com/image/fbniBmAlQZeLaepmrbapBg.png" class="thumbnail onebox-avatar" width="16" height="16">

<h3><a href="https://prnt.sc/zlumup" target="_blank" rel="noopener nofollow ugc">Screenshot</a></h3>

<p>Captured with Lightshot</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
ModuleNotFoundError: No module named 'networks.model_unet'

Soma: <p>Hello Everyone,</p>
<p>Can anyone please help me understand the reason why I fail to execute the following line in jupyter notebook  but when try the same in spyder it works ??</p>
<p><span class="hashtag">#---------------------------------------------------------------</span><br>
from networks.model_unet import Unet<br>
from utils.CustomDataset import Construct</p>
<p>import os<br>
import time<br>
import random<br>
import torch<br>
import torchvision<br>
import torch.nn as nn<br>
import torch.backends.cudnn as cudnn<br>
import torch.nn.functional as F<br>
import numpy as np<br>
import scipy.stats as st</p>
<p><span class="hashtag">#from</span> skimage.measure import compare_ssim, compare_psnr</p>
<hr>
<p>ModuleNotFoundError                       Traceback (most recent call last)<br>
 in <br>
----&gt; 1 from networks.model_unet import Unet<br>
2<br>
3 from utils.CustomDataset import Construct<br>
4<br>
5 import os</p>
<p>ModuleNotFoundError: No module named ‘networks.model_unet’</p>
<p>I shall be obliged for the help . Thank you.</p>

CedricLy: <aside class="quote no-group" data-username="Soma" data-post="1" data-topic="111860">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soma/48/34669_2.png" class="avatar"> Soma:</div>
<blockquote>
<p>nyone please help me understand the reason why I fail to execute the following line in jupyter notebook but when try the same in spyder it w</p>
</blockquote>
</aside>
<p>The error suggests that no module was found in your networks directory.<br>
You can try to check two things:</p>
<ol>
<li>Do you have a networks directory with an model_unet.py</li>
<li>At what path does your jupyter notebook starts.</li>
</ol>

Soma: <p>Hello CedricLy,</p>
<p>Thank you for the suggestion.</p>
Where is __str__ and __repr__ methods in torch.optim.Optimizer

omarfoq: <p>Hello</p>
<p>When printing an optimizer instance, we get a list of parameter groups, which is normal, however I can’t find the <code>__str__</code> and <code>__repr__</code> methods in Optimizer class or in its children. Do you you know where is the script responsible for this print.</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/blob/1903b32c35a7e7432d62e43a2d3a679f6f01066d/torch/optim/optimizer.py#L67-L76">This <code>__repr__</code> method</a> should be used.</p>
What should be the input shape for 3D CNN on a sequence of images?

Augustas_Macys: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html#conv3d" class="inline-onebox" rel="noopener nofollow ugc">Conv3d — PyTorch 1.7.1 documentation</a> Describes that the input to do convolution on 3D CNN is (N,C<sub>in</sub>,D,H,W). Imagine if I have a sequence of images which I want to pass to 3D CNN. Am I right that:</p>
<ol>
<li>N → number of sequences (mini batch)</li>
<li>C<sub>in</sub> → number of channels (3 for rgb)</li>
<li>D → Number of images in a sequence</li>
<li>H → Height of one image in the sequence</li>
<li>W → Width of one image in the sequence</li>
</ol>
<p>The reason why I am asking is that when I stack image tensors: <code>a = torch.stack([img1, img2, img3, img4, img5])</code> I get shape of a <code>torch.Size([5, 3, 396, 247])</code>, so is it compulsory to reshape my tensor to <code>torch.Size([3, 5, 396, 247])</code> so that number of channels would go first or it does not matter inside the Dataloader?</p>
<p>Note that Dataloader would add one more dimension automatically which would correspond to N.</p>

ptrblck: <p>You could use the depth dimension to stack a sequence of images. Note however, that the kernels in each conv layer would also have a depth dimension and would thus convolve through it as well (unless you set the kernel size to 1 for the depth).<br>
Often <code>nn.Conv3d</code> is used on a “stack” of images, such as medical CT scans using slices for the depth dimension.<br>
Depending on your use case, the same approach might also work for your “sequences”.</p>
Why empty parameter list?

akt_09: <pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
    
    def forward(self,x):
        m=nn.LSTMCell(x.shape[1],x.shape[1]*2)
        output,_=m(x)
        dense=nn.Linear(output.shape[1],1)
</code></pre>

ptrblck: <p>Your current code snippet recreates the <code>nn.LSTMCell</code> and <code>nn.Linear</code> modules in each forward pass randomly and doesn’t register them.<br>
The proper way is to initialize submodules in the <code>__init__</code> method and use them in the <code>forward</code>.</p>
IndexError: Target <value of the first label in y_train> is out of bounds

shizar_zarbafian: <p>Dear all,</p>
<p>I am trying to train a model on a dataframe.<br>
I was following a training at this address:<br>
<a href="https://medium.com/swlh/my-first-work-with-pytorch-eea3bc82068" rel="noopener nofollow ugc">https://medium.com/swlh/my-first-work-with-pytorch-eea3bc82068</a></p>
<p>This is my code until the error pops up:</p>
<pre><code class="lang-auto">from datetime import time

import torch
import torch.nn as nn
import torch.nn.functional as F
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
import kaggle



price = pd.read_csv('D:/Kaggle Datasets/Beer/price.csv')


X = price.drop('price', axis=1).values
y = price['price'].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train = torch.FloatTensor(X_train)
X_test = torch.FloatTensor(X_test)
y_train = torch.LongTensor(y_train)
y_test = torch.LongTensor(y_test)

print(X_train)


print(len(X_train))
print(y_train)

class ANN(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(in_features=5, out_features=10)
        self.fc2 = nn.Linear(in_features=10, out_features=4)
        self.output = nn.Linear(in_features=4, out_features=1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.output(x)
        return x


model = ANN()
print(model)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)


print(time)
epochs = 5
loss_arr = []

for i in range(epochs):
    y_hat = model.forward(X_train)
    loss = criterion(y_hat, y_train)
    loss_arr.append(loss)

    if i % 10 == 0:
        print(f'Epoch: {i} Loss: {loss}')

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


</code></pre>
<p>My dataframe is different from the dataframe in the reference. Mine looks like this:</p>
<p>A,B,C,D,E,price<br>
70.00,1.30,6.00,132.00,66.00,55528.00<br>
52.00,1.30,15.00,512.00,600.00,285792.00<br>
53.00,1.00,9.00,105.00,44.00,56700.00<br>
65.00,1.00,22.00,215.00,115.00,73068.00<br>
58.00,1.30,6.00,186.00,112.00,101293.00<br>
73.00,1.30,4.00,104.00,31.00,79100.00<br>
77.00,1.15,11.00,36.00,71.00,74100.00</p>
<p>The purpose is to calculate the price column so this is not a classification.<br>
This is the error:</p>
<pre><code class="lang-auto">    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 55528is out of bounds.
</code></pre>
<p>As you can see, the error points to the first label of my dataframe which happens to be first label of my train data (y_train) as well.</p>
<p>Maybe I should do a normalization but I am not sure since my experience and knowledge is limited.<br>
I appreciate your help.</p>

ptrblck: <p>Your use case sounds like a regression in which case you could use e.g. <code>nn.MSELoss</code> as the criterion instead of <code>nn.CrossEntropyLoss</code>, which is used for a multi-class classification/segmentation.</p>
Help in understanding pytorch implementation of VAE

Shravya_Kuldeep: <p>I am trying to implement a simple VAE. By reading theory I understood that encoder tries to learn Mu and Sigma for a distribution which is as close as the true distribution with unit variance and zero mean.<br>
In the VAE implementation of <a href="https://github.com/pytorch/examples/blob/master/vae/main.py" rel="noopener nofollow ugc">here</a> I am unable to understand how does the linear layer return mean and variance. (line number 51).</p>
<p>return self.fc21(h1), self.fc22(h1)</p>
<p>Also, does the line<br>
self.fc21 = nn.Linear(400, 20) mean that we are trying to learn 20 mean’s or basically 20 distributions each for a specific feature in my latent space.</p>