akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I wanna to fixed the start city 0 for each tour solution.... please i need some help?

username: Younes_Sellimi
cooked: <pre><code class="lang-auto"></code></pre>
<pre><code class="lang-auto">import os
import numpy as np
import torch
from torch.utils.data import Dataset
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt


class TSPDataset(Dataset):

    def __init__(self, size=50, num_samples=1e6, seed=None):
        super(TSPDataset, self).__init__()

        if seed is None:
            seed = np.random.randint(123456789)

        np.random.seed(seed)
        torch.manual_seed(seed)
        self.dataset = torch.rand((num_samples, 2, size))
        self.dynamic = torch.zeros(num_samples, 1, size)
        self.num_nodes = size
        self.size = num_samples
       

    def __len__(self):
        return self.size

    def __getitem__(self, idx):
        # (static, dynamic, start_loc)
        return (self.dataset[idx], self.dynamic[idx], self.dataset[idx, :, 0:1])


def update_mask(mask, dynamic, chosen_idx):
    """Marks the visited city, so it can't be selected a second time."""
    mask.scatter_(1, chosen_idx.unsqueeze(1), 0)
    return mask
</code></pre>
<pre><code class="lang-auto"></code></pre>

username: ptrblck
cooked: <p>I don’t quite understand the question in the title related to the posted code snippet.<br>
Could you explain your use case as well as the problem you are facing a bit more?</p>

username: Younes_Sellimi
cooked: <p>Ok , i 'am looking for create a TSP envirenement to solving by deep reinforcment learning<br>
the class TSPDataset is my env_tsp, getItem is the methods to selecte the action in every step, i assume that is the key to fixed the start city in each predicted tour. The TSPdataset  is a combination of dataset which contain the (xcoords, ycoords) and a dynamic vector contain the city index.<br>
In briefly i whish to help me, to fixed the start city that is city with index 1 and used  it as a  rule for agent policy .<br>
for example : i want to give me a permutatio n of city like this:<br>
0,4,8,1,4,9…,0<br>
0,1,3,9,5,7,…,0<br>
thank’s</p>
Ensemble model In a stepwise fashion

username: Jamesong7822
cooked: <p>Hi Pytorch Gurus,</p>
<p>I found a similar question, but it was left unanswered: <a href="https://discuss.pytorch.org/t/training-an-ensemble-model-with-and-without-saperately/120145" class="inline-onebox">Training an ensemble model with and without saperately</a></p>
<p>I believe I have a similar question.<br>
I would like to ensemble 2 models together which I have trained. The classifier will be in charged of predicting whether there will be retweets or not (basically a 0 or 1). The predictor was trained only with retweets data and will only be used if the classifier model returns a 1.</p>
<p>However, I have some issues with the forward function of this “Ensemble” Model, due to batch size problems. The workaround of setting batch_size=1 works, but this would mean it is very inefficient and slow. Could I seek for advice and suggestions how to maintain a large batch_size?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a56f150b9aa97a5ed966edc31a25143fa71ee7a.jpeg" alt="image" data-base62-sha1="m1lNRjtAjGAbVP5XjNuqzP96hei" width="646" height="445"><br>
Without a batch_size of 1, it gives an ambiguity error: “RuntimeError: bool value of Tensor with more than one value is ambiguous”</p>

username: ptrblck
cooked: <p>As the error message claims, you cannot use an <code>if condition</code> on multiple values and would need to split the tensor.<br>
Assuming you want to pass the values matching the condition to <code>self.modelPredictor</code> and return the others, you could apply this workflow:</p>
<pre><code class="lang-python"># output from previous layer
x = torch.randn(10, 1)

# create condition
idx = x &gt; 0.

# split the data based on condition
x_true = x[idx]
x_false = x[~idx]

# check shapes
print(x_true.shape)
&gt; torch.Size([3])
print(x_false.shape)
&gt; torch.Size([7])

# apply special path for True values
out_true = self.modelPredictor(x_true)

# concatenate outputs
out = torch.zeros_like(x)
out[idx] = out_true
out[~idx] = x_false
</code></pre>
<p>You might also want to add a few more checks to handle e.g. empty tensors etc.</p>

username: Jamesong7822
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for taking the time for your response.</p>
<p>Yes, I am aware of the message error, and I decided to take a short break from the project. After a well rested break, I realised that the solution I am looking for was relatively simple and I have overlooked it.</p>
<p>As my main intention was to use the binary classifier to feed into my predictor network, simply rounding the output of the binary classifier followed by multiplying it with the output from my predictor network would yield the answer. This allows for <code>batch_size &gt;1</code> evaluation as well, which was good enough in my books!<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8ff18086dd9d5c044bcefe8674b851bac2e4b44a.png" alt="image" data-base62-sha1="kxnH1UOO4FCbzygJ8McTnq0H4n0" width="637" height="359"></p>
RNN Implementation for point classification

username: drbeethoven
cooked: <p>Hello,</p>
<p>I understand most implementation of RNN’s are used with NLP, at least all the ones I have seen online. Has anyone found an implementation that uses classification with a group of points?</p>
<p>So the sequence length will be the number of points in a group, but I am confused on such parameters like “dict_size” that I have frequently seen in RNN implementations. I am not sure if the “dict_size” comes from the number of possible inputs (infinite because the points are on a continuous scale) or if it comes from the number of possible outputs (2 here because)I am currently only classifying between two classes.)</p>
<p>I am using an RNN because I want the choice of classification for one point to affect the classification of subsequent points.</p>
<p>Thank you so much for the help!</p>
Need help RuntimeError: CUDA error: device-side assert triggered

username: mishi
cooked: <p>Hii, I’ have made a Graph convolution network which takes 2 graphs as inputs and gives output between 0 to 1 , After running on one sample , it shows this error . Please help</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61d854439e4bec2335429d1abc5347a7ec29b2f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/61d854439e4bec2335429d1abc5347a7ec29b2f6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61d854439e4bec2335429d1abc5347a7ec29b2f6_2_690x328.png" alt="image" data-base62-sha1="dXzNspFNaBmvU8pAcqPu5GpM6QS" width="690" height="328" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61d854439e4bec2335429d1abc5347a7ec29b2f6_2_690x328.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61d854439e4bec2335429d1abc5347a7ec29b2f6_2_1035x492.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/1/61d854439e4bec2335429d1abc5347a7ec29b2f6.png 2x" data-dominant-color="302E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1089×518 61.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The error points to an invalid index.<br>
You could rerun the code either on the CPU to get the line of code causing the issue or on the GPU via <code>CUDA_LAUNCH_BLOCKING=1</code> as suggested in the error message.</p>

username: mishi
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> .  I was working with graphs and after trying to run on cpu ,i found that there was a mismatch between number of nodes (N) and shape of adjacency matrix which in ideal case should be N x N .</p>
CUDA parallel for loop with BERT tokenizer

username: matteomedioli
cooked: <p>I need to detokenize a batch of 8 input_ids tensors and apply a function to each single sentence tensor. I have a <code>function()</code>:</p>
<pre><code class="lang-python">def function(sentence):
    for source in sentence:
        for target in sentence:
            # DO STUFF WITH source AND target
</code></pre>
<p>And a model with a <code>forward()</code> method:</p>
<pre><code class="lang-python">def forward(input_ids, tokenizer):
    sentences_batch = tokenizer.batch_decode(input_ids, skip_special_tokens=False)
    for sentence in sentences_batch:
        tensor = function(sentence)
        batch.append(tensor)
    result = torch.stack(batch)
    # DO STUFF WITH result
</code></pre>
<p>Does exist a way to leverage CUDA to run in parallel the for loop in the <code>forward()</code> method? Will <code>.to(device)</code> solve my problem? If yes, how I can put this statement?<br>
I run the training script where <code>forward()</code> appears with:</p>
<p><code>python3 -m torch.distributed.launch --nproc_per_node 1 training.py</code></p>
<p>Thanks in advance.</p>
How to find if the layer is convolutional and linear from the entire network, and then read its parameters?

username: Ajinkya.Bankar
cooked: <p>I want to find if the layer is convolutional and linear from the entire network. If it is one of them, then I want to read the shape and parameters of the layer. I have found a solution to check if the module is convolutional or linear <a href="https://discuss.pytorch.org/t/how-do-you-determine-the-layer-type/19309/2">here</a>. But I don’t find a way to read the parameters and shape of the corresponding layer. Would you please suggest a solution?</p>

username: ptrblck
cooked: <p>The posted condition would be the right approach and once the condition is met you could use e.g. <code>layer.weight</code> and <code>layer.bias</code> or <code>layer.parameters()</code> depending on your use case.</p>

username: Vrushank98
cooked: <p><a class="mention" href="/u/ajinkya.bankar">@Ajinkya.Bankar</a><br>
Do you want to know the number of parameters &amp; shape, or just copy/read the weights of each layer?</p>

username: Ajinkya.Bankar
cooked: <p><code>layer.weight.data</code> worked for me. Thank you.</p>
Does PyTorch/FBGEMM support AVX512 (VNNI) for INT8 Quantization and does it improve performance?

username: saykartik
cooked: <p>The <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">PyTorch Quantization doc</a> suggests that for efficient optimization, we must use a CPU that has AVX2 support or higher.<br>
If we were to consider transformer class models trained/quantized and served on x86 architectures using FBGEMM as the Quantization Engine,</p>
<ul>
<li>Does INT8 quantization using native pytorch APIs take advantage of AVX512 instruction set, and if so in which version of PyTorch was this support introduced?</li>
<li>If AVX512 is supported, what should we expect in terms of performance impact on latency and throughput by moving from AVX2 to AVX512 using native pytorch quantization support?</li>
<li>Might there be any benchmarks which already show the quantified improvement?<br>
Notes:</li>
<li>By Native pytorch quantization, we mean what is natively supported by PyTorch without export to ONNX runtime</li>
<li>I See a discussion here on <a href="https://github.com/pytorch/pytorch/issues/56187" rel="noopener nofollow ugc">Vec512 and AVX512</a>, and couldn’t quite follow if
<ul>
<li>A) this support is already introduced, and</li>
<li>B) what the actual performance difference is</li>
</ul>
</li>
</ul>
<p>Would be great to get some insight here.</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <span class="mention">@saykarthik</span>, please feel free to take a look at <a href="https://github.com/pytorch/pytorch/pull/56992" class="inline-onebox" rel="noopener nofollow ugc">Add AVX512 support in ATen &amp; remove AVX support by imaginary-person · Pull Request #56992 · pytorch/pytorch · GitHub</a> which is in progress and which aims to add this support.  There are performance numbers planned to be added the PR before it’s ready to land (they might not be there yet).</p>
DC-GAN Training Loss Instability

username: jman4162
cooked: <p>Sometimes when I am training a DC-GAN on an image dataset, similar to the DC-GAN PyTorch example (<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">DCGAN Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a>), either the Generator or Discriminator will get stuck in a large value while the other goes to zero. How should I interpret what is going on right after iteration 1500 in the example loss function image shown below? Is this an example of mode collapse? Any recommendations for  how to make the training more stable? I have tried reducing the learning rate of the Adam optimizer with varying degrees of success. Thanks!</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5518f599f476ffc1f9e82dc4ab0a65303c44298f.png" alt="lossFunc" data-base62-sha1="c8O3JsHyLU7nJi3o1QXGhOC3b4z" width="609" height="333"></p>

username: import-antigravity
cooked: <p>You won’t know that it’s mode collapse or not unless you look at the images</p>
Bus error (core dumped) keep happening during training

username: malteekj
cooked: <p>I keep getting the error message ‘Bus error (core dumped)’ at random times during training.</p>
<p>I’m running Ubuntu 18.04 with anaconda, conda 4.8.1, Python 3.7.4 and pytorch 1.4</p>
<p>I have a GTX 2080 Ti, and it only uses about 3 GB of RAM, so it shouldn’t be that.</p>
<p>Does anyone have a suggestions for what to try?</p>

username: Lornatang
cooked: <p>Generally speaking, SIGILL error occurs when the binary contains instructions not supported by CPU, because the installation method of the official website is binary, that is to say, the official binary does not match my CPU (AMD instead of Intel), so it needs to be installed from source, as follows</p>
<pre><code class="lang-nohighlight">git clone https://github.com/pytorch/pytorch

sudo conda uninstall pytorch torchvision -c soumith

sudo python3 setup.py install
</code></pre>

username: QingchaoShen
cooked: <p>I come across the same question, could you solve it? Thanks in advance.</p>
Cannot use tensor in dataloader, RuntimeError: Cannot re-initialize CUDA in forked subprocess

username: isalirezag
cooked: <p>I save a tensor using <code>torch.save(Tensor, A.pth)</code> and load it in my dataloader process via <code>T = torch.load("A.pth")</code><br>
Then i cahnge the tensor to numpy via <code>T_Numpy =T.data.cpu().numpy()</code>,<br>
Why am I getting the following error:</p>
<pre><code class="lang-auto">    "Cannot re-initialize CUDA in forked subprocess. " + msg)
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
</code></pre>
<p>How I can fix it? <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=6" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>P.S. I am using Dataparallel and im in pytorch 0.3.0</p>

username: isalirezag
cooked: <p>I figured out if I save the tensors withough variable meaning that use Tensor.data for saving that problem will disappears <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=6" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: morgankohler
cooked: <p>Since I had this problem but this solution did not work for me (possibly because Variable usage is now deprecated) the solution was to load the tensor onto the cpu via the map_location argument. When using dataloader with torch.multiprocessing, loading the tensor to gpu (which was being done automatically) caused this issue. Testing without torch.multiprocessing and this error did not occur.</p>
Definition of 'dense' in ConvNets

username: sigma_x
cooked: <p>Reading Long et al (2015, FCN), Huang et al (2018, DenseNet), Chen et al (2017, atrous convolution), Badrinarayanan et al (2017, SegNet), Chen et al (2017, DeepLab), Chen et al (2016, CRFs), I got confused by the use of term ‘dense’.</p>
<p>From what I understood, in Long et al, ‘dense’ features are the opposite of ‘sparse’ features, i.e. feature maps with few or no zeros instead of many 0s obtained by convolving with the sparse features, so that the features are ‘densified’ after convolution.</p>
<p>At the same time, in Chen et al (atrous convolution), it says ‘extract denser feature maps by removing the downsampling operations from the last few layers and upsampling the corresponding filter kernels, equivalent to inserting holes between filter weights’. I don’t quite understand this statement, as it seems to apply another meaning to the term ‘dense features’ than the one above. It seems to relate to the term ‘feature resolution’, meaning, roughly speaking, how much the features in the layer resemble the input image.</p>
<p>In Huang et al, Chen et al (2016) and Chen et al (DeepLab) it seems that ‘dense connections’ simply means ‘many connections’, e.g. thorough connections to the previous concatenated layers (DeepLab), or through fully connected layers (CRF, DeepLab).</p>
<p>Finally, ‘dense prediction/problem’ means ‘predict every pixel’, e.g. semantic segmentation, or simply ‘predict many things’, e.g. object detection.</p>
<p>Could someone clarify that?</p>
Can i use model.half() to replace autocast when doing inference?

username: treasureqqm
cooked: <p>i have a trained model(mainly based on maskrcnn-benchmark, and modules include conv, linear, upsample) and use fp16 when doing inference. I used model.half() and patch_norm_fp32( borrowed from mmcv to stabilize bn) and it works well. So i wonder 1) can i just use the above method(i.e. model.half() and patch_norm_fp32) instead of autocast api? Or are there any differece between the two? 2) And i have done a few simple tests to find that autocast is faster than model.half(), is it normal?</p>

username: ptrblck
cooked: <p><code>torch.cuda.amp.autocast()</code> uses an internal mapping of operations, which have to use FP32 for numerical stability as descrbed <a href="https://pytorch.org/docs/stable/amp.html#op-eligibility">here</a>. If your model works fine (and the accuracy doesn’t decrease), I don’t see an argument against using <code>model.half()</code>.<br>
No, I wouldn’t expect <code>autocast</code> to be faster.</p>

username: treasureqqm
cooked: <p>Thank you for your answer, i will do some experiments to test the speed.</p>
Weighted average pooling

username: kuonlp
cooked: <p>Hello. I’m trying to develop a “weighted average pooling” operation. Regular avg pooling takes a patch and gives you the average, but I want this average to be weighted. This can be easily achieved with a convolution by convolving the weight (say, a 3x3 kernel) with the feature maps. However, there is a fundamental difference between convs and pooling operations: the latter is applied batch- and channel-wise, whereas convs will mix up the channels.</p>
<p>There are two ways to solve this:</p>
<ol>
<li>
<p><code>Conv(num_fi, num_fi 3, group=num_fi)</code> so that the convolutions are performed independently. One problem with this approach is that you need to artificially duplicate the kernel size to the number of filters. Thus, at forward time, you cannot receive a different number of filters; so, in practice, you would need one of this convolutions for every feature map you want to pass.</p>
</li>
<li>
<p>In order to accept any number of filters at forward time, you could reshape the tensors. Therefore, your convolution would be defined as <code>Conv(1, 1, 3)</code> and you reshape the tensor to (B*C, 1, H, W) and later reshape it back to its original size.</p>
</li>
</ol>
<p>The problem with these approaches is that they are not as efficient as pooling, taking twice is time. For this reason, I would like to know if there is a way to tweak Avg Pool to multiply the patch that is averaged with a constant matrix of weights. Any other suggestions on how to tackle this is also appreciated.</p>
<p>Thanks!</p>
Model not using entire GPU memory

username: aqib
cooked: <p>Hi all,</p>
<p>I am quite new to Pytorch so question might be naive. I have installed Pytorch version 1.7.1 with CUDA 10.1. I have K80 GPU, when I train a model it only uses around 1.5 GB of memory instead of utilizing completely. Is it the expected default behavior in Pytorch?</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>Yes, this might be expected and depends on the actual model, data shapes, as well as the general memory requirement of the training script. You could compare it to your normal workstation, which could also run e.g. Firefox without filling all RAM.</p>

username: aqib
cooked: <p>Makes sense! I was comparing it with TF process, which by default use entire GPU memory. Thanks.</p>

username: ptrblck
cooked: <p>Yes, I believe TF allocated all available memory by default (and might not be using it), while PyTorch adds the allocated memory if needed and adds it to the internal cache to reuse it later.</p>
Can't understand the output shape of Identity layer while using it to replace Fully Connected layer

username: Damon_Roux
cooked: <p>I’m computing the Frechet Inception Distance(FID) with Inception-v3 Network.</p>
<p>And I try to replace the Fully Connected layer with Identity layer to extract the features.</p>
<p>Before implementing the replacement <code>inception_model.fc = torch.nn.Identity()</code>, check the model summary and get the output shape of last two layers as below:</p>
<ul>
<li>InceptionE-293          [-1, 2048, 8, 8]</li>
<li>Linear-294                 [-1, 1000]</li>
</ul>
<p>After implementing the replacement, check and get the output shape of last two layers as below:</p>
<ul>
<li>InceptionE-293          [-1, 2048, 8, 8]</li>
<li>Identity-294                 [-1, 2048]</li>
</ul>
<p>The confusion is:<br>
why the output shape of Identity layer is [-1, 2048], and not the same as its input, which is [-1, 2048, 8, 8] ?</p>
<p>Thanks~</p>

username: zetyquickly
cooked: <p>Hello <a class="mention" href="/u/damon_roux">@Damon_Roux</a>,</p>
<p>Have you looked at the model text description? There might be layers or layers between 293 and 294.</p>
<p>I don’t know exactly which implementation of InceptionV3 do you use but it is highly likely that there might be <code>AvgPool2d</code> or some other pooling layer between <code>InceptionE-293</code> and <code>Linear-294</code>. Pooling layer reduces the dimensionality in the first case when the tensor passed to a fully connected layer and in the second case when it’s passed to identity layer.</p>
Dataset, Dataloader error

username: arya47
cooked: <p>I have defined a custom dataset where  <strong>getitem</strong>  involves some heavy processing.<br>
When running a dataloader i get the error<br>
(not passed num_workers to the dl)</p>
<pre><code class="lang-auto">train_image, train_labels = next(iter(train_dl))
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/queues.py", line 242, in _feed
    send_bytes(obj)
  File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/queues.py", line 232, in _feed
    close()
  File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 177, in close
    self._close()
  File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 361, in _close
    _close(self._handle)
OSError: [Errno 9] Bad file descriptor
</code></pre>
<p>However a value is assigned to train_images and train_labels.</p>
<p>I believe this error has to do with multithreading/forking and pipes in the dataloader implementation.<br>
How can this be resolved?</p>
<p>Thanks</p>
Learning to transform CNN filter weights

username: robinhe
cooked: <p>Hi, I have the following problem for which I can not find a solution:</p>
<p>There are two models. A pre-trained CNN, which can be assumed to be fixed. And another fully connected model to be trained, which transforms the filter weights of the CNN based on a parameter.</p>
<p>For example, a possible learned transformation could be a rotation of the filters by a certain angle. This way, the transformation model could transform the CNN such that it might be capable of classifying rotated images.</p>
<p>My problem is that I do not manage to get gradients for the transformation model. I guess when assigning the parameter weights, the graph is broken. Since the problem is probably easier to understand with code, here is a small illustrative example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import copy

# define models
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv = nn.Conv2d(1, 1, 3)

    def forward(self, x):
        return self.conv(x)
    

class TransformationNet(nn.Module):
    def __init__(self):
        super(TransformationNet, self).__init__()
        self.linear = nn.Linear(10, 9, bias=False)

    def forward(self, x, param):
        x = torch.cat([x,param], dim=1)
        return self.linear(x)
    
model_cnn = CNN()
# Since we are changing the CNN filter weights while training, the copy is used to keep the original pre-trained model. 
model_cnn_copy = CNN()
model_cnn_copy.load_state_dict(copy.deepcopy(model_cnn.state_dict()))

model_transformation = TransformationNet()
model_transformation.train()

# only optimize the model for transforming the filters
optimizer_model_transformation = torch.optim.SGD(model_transformation.parameters(), lr=0.1, momentum=0.9)

# training loop
for i in range(2):
    B = 1 # batch size
    image = torch.ones((B,1,3,3)) # some input    
    
    # update the CNN using the model for transforming the filters. The second argument torch.tensor([[1.0]]) is also just chosen arbitrarily for this example.
    model_cnn.conv.weight.data = model_transformation(model_cnn_copy.conv.weight.data.view(B, -1), torch.tensor([[1.0]])).view((B,1,3,3))
    
    out = model_cnn(image)
    
    # just some arbitrary loss
    loss = out.sum()
    
    optimizer_model_transformation.zero_grad()
    loss.backward()
    print(model_transformation.linear.weight.grad)
    optimizer_model_transformation.step()
    
    print(loss.item())
</code></pre>
<p>The gradient of the linear layer is None and the loss stays constant. I assume the line</p>
<pre><code class="lang-auto">model_cnn.conv.weight.data = model_transformation(model_cnn_copy.conv.weight.data.view(B, -1), torch.tensor([[1.0]])).view((B,1,3,3))
</code></pre>
<p>causes the problem.</p>
<p>Thank you very much for any hints or ideas!</p>
Given list of index, obtain the corresponding element from a tensor

username: yuejiang_li
cooked: <p>Hi, I have a question of getting certain indexed element from a tensor.</p>
<p>For example, I have a tensor</p>
<pre><code class="lang-python">a = torch.arange(24).to(torch.float).reshape(2, 3, 4)
a.requires_grad = True
print(a)
# output
tensor([[[ 0.,  1.,  2.,  3.],
         [ 4.,  5.,  6.,  7.],
         [ 8.,  9., 10., 11.]],

        [[12., 13., 14., 15.],
         [16., 17., 18., 19.],
         [20., 21., 22., 23.]]], requires_grad=True)
</code></pre>
<p>I also have a list of indexes, and I want to extract the element from tensor <code>a</code> with the indexes. Therefore, I tried the following way:</p>
<pre><code class="lang-python">index = [(0, 0, 0), (0, 1, 1)]
b = torch.Tensor([a[i[0], i[1], i[2]] for i in index_list])
# output
tensor([0., 5.])
</code></pre>
<p>However, when I check <code>b.requires_grad</code>, it becomes <code>False</code>.<br>
I want to know how to achieve the same goal without loss of “requires_grad” property.</p>
<p>Thanks ahead!</p>

username: ptrblck
cooked: <p>Rewrapping a tensor in <code>torch.Tensor</code> will detach it from the computation graph, so you might want to use <code>torch.cat</code> or <code>torch.stack</code> instead.</p>

username: yuejiang_li
cooked: <p>Yes, I tried <code>cat</code> and it works. Thanks a lot!</p>
When inference(so there are no weights update), model's half-weights are reused or calculated every forward

username: treasureqqm
cooked: <p>I use <span class="mention">@autocast</span>() to decorate my module forward functions and I wonder that, when inference(so there are no weights update), model’s half-weights are reused or calculated every forward? (i.e. will float2half for each weight be called every time?)</p>

username: ptrblck
cooked: <p>You could wrap the entire inference evaluation into <code>autocast</code>, which would then use the internal caching to avoid transforming the parameters every time.</p>

username: treasureqqm
cooked: <p>So, if i use <span class="mention">@autocast</span>() to decorate a forword function (without wraping the entire model), float2half will be called every time when calling that forward?</p>

username: ptrblck
cooked: <p>Yes, I think exiting the outermost <code>autocast</code> context would clear the cache.</p>

username: treasureqqm
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your quick reply, it helps a lot <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Pytorch inplace operations slow down the inference process on GPU

username: zihaog
cooked: <p>Hi,<br>
I’m trying to add some in-place operations inside the forward function but it seems like these operations slows down the inference process.</p>
<p>My forward function is like this:</p>
<pre><code class="lang-auto">def forward(self, x):
    x = x.permute(0, 3, 1, 2)
    x = F.interpolate(x, [224, 224], mode="bilinear")
    x[:, 0, :, :] -= 149.89  # inplace operation
    x[:, 0, :, :] /= 37.35  # inplace operation
    x[:, 1, :, :] -= 113.11 # inplace operation
    x[:, 1, :, :] /= 37.16  # inplace operation
    x[:, 2, :, :] -= 130.63  # inplace operation
    x[:, 2, :, :] /= 37.48  # inplace operation
    x = self.model(x)
    return nn.Softmax(dim=1)(x)
</code></pre>
<p>As you can see, I add some inplace operations onto BGR channel. In this case, if I do inference on a 512~batch input, it takes 0.83s on GPU.</p>
<p>However, If I get rid of these inplace operations, the code would be:</p>
<pre><code class="lang-auto">def forward(self, x):
    x = x.permute(0, 3, 1, 2)
    x = F.interpolate(x, [224, 224], mode="bilinear")
    x = self.model(x)
    return nn.Softmax(dim=1)(x)
</code></pre>
<p>And it only takes 0.23s (almost 1/4 of the former one) to do the inference on the same 512 batch.</p>
<p>Is it normal? I can’t believe the inplace operation takes so much time. Any ideas to optimize the code? Thanks a lot.</p>

username: ptrblck
cooked: <p>Could you share the code you’ve used to profile these operations, please?</p>
Optimizer python loops

username: import-antigravity
cooked: <p>I’ll admit I’m a bit confused about how the optimization code works under the hood. The code for SGD uses python for loops to iterate through each parameter:</p>
<pre><code class="lang-auto">            for p in group['params']:
                if p.grad is not None:
                    params_with_grad.append(p)
                    d_p_list.append(p.grad)

                    state = self.state[p]
                    if 'momentum_buffer' not in state:
                        momentum_buffer_list.append(None)
                    else:
                        momentum_buffer_list.append(state['momentum_buffer'])
</code></pre>
<p>Wouldn’t this be incredibly slow? Is there some sort of just in time compilation going on to speed things up? If I were to implement my own optimizer would I have access to the same performance?</p>

username: ptrblck
cooked: <p>The slow for loop approach allows for “hackable” optimizers, so that researchers can easily change the optimizer and run new experiments.<br>
<a class="mention" href="/u/crcrpar">@crcrpar</a> is working on speeding up the <code>step</code> by using the internal <code>for_each</code> methods (previously used in <code>apex</code> as <code>multi_tensor_apply</code>).</p>
The time of training in each epoch increase with no stop

username: el_youssfi_azeddine
cooked: <p>Hello, I use GPU (cuda 11.1) to train a massive IA model based on pytorch, but while training the time of every epoch is contantly increasing (starting by 20 s on going up) I do not why, can you help please?<br>
here is the training loop.<br>
thanks</p>

username: ptrblck
cooked: <p>Could you try to narrow down a minimal code snippet and post it with random input tensors so that we could try to reproduce this issue?<br>
Also, in case you are seeing an increased memory usage besides the slowdown, I guess you are storing the computation graph (accidentally) in each iteration.<br>
You can also post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>
How does reshape moves entries?

username: deepfailure
cooked: <p>Hello, I have a neural network with two input tensors:</p>
<pre><code class="lang-auto">a shape (batch_size=128, 3)
b shape (60, batch_size=128, 1)
</code></pre>
<p>I need to augment a to have the same shape as b:</p>
<p><code>a.repeat(60, 1, 1) -&gt; a shape (60, 128, 3)</code></p>
<p>Then I want to put a and b in the same neural network that accepts two dimensions. I need to reshape so that the 60 can be considered the same as the batch size:</p>
<p><code>result = neural_network(a.reshape(-1, 3), b.reshape(-1, 1))</code></p>
<p>This result now has the same <code>((60 x 128), 1)</code></p>
<p>Now I want to do the reshape so that the result has the shape (60, 128, 1) and it has the same order than a and b had initially. If I do:</p>
<p><code>result_reshaped = result.reshape(60, 128, 1)</code> I will get that, however, how can I be sure that the order is correct?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>One way to double check the reshaping operations is to initialize the tensor via <code>torch.arange</code>, reshape it, and <code>print</code> the tensor to see where the values landed. To do so I would reduce the shapes a bit, as the currently used number of elements is quite large.</p>
Add noise to layer output

username: yushu
cooked: <p>Hi, I am trying to add noise to layers’ output. I take the code from <a href="https://github.com/rosinality/stylegan2-pytorch/blob/3dee637b8937bf3830991c066ed8d9cc58afd661/model.py#L305" rel="noopener nofollow ugc">stylegan2-pytorch</a>.</p>
<p>I thought the <strong>self.weight</strong> is a learnable parameter thus it can update during training since it is nn.Parameter(), but when I track the value of <strong>grad</strong> and <strong>weight</strong> of this layer the <strong>weight</strong> remains 0 and <strong>grad</strong> is <strong>None</strong> from the beginning. Can any one give me suggestions about this? thanks.</p>
<pre><code class="lang-python">class NoiseInjection(nn.Module):
    def __init__(self):
        super().__init__()

        self.weight = nn.Parameter(torch.zeros(1), requires_grad=True)

    def forward(self, x, noise=None):
        if noise is None:
            batch, _, height, width = x.shape
            noise = x.new_empty(batch, 1, height, width).normal_()
        return x + self.weight * noise
</code></pre>

username: ptrblck
cooked: <p>I cannot reproduce the issue using your code snippet and get a valid gradient:</p>
<pre><code class="lang-python">model = NoiseInjection()
x = torch.randn(16, 3, 24, 24)
out = model(x)
out.mean().backward()
print(model.weight.grad)
&gt; tensor([0.0115])
</code></pre>

username: yushu
cooked: <p>Thanks, I double check my scripts and notice I didn’t pass the value correctly.</p>
Pytorch only works in the pytorch directory

username: miner_tom
cooked: <p>After installing pytorch from source, along with CUDA and CUDNN, I am only able to successfully import torch from the pytorch directory, so know that the installation has all of the required modules.</p>
<p>However, In any other directory, I get an error:</p>
<p>import torch<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/home/tom/.virtualenvs/vm4ml/lib/python3.7/site-packages/torch/<strong>init</strong>.py”, line 267, in <br>
_C._initExtension(manager_path())<br>
AttributeError: module ‘torch’ has no attribute ‘QUInt4x2Storage’</p>
<p>How to approach this problem?</p>

username: Panic
cooked: <p>I have same issue too.<br>
One virtual env have no problem to run torch, but another have same issue.</p>
<p>python==3.7<br>
torch==1.7.1</p>
<p>Error:</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/home/ubuntu/anaconda3/envs/koBART-trans/lib/python3.7/site-packages/torch/<strong>init</strong>.py”, line 500, in <br>
_C._initExtension(manager_path())<br>
AttributeError: module ‘torch’ has no attribute ‘QUInt4x2Storage’</p>
</blockquote>
</blockquote>
</blockquote>
Some GPU memory is still in use and GPU util is 0%

username: Ardeal
cooked: <p>Hi,</p>
<p>My example code is as follows,</p>
<p>the logs in log file includes ‘XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n’.<br>
however,  some GPU memory is still in use and GPU util is 0%.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af14380769c6720ed7305d79c94f56ea44ddebf9.png" alt="图片" data-base62-sha1="oYOSjvA7fTuTzIWuMVpSp1y5d6V" width="643" height="145"></p>
<p><strong>it seems that process doesn’t exit, or process exist but memory is not cleaned.</strong><br>
is there any solution for this problem?</p>
<p>I used the following code to start 2 processes:<br>
<code>CUDA_VISIBLE_DEVICES=6,7 WORLD_SIZE=2 python -m torch.distributed.launch --nproc_per_node=2 --master_port 44744 train.py </code></p>
<pre data-code-wrap="python"><code class="lang-python">def runner_full(ctp):
   
	logs = open('log.txt', 'a')
	train(ctp) # in train we will use GPU to train network
	torch.cuda.empty_cache()

    logs.write('XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\n')
    logs.flush()
    logs.close()
    return

if __name__ == "__main__":

    ctp = {'a':0, 'b':1, 'c':2}
    runner_full(ctp)


</code></pre>

username: ptrblck
cooked: <p>Based on the high memory usage I assume that you are either not deleting all references to CUDATensors or other processes might be using the GPU memory.<br>
PyTorch itself will create the CUDA context, which won’t be released by calling <code>torch.cuda.empty_cache()</code>, and which will use approx. 600-1000MB of device memory.</p>

username: Ardeal
cooked: <p>Hi,</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!</p>
<p>I did many experiments and found out that the process won’t exit if I enumerate dataload in my train function.</p>
<p>the interesting thing is:<br>
if I use torch.distributed.launch to launch 2 processes, the loader will result in the process won’t exit.<br>
If I run 1 process directly(python train.py), the process will exit correctly.</p>
<p>Furthermore, I confirm no process use the GPU as I am doing experiment on the machine myself.</p>
<p>I suspect there might be 2 reasons:</p>
<ol>
<li>as you said some Tensor or memory is not released. is there a way to check/confirm this?</li>
<li>generator(like yeild in dataloader) might not exit.</li>
</ol>
<p>is there any way to check/confirm above 2 issues?<br>
or do you have any idea about what I explained?</p>
Is it possible to install torch=1.4 with cuda 11?

username: Ahmed_Abdelrhman
cooked: <p>I work with robotics project on ros which needs python=2.7 which is compatible with torch=1.4 and cuda=10.  I upgraded my GPU to RTX 3080 which is compatible only with cuda 11. Would it possible to install torch=1.4 with cuda 11??? or What can I do with this problem as I tried RTX 3080 with cuda=10 and torch freezes     Thanks in advance</p>

username: ptrblck
cooked: <p>You won’t be able to build PyTorch <code>1.4.0</code> with CUDA11 without some cherry-picks, as <code>1.4.0</code> was released before CUDA11 was available.<br>
The right approach would be to drop the support for Python2 (alone for the potential security risks) and to update PyTorch to the latest release with CUDA11.<br>
In case that’s not possible, you could search all CUDA11-related commits, try to cherry-pick them into the 1.4.0 branch, and rebuild, although that sounds quite tedious.</p>
Precision first increases, then decreases

username: Caipi
cooked: <p>Hello there,</p>
<p>I have a simple multiclass deep learning model with CrossEntropyLoss. In a total of 229 classes and 8k sample, I split them into 70% training, 20% validation and 10% training. On the fly whilst training and validation I calculate the precision and recall value to calculate dice and other metrics…</p>
<p>The precision value, looks wierd tho, unless the losses looks good… Why is the precision decreasing after several epochs when network is not overfitting yet, measured by the validation loss? I calculate the precision and recall as followed:</p>
<p>import sklearn.metrics as skl<br>
precision = skl.precision_sorce(target, data, average=‘weighted’, zero_division=1)<br>
recall = skl.recall_sorce(target, data, average=‘weighted’, zero_division=1)</p>
<p>Heres the full code if anyone wants it…: <a href="https://colab.research.google.com/drive/1QD0D_bj-1wstUW3xvgr124E4DgAYl2hV?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>Below my training loss blue, validation loss red, training precision cyan and training recall purple of 25 epochs:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/815da76e69a25b9cc9b7aad1b9c210724d612ae0.png" alt="Capture" data-base62-sha1="isqfhChTZ1mK5sbPw0CBXoRwXe0" width="430" height="319"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/443e98b6bacfa516fb31a72a4f3ccea98a9af20b.png" alt="Capture2" data-base62-sha1="9JIzZgIOsmxcvqqv9MsuBZRuQ9d" width="425" height="319"></p>
How to use 8GPUs as 1

username: DidierDeschamps
cooked: <p>Hi I’m trying to do a learning a certain way and I need a big batch sufficiently big for it.<br>
Indeed I need to compute a mean vector that converges weekly in O(1/sqrt(n)) (cf. CLT).</p>
<p>I have a powerful machine to use: 8 RTXA6000</p>
<p>When I run my model with a certain batch n it will share n/8 sub-batch on each GPU and then compute 8 different vectors. I’d like to compute only one vector to optimize the weak 1/sqrt(n)-convergence I deal with is it possible to use a trick to do that?</p>

username: wanchaol
cooked: <p><a class="mention" href="/u/didierdeschamps">@DidierDeschamps</a> thanks for posting, sorry I don’t quite get your problem, is this a pytorch distributed framework related issue? or an algorithm question?</p>

username: DidierDeschamps
cooked: <p><a class="mention" href="/u/wanchaol">@wanchaol</a> Pytorch automatically splits my batch across the 8 GPUs and compute 8 Losses with 8 backpropagations. Is there a way to prevent this and compute only 1 Loss with 1 backpropagation?</p>
<p>eg. if my batch is n=80 pytorch will split n/8=10 sub batch on each GPU and compute 8 Losses, unfortunately a loss computed on 10 datapoint is less consistent than a loss computed on 80 datapoint, that’s why I’d like to avoid this to happen.</p>
Torch.jit.trace hangs

username: karimacma
cooked: <p>I’m calling torch.jit.trace() on my model and it appears to be stuck in a loop at<br>
these lines</p>
<pre><code class="lang-auto">  File "/data/scratch/karima/anaconda3/envs/mother/lib/python3.7/difflib.py", line 1032, in _fancy_helper
    yield from g
  File "/data/scratch/karima/anaconda3/envs/mother/lib/python3.7/difflib.py", line 1020, in _fancy_replace
    yield from self._fancy_helper(a, best_i+1, ahi, b, best_j+1, bhi)
  File "/data/scratch/karima/anaconda3/envs/mother/lib/python3.7/difflib.py", line 1032, in _fancy_helper
    yield from g
  File "/data/scratch/karima/anaconda3/envs/mother/lib/python3.7/difflib.py", line 1020, in _fancy_replace
    yield from self._fancy_helper(a, best_i+1, ahi, b, best_j+1, bhi)
  File "/data/scratch/karima/anaconda3/envs/mother/lib/python3.7/difflib.py", line 1032, in _fancy_helper
</code></pre>
<p>my model is implemented as a DAG of python objects that inherit from nn.Module and call the .run() functions of their child nodes which in turn call the .forward() implementations like in the example below:</p>
<pre><code class="lang-auto">class InterleavedSumOp(nn.Module):
  def __init__(self, operand, C_out):
    super(InterleavedSumOp, self).__init__()
    self.C_out = C_out
    self._operands = nn.ModuleList([operand])
    self.output = None

  def _initialize_parameters(self):
    self._operands[0]._initialize_parameters()

  def run(self, model_inputs):
    if self.output is None:
      operand = self._operands[0].run(model_inputs)
      self.output = self.forward(operand)
    return self.output 

  def forward(self, x):
    x_shape = list(x.shape)
    x_reshaped = torch.reshape(x, (x_shape[0], x_shape[1]//self.C_out, self.C_out, x_shape[2], x_shape[3]))
    out = x_reshaped.sum(1, keepdim=False) 
    return out
    
  def to_gpu(self, gpu_id):
    self._operands[0].to_gpu(gpu_id)
</code></pre>
Classification on difficult video dataset with Pytorch

username: Yelnat10
cooked: <p>Hey all, had a quick question about classifying video data in Pytorch. I have a video dataset with clips ranging from 1-10 minutes that have a simple binary label attached to them, so it’s not possible to indicate which frames show the action occurring and which frames don’t.</p>
<p>I first wanted to ask, what’s the correct way to go about loading in this video? currently I’m just using a custom dataset class and batching every 5 frames of the video, then getting the next 5 frames of the same video until that video is done, shuffling the indices determining which video to grab at the start of each epoch. I was looking at <a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - RaivoKoot/Video-Dataset-Loading-Pytorch: Generic PyTorch dataset implementation to load and augment VIDEOS during Training.</a>, but this isn’t much help as it presumes that every frame of the video is displaying an action, and each batch contains some frames of a different video.</p>
<p>Second, How should I go about classifying these videos? I can presume every 30 seconds/minute or so will include multiple examples of actions occurring. I was planning on using convolutional followed by LSTM layers, which I think might not be too good for holding events that happened a long time ago.</p>
<p>Any help at all would be appreciated!</p>