akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Setting .data of Tensor doesn't change it

username: Kamer_Ali_Yuksel
cooked: <p>The below statements doesn’t change the values in opt_weights[tr == kth].data</p>
<p>opt_weights[tr == kth].data = dist.sample_n(opt_weights[tr == kth].shape[0])<br>
opt_weights[tr == kth].data = dist.sample_n(opt_weights[tr == kth].shape[0]).data<br>
opt_weights[tr == kth].data.copy_(dist.sample_n(opt_weights[tr == kth].shape[0]))</p>
<p>I don’t get an error in above cases but as I said they’re not helpful in changing values.</p>
<p>When I try to change it without .data part, I get the following.<br>
*** RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</p>
<p>What is the right way of overwriting certain values in a tensor with samples of a distribution?</p>

username: Kamer_Ali_Yuksel
cooked: <p>I have managed to do it via index_copy_ , is this the right solution?</p>
<p>opt_weights.data.index_copy_(0, (tr == kth).nonzero().flatten(), dist.sample_n(opt_weights[tr == kth].shape[0]))</p>

username: albanD
cooked: <p>Hi,</p>
<p>In general, you should never use <code>.data</code> otherwise you might get unexpected behaviors like that.</p>
<p>The other error comes from the fact that you modify a leaf Tensor inplace. You can add a <code>opt_weights = opt_weights.clone()</code> before this code to make sure you don’t modify the original Tensor.</p>
What's the difference between `torchvision.datasets.samplers.RandomClipSampler` and `torch.utils.data.sampler.RandomSampler`

username: 111429
cooked: <p>Hi,<br>
What’s the difference?<br>
I create a dataset with torchvision.datasets.Kinetics400 class.<br>
And get different length of dataloader with these two sampler.<br>
And torch.utils.data.sampler.RandomSampler is correct to me.<br>
Thanks.</p>
Move activations to device

username: Bruce_zhuang
cooked: <p>I need to somewhat move the activations to cpu after the computation graph is generated. For instance, some computation graphs  are generated in the GPU as follows:</p>
<blockquote>
<p>input1, input2 = torch.randn(128,3,32,32)<br>
output1 = model(input)<br>
output2 = model(input)</p>
</blockquote>
<p>which would need the memory of about <code>2M</code> with 2 computational graphs generated. Now I need to place the activations associated with <code>output1</code> to <code>cpu</code>, such that the memory in GPU would fall down to about <code>M</code>. This should make sense, but I don’t know how to implement it. Thanks!</p>

username: richard
cooked: <p>You can move <code>output1</code> to cpu by doing <code>output1.cpu()</code>. Or are you asking how to move activations that were saved by the PyTorch autograd engine to CPU to avoid OOM-ing?</p>

username: Bruce_zhuang
cooked: <p>Yes. I am trying to move all the activations linked with <code>output1</code>, not just the tensor itself.</p>

username: richard
cooked: <p>I don’t think there’s an easy way to do that in PyTorch today. One way you could save memory is to to use <a href="https://pytorch.org/docs/stable/checkpoint.html"><code>torch.utils.checkpoint</code></a>, which re-computes the intermediate activations.</p>
<p>I think it would make sense for there to be a checkpointing option in PyTorch that saves activations on CPU to avoid using too much GPU memory. Please feel free to open us a feature request at <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox">Issues · pytorch/pytorch · GitHub</a>.</p>
<p>If you do want to go down the route of manually implementing this by yourself, one (very tedious) way is to</p>
<ul>
<li>For each operator you’re using, write a custom <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function">autograd.Function</a>
</li>
<li>the custom autograd.Function has a forward pass. You’d have to manually write a backward pass for it, but that can be doable by copying and pasting the relevant code from <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml" class="inline-onebox">pytorch/derivatives.yaml at master · pytorch/pytorch · GitHub</a>
</li>
<li>When saving tensors for the backward pass, convert them to CPU before saving.</li>
</ul>
Visually aligning 2 sets of data

username: ds2268
cooked: <p>Let’s say that I have 2 sets of data (images). Domain A and domain B. I randomly sample a batch_size of samples from domain A. I want to find batch_size of samples in domain B, which are the most similar (by some metric) to batch_size samples in domain A. The most similar samples in domain B are sampled from a variable-sized random subset (no to do an extensive search of the whole sample space).</p>
<p>Basically, if I sample 16 images in domain A, I want to find the most similar example for each sample in domain B (from a random subset e.g. 100 images). 16 images from A and 16 (closest pair-wise) then present input to the model.</p>
<p>What would be the cleanest approach to implement such a pipeline in PyTorch (using Datasets, DataLoaders, Samplers, etc.)</p>
Finding the exact moment of firearm shot

username: talgold9
cooked: <p>I have a video of a man shots with a firearm. I want to detect the exact moments (in seconds) that the man shots using the video, and of course the audio. An important thing to know is that this man is not alone. There are more people that also shot with firearms, but the camera records him (maybe some other people are also recorded but the camera points on him).<br>
Does anyone have any idea how can I perform this task?</p>
<p>Best regards!</p>
Stacking LSTM hidden layers vs using the last one for language models

username: sigma_x
cooked: <p>In several tutorials on n-gram language models (e.g. <a href="https://medium.com/analytics-vidhya/ai-writing-poems-building-lstm-model-using-pytorch-d1c58a24bb64" rel="noopener nofollow ugc">this</a>) the output of LSTM is stacked:</p>
<pre><code class="lang-auto">out ,_=lstm(input)
out = out.contiguous().view(-1, self.n_hidden)

</code></pre>
<p>which is dimensionality (<code>batch_size*seq_length, hidden_features</code>) and becomes the input in the next layer. Target is also resized to <code>batch_size*seq_length</code>.</p>
<p>I don’t understand this approach as it’s the job of LSTM to learn the relationship among inputs in the sequence. The correct output should be</p>
<pre><code class="lang-auto">out,_ = lstm(input)
out = out[:,-1,:]
</code></pre>
<p>the last state for each batch, and become the  input in the next layer, and the target is size <code>batch_size</code>.</p>
<p>So why use the full sequence then?</p>

username: vdw
cooked: <p>The problem is that the article is not very well document, so it’s not easy to see the important details without knowing what to look for. Let me try to make the comparison between your approach and the one in the article.</p>
<p>Let’s assume you have a long text based on which you want to build your language model, e.g.:<br>
A B C D E F G H I H K L M N O …</p>
<p>Your approach is not wrong</p>
<pre><code class="lang-auto">out,_ = lstm(input)
out = out[:,-1,:]
</code></pre>
<p>but only assumes the training data of the form (input sequence, next word). So you training data might look like:<br>
A, B<br>
A B, C<br>
A B C, D<br>
B C D, E<br>
C D E, F<br>
…<br>
(assuming a max. length of 3 for the input sequences)</p>
<p>The method in the article does something different. The important details is <code># The targets, shifted by one</code>. So here the dataset looks like:<br>
A B C, B C D<br>
B C D, C D E<br>
C D E, D E F<br>
…</p>
<p>If you take all outputs of the LSTM – i.e., not just after the last step – you implicitly get (A, B) after the 1st step, (A B, C) after the 2nd step and (A B C, D) after the 3rd/last step for the first data item (A B C, B C D).</p>
<p>There’s no fundamental difference between both approaches, but the 2nd is more efficient.</p>

username: sigma_x
cooked: <p>Thanks, it makes sense. So at each stage LSTM encodes implicitly the sequence so far, e.g. A, AB, ABC, etc. So If I had a batch of 2: (D|A, B,C) and (E|B, C,D), I would have these sequences for a fourgram:</p>
<p>Batch1:<br>
A-&gt;A<br>
B-&gt;AB<br>
C-&gt;ABC</p>
<p>batch 2:<br>
B-&gt;B<br>
C-&gt;BC<br>
D-&gt;BCD</p>
<p>So If I were to build a language model, I’d need a total of 2x4xvocabulary size labels: (A, B, C, D) and (B, C, D, E) rather than (D) and (E).</p>
Arbitrary pairwise operation

username: eric1
cooked: <p>Hello! I am attempting to compute a dense N x N matrix by evaluating a function f(x_i, x_j) on the N^2 elements in X (X.shape = (N, F)). An example would be calculating an euclidean distance matrix (if f was euclidean distance) but in general I would like for f to be a more complicated function. However, I’m really struggling to find a way to do this without blowing up my memory in pytorch. For my application, N is ~2000 and F is ~128. My first (naive) way is to generate two index vectors, idx_r and idx_c,</p>
<pre><code class="lang-auto">idx_r = [0, 0, 0, ...., 1, 1, 1, ....N-1,N-1,N-1]
idx_c = [0, 1, 2, 3,... 0,1, 2, 3, ...., N-3, N-2, N-1]
</code></pre>
<p>and then compute:</p>
<pre><code class="lang-auto">X_r = X[idx_r]
X_c = X[idx_c]  
y = f(X_r, X_c)
</code></pre>
<p>but the explicit instantiation of X_r and X_c easily consumes all of my GPU memory. Are there any recommended pytorch tricks that will let me compute the elements of X_r and X_c on the fly? or any other suggested ways of dealing with this?</p>
<p>Thanks!</p>
Slicing 3d tensor with 2d tensors

username: nghind
cooked: <p>Hi,<br>
I have 3d tensor like this:</p>
<blockquote>
<p>tensor([[[0.6066, 0.0390, 0.0141, 0.3977, 0.2390, 0.1982, 0.8321],<br>
[0.5124, 0.5975, 0.6119, 0.0933, 0.5048, 0.6961, 0.4293],<br>
[0.0222, 0.7371, 0.0867, 0.0660, 0.5737, 0.4022, 0.9751],<br>
[0.6079, 0.9880, 0.5903, 0.8366, 0.9752, 0.1610, 0.8460],<br>
[0.3080, 0.8679, 0.4383, 0.2642, 0.7401, 0.6404, 0.0325]],<br>
[[0.9429, 0.7948, 0.9669, 0.7277, 0.9147, 0.9281, 0.4738],<br>
[0.0109, 0.7387, 0.5509, 0.5736, 0.3486, 0.9945, 0.6516],<br>
[0.0915, 0.0928, 0.2610, 0.0317, 0.3806, 0.2865, 0.5693],<br>
[0.1901, 0.1824, 0.1148, 0.0235, 0.1220, 0.3003, 0.7579],<br>
[0.8647, 0.9720, 0.6833, 0.6454, 0.0365, 0.6508, 0.5220]]])</p>
</blockquote>
<p>and 2d tensor like this:</p>
<blockquote>
<p>tensor([[1, 3], [2, 3]])</p>
</blockquote>
<p>I want to get tensor in range (1, 2, 3), (2, 3) based on index tensor to get output:</p>
<blockquote>
<p>tensor([[[0.5124, 0.5975, 0.6119, 0.0933, 0.5048, 0.6961, 0.4293],<br>
[0.0222, 0.7371, 0.0867, 0.0660, 0.5737, 0.4022, 0.9751],<br>
[0.6079, 0.9880, 0.5903, 0.8366, 0.9752, 0.1610, 0.8460]],<br>
[[0.0915, 0.0928, 0.2610, 0.0317, 0.3806, 0.2865, 0.5693],<br>
[0.1901, 0.1824, 0.1148, 0.0235, 0.1220, 0.3003, 0.7579]]])</p>
</blockquote>
<p>then average tensor by batch.</p>
<p>Any idea or help on how to solve this is well appreciated!<br>
Thank you in advance.</p>

username: ptrblck
cooked: <p>I don’t quite understand the second tensor shape and how it corresponds to the index tensor.<br>
However, since the index tensor is using a different number of indices, you would need to either transform the indices to a flattened version (and the same with the original input) or index them separately.</p>

username: caonv
cooked: <p>Do you mean that you want to use the second tensor for indexing the first tensor? Did you try <code>torch.gather</code>?</p>
Creating a tensor from other tensors for training

username: yuanqi_chau
cooked: <p>hey there, I have a model now requiring me to find the average of item embeddings of the user as the user interest at the particular time.</p>
<p>The problem is that the item interacted at different times are different, ie 5 for this month and 10 for the other. Therefore, I couldn’t just pack them in one tensor and make a torch.mean computation.</p>
<p>Here are my codes to tackle this problem by concatenating the mean row by row and creating a new tensor.</p>
<pre><code class="lang-auto">items_emb = nn.Embedding(6000, 10)

items = [[3245, 3976, 4136, 2757, 2524],
 [5415, 2324, 3246, 1418, 298, 1052],
 [903, 3428, 106, 1130, 5717, 745, 5576, 4757, 4867, 2897],
 [5387, 1911, 5866, 824, 2462]]

def calc_mean(inputs):
    out = torch.mean(inputs, 0)
    return out

def get_mean_tensor(items_emb, items):
    first = items_emb(torch.LongTensor(items[0]))
    second = items_emb(torch.LongTensor(items[1]))
    embs = torch.cat([calc_mean(first), calc_mean(second)], 0)

    for i in range(2, len(items)):
        cur = items_emb(torch.LongTensor(items[i]))
        embs = torch.cat([embs, calc_mean(cur)], 0)
        
    embs = embs.unsqueeze(0).view(-1, 10)
    return embs

print(get_mean_tensor(items_emb, items))
</code></pre>
<p>The output would be a tensor of len(items) x embedding dimension.</p>
<pre><code class="lang-auto">tensor([[ 0.0158, -0.2520,  0.2216, -0.2114, -0.6670,  0.4971,  0.5369,  0.7796,
          0.4892, -0.3147],
        [ 0.0904,  0.1451,  0.2905, -0.5891, -0.1611, -0.1272, -0.1093, -0.1912,
         -0.1019,  0.0333],
        [ 0.2133, -0.4288, -0.5960, -0.1288, -0.0509, -0.1023, -0.0398, -0.2128,
         -0.2405,  0.0800],
        [ 0.1568, -0.3687,  0.0273,  0.2744, -0.7340, -0.0794,  0.2946, -0.7839,
         -0.0467, -0.0430]], grad_fn=&lt;ViewBackward&gt;)
</code></pre>
<p>Tell me if there’s a better way to do this, thanks!</p>
Share GPU model between threads in C++

username: Ognjen_Kocic
cooked: <p>I have an EncoderDecoder architecture which uses custom RNN in the decoder, and consequently GPU is not fully utilized during decode phase when the single instance of model is used (target platform is V100). The model is exported to TorchScript to optimize Python code and custom RNN.</p>
<p>Currently, my workaround was to spawn multiple workers which will handle requests concurrently and hence utilize GPU better (this was also confirmed when measuring real-world data throughput). The problem is each worker needs to host not only this, but multiple models and each of them has its own copy of the model. This kind of setup sometimes causes memory problems as I’m keeping several copies of each model in GPU memory (every worker has a copy of each model).</p>
<p>Questions:</p>
<ol>
<li>Since Python doesn’t have a real notion of threads, I was wondering is it possible to load GPU model(s) in C++ in main thread, spawn several worker threads and share model between those threads so each thread handles its own workload independently?</li>
<li>If answer to 1. is positive, are there any blockers for performance and utilization improvement (e.g. kernels from same CUDA context cannot overlap etc.)?</li>
</ol>
<p>EDIT: I should have added more context on why GPU is not fully utilized in inference, so I’m updating the post. The problem is not about validation inference where I can control the batch size and fully utilize GPU(s). This is all about real-world scenario where I’m processing a request and decoding single sequence at the time, maybe a few more if dynamic batching is successful, but the problem being solved by the network is not easy to batch (both input&amp;output have dynamic axes).</p>
How to merge two torch.utils.data dataloaders with a single operation

username: Stefano_Barone
cooked: <p>I have two dataloaders and I would like to merge them without redefining the datasets, in my case train_dataset and val_dataset.</p>
<pre><code class="lang-auto">train_loader = DataLoader(train_dataset, batch_size = 512, drop_last=True,shuffle=True)
val_loader = DataLoader(val_dataset, batch_size = 512, drop_last=False)
</code></pre>
<p>Wanted result:</p>
<pre><code class="lang-auto">train_loader = train_loader + val_loader</code></pre>

username: caonv
cooked: <p>No, there is no simple way to do that. You can start sampling from <code>val_loader</code> after getting all samples from <code>train_loader</code></p>

username: 1chimaruGin
cooked: <aside class="quote quote-modified" data-post="2" data-topic="649">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/train-simultaneously-on-two-datasets/649/2">Train simultaneously on two datasets</a> 
  </div>
  <blockquote>
    I’d recommend creating a new dataset and concatenating the images there, so the copy will be done inside the worker processes: 
class ConcatDataset(torch.utils.data.Dataset):
    def __init__(self, *datasets):
        self.datasets = datasets

    def __getitem__(self, i):
        return tuple(d[i] for d in self.datasets)

    def __len__(self):
        return min(len(d) for d in self.datasets)

train_loader = torch.utils.data.DataLoader(
             ConcatDataset(
                 datasets.Ima…
  </blockquote>
</aside>
nn.Module add new parameter, setattr() VS register_parameter()

username: Jiajun_Zha
cooked: <p>When I want to add a new parameter to an nn.Module, I basically see 2 approaches.</p>
<p>First is just to use the torch builtin register_parameter() function, and the added Tensor will show up in the Module’s state_dict.</p>
<p>Second is to use the python function setattr(), but the added Tensor will not show up in the Module’s state_dict. But what’s really amazing about this is that, this hidden Tensor can actually be accessed from the Module’s state_dict. I’ll give my example below.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; conv = torch.nn.Conv2d(512, 512, kernel_size = 1, padding=0)
&gt;&gt;&gt; testTensor = torch.randn(512)
&gt;&gt;&gt; setattr(conv, 'testTensor', testTensor)
&gt;&gt;&gt; conv._parameters.keys()
odict_keys(['weight', 'bias'])
&gt;&gt;&gt; conv._parameters[testTensor]
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
KeyError: tensor([ 1.6598e-01, -2.5838e+00,  8.0253e-01, -7.5928e-01,  1.3394e-01,
        ......
</code></pre>
<p>So you can see, despite the error message, the last command actually gives out the correct testTensor value. And I tried to add this testTensor into the computation graph (just by adding it to the bias), the backpropagation works just as good as usual.  So the question is, what really happens when using setattr() to add another tensor into nn.Module? And is it really equal to using register_parameter?</p>

username: pascal_notsawo
cooked: <p><code>setattr</code> just adds an attribute to a python class, and that’s all: you can use <code>getattr</code> to get this attribute. But it is not sufficient to be an attribute of a pytorch model to be a parameter of this model, no.</p>
<p>To be a parameter of a pytorch model, it is necessary to be an instance of the class <a href="https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html#torch.nn.parameter.Parameter" rel="noopener nofollow ugc"><code>torch.nn.parameter.Parameter</code></a> and to be in the dictionary (<a href="https://pytorch.org/cppdocs/api/classtorch_1_1_ordered_dict_1_1_item.html#class-documentation" rel="noopener nofollow ugc"><code>OrderedDict</code></a>)  <code>_parameters</code> of this model (which is one of its attributes). This is what <code>register_parameter(name: str, param: Optional[Parameter])</code> does : <code>self._parameters[name] = param</code> (after several checks) : see the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module.register_parameter" rel="noopener nofollow ugc">source code</a></p>
What is the difference between nn.Dropout2d and nn.Dropout?

username: PytorchLearning
cooked: <p>Hi all,</p>
<p>Can someone please explain to me the difference between nn.Dropout2d and nn.Dropout</p>
<p>Thanks for your help</p>

username: 1chimaruGin
cooked: <p><code>nn.Dropout</code> - Input can be of any shape.</p>
<p><code>nn.Dropout2d</code> - Input (N, C, H, W).</p>
<p>Usually the input comes from <code>nn.Conv2d</code> modules.<br>
If adjacent pixels within feature maps are strongly correlated (as is normally the case in early convolution layers) then i.i.d. dropout will not regularize the activations and will otherwise just result in an effective learning rate decrease.</p>
<p>In this case, <code>nn.Dropout2d()</code> will help promote independence between feature maps and should be used instead.</p>

username: PytorchLearning
cooked: <p>Thanks a lot <a class="mention" href="/u/1chimarugin">@1chimaruGin</a>  for this explanation. Now I understand better.</p>
How to use pretrained model for prediction?

username: Adamya_Shyam
cooked: <p>Hello there,<br>
I have a pretrained model for Image Colorization using captions. The model was trained using the image, a caption and the features extracted using ResNet101. I have that model saved as .pth.tar file. I want to predict the output for an image, how can I load that model and use it for prediction?<br>
Please help.</p>

username: CedricLy
cooked: <p>Hi,</p>
<p>the common way would be this.</p>
<pre><code class="lang-auto">model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()
</code></pre>
<p>here you can read more details:<br>
<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 1.7.1 documentation</a></p>
Using forward hook for data parallel with multiple GPUs

username: bellos1203
cooked: <p>Hi, I’m trying to extract intermediate features from general ResNet.<br>
I’m able to extract intermediate feature using the following code snippet with a <strong>single gpu</strong>.<br>
The following code snippet is part of my own network module class, which uses ResNet as <code>self.base_model</code>.</p>
<pre><code class="lang-auto">self.hooks = [None]
def set_hooks(self):

    self._activations = [None]

    def forward_hook(module, input, output):
       self.activations[0] = output
       return None

    self._hooks[0] = self.base_model.layer4.register_forward_hook(forward_hook)
</code></pre>
<p>After a single forward step, I call <code>model.activations[0]</code> to extract the intermediate features.</p>
<p>The <strong>problem</strong> is, when I modify my code to use <strong>nn.DataParallel</strong> to use <strong>multiple gpus</strong>, <code>model.module.activations[0]</code> only returns the results of the first gpu (I’m not sure whether it is the result from the first gpu or other gpus).</p>
<p>How can I get the results of the forward hook from all of the gpus?<br>
One possible option is to change the code by appending the result of the forward hooks to a list first and then getting the results by indexing the list, but I’m not sure whether the order (or rank) of the gpus is guaranteed at every iteration.</p>
<p>I’d greatly appreciate any help you can offer.<br>
Thank you.</p>

username: bellos1203
cooked: <p>For those who still need help, I solved the problem by applying the solution from <a href="https://discuss.pytorch.org/t/aggregating-the-results-of-forward-backward-hook-on-nn-dataparallel-multi-gpu/28981">https://discuss.pytorch.org/t/aggregating-the-results-of-forward-backward-hook-on-nn-dataparallel-multi-gpu/28981/10</a>.</p>
Should I do the following if I implement quantization aware training?

username: feiyuhuahuo
cooked: <p>I read this from the PyToch docs.<br>
<code>For static quantization techniques which quantize activations, the user needs to do the following in addition:</code></p>
<pre><code>Specify where activations are quantized and de-quantized. This is done using QuantStub and DeQuantStub modules.

Use torch.nn.quantized.FloatFunctional to wrap tensor operations that require special handling for quantization into modules. Examples are operations like add and cat which require special handling to determine output quantization parameters.

Fuse modules: combine operations/modules into a single module to obtain higher accuracy and performance. This is done using the torch.quantization.fuse_modules() API, which takes in lists of modules to be fused. We currently support the following fusions: [Conv, Relu], [Conv, BatchNorm], [Conv, BatchNorm, Relu], [Linear, Relu]
</code></pre>
<p>`<br>
Seems that these three operations are not needed for quantization aware training. But from the API example and another tutorial(<a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#post-training-static-quantization" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.7.1 documentation</a>), these three operations are also inclueded. So should they be implemented?</p>

username: Vasiliy_Kuznetsov
cooked: <p>Yes, all of these are used for QAT in Eager mode.  Fusion is optional but recommended for higher performance and accuracy.  Is there any information / doc stating otherwise?</p>

username: feiyuhuahuo
cooked: <p>Hi, <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a>, since <code>add</code> and <code>cat</code> are common operations in CNN models, so every <code>add</code> or <code>cat</code> needs to be wrapped by  <code>torch.nn.quantized.FloatFunctional</code>? And is there any other operation needs to do so? Only <code>add</code> and <code>cat</code>?</p>

username: Vasiliy_Kuznetsov
cooked: <p>In Eager mode - yes.  There is a full list of ops supported by FloatFunctional here: <a href="https://pytorch.org/docs/stable/torch.nn.quantized.html#torch.nn.quantized.FloatFunctional" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.quantized — PyTorch 1.7.0 documentation</a>.</p>
Loss of Neural network is constant

username: Muhammad_Ahmed
cooked: <p>I’m trying to apply the facenet paper myself.<br>
I’ve stumbled upon a problem, My loss is always constant.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e8351f9843ece0ac0447b4c24d9404809030c51f.png" alt="image" data-base62-sha1="x8cAyc09s6whTeQ0IaxB6hilsdN" width="666" height="425"><br>
This is my code for training my NN<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/95bb2596a772f1321513292463be53605bde82f7.png" alt="image" data-base62-sha1="lmA6pcGLeS0PEEVRUXX6g21DIQn" width="559" height="292"><br>
This is my forward and loss function<br>
For some reason the loss is always coming out to be constant and the model isn’t learning.<br>
This can be seen in the image given below that the loss is constant in every epoch.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbcf6b0588a521c5349e3672c62a0e8f7cf1b738.png" alt="image" data-base62-sha1="t4ZjGYgxciKWnSfNXCsk7rUtet2" width="580" height="141"><br>
Please if someone knows the solution, do tell me as I require to complete the project in a very small time.</p>

username: Muhammad_Ahmed
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sir, if you can help me with this, I’ll be very grateful to you.<br>
Thanks.</p>

username: Chirath_Pansilu
cooked: <p>Try adding <code>optimizer.zero_grad()</code> in the batch loop</p>
<p>In your code your’e accumulating gradients in the batch loop . So you need to clear gradients in the batch loop rather than in the epoch loop</p>

username: Muhammad_Ahmed
cooked: <p>Thank you <a class="mention" href="/u/chirath_pansilu">@Chirath_Pansilu</a>.<br>
It worked.</p>

username: outInsideOut
cooked: <p>Hey there, where did you place the <code>optimizer.zero_grad()</code> line in the batch loop? I am having the same issue, however I have the zero_grad() function being called.</p>
Best Way to Calculate/Approximate The Diversity of a Class-Probability Population for Novelty Search

username: Kamer_Ali_Yuksel
cooked: <p>I am performing Novelty Search; hence, I need to calculate the diversity of a population (vectors) in a batch. The diversity is generally measured by the average k-Nearest distance of the population in the literature. What would be the best and fastest way to calculate that in PyTorch? Is there a better and more efficient way to approximate the diversity, preferably without the k parameter? Also, in my case, the population consists of class probabilities. Thus, I believe that KL or JS divergence may be preferable over other type of distances when calculating the diversity. I would be more than glad if you can suggest methods or implementations accordingly. It would be grad having a normalized measure of diversity across the population of class probabilities. I guess that an internal step could be calculating a similarity matrix first using one of the divergence methods. Then, calculating the average of minimum pair-wise divergence within population could be meaningful. Thank you very much in advance for your help.</p>
<p>Edit: Currently, I am calculating the mean pairwise JS divergence as follows; using topk to get for kNN would also be possible but I am not sure what advantage that would have for the novelty search.<br>
torch.mean(weights[:,None,:] * (weights[:,None,:]/weights).log())</p>
Loaded model is not the same

username: CedricLy
cooked: <p>Hi everyone,</p>
<p>I have a problem wih loading my model to continue the training.<br>
I have a minimalistic example:</p>
<pre><code class="lang-auto">for epoch in range(...):
    train model
for k, data in enumerate(train_loader):
    preparing inputs
    output = model(input1.float(), input2.float())
    torch.save(model,'output/model_saved')
    break
model2 = torch.load('output/model_saved')
print('Get the parameters states values')
for key in model2.state_dict().keys():
    print(torch.equal(model.state_dict()[key],model2.state_dict()[key]))

print('Compare outputs')
print(torch.equal(output,model2(input1.float(), input2.float())))
</code></pre>
<p>The loaded state are the same.<br>
The comparison of the outputs are different.<br>
How can the outputs between model and model2 be  different if the model parameters and the inputs are the same?</p>

username: CedricLy
cooked: <p>Ok, the problem seems to be known already.<br>
By saving the model it saved it in eval() mode.<br>
The outputs will be the same for the eval() mode.</p>
<p>Unfortunately I calculate the loss in model.train().<br>
So my new question would be, if there is a way to continue training, without having a “kink” in the loss values?</p>

username: ptrblck
cooked: <p>The different outputs in training mode could come from the usage of e.g. dropout layers or any other layers, which change their behavior between training and evaluation.</p>

username: CedricLy
cooked: <p>Yes, this is indeed the issue.<br>
One style question:<br>
Is it something like a unsoken rule, to calculate per epoch the loss in model.eval(),<br>
in order to have a plot wihtout this behavior?</p>

username: ptrblck
cooked: <p>You would usually use <code>model.eval()</code> to calculate the epoch loss using the validation set, as this would also be the workflow for the model deployment. Using the model in training mode during validation and test would leak the data information e.g. into the batchnorm statistics, and could create a biased loss estimation for “real unseen” data. Also, it’s usually not desired to apply dropout during validation/testing/deployment.</p>
An if statement causes a catastrophic GPU memory leak

username: Michael_Moran
cooked: <p>In my model I apply self attention and have 3 different pooling strategies of using the cls token, max pooling, and a combination of max pooling and the cls token.</p>
<p>What mode the module is in is defined as a string kwarg and these are the effects during init</p>
<blockquote>
<pre><code>    if pool_strat == "both":
        post_dim = pre_pool_layers[-1] + attention_dim_per_head * attention_heads

    if pool_strat == "max":
        post_dim = pre_pool_layers[-1]

    if pool_strat == "cls":
        post_dim = attention_dim_per_head * attention_heads
</code></pre>
</blockquote>
<p>All this does is tell the first layer after pooling what the dim will be. In forward this is handled with</p>
<blockquote>
<pre><code>    if self.pool_strat == "both":
        # Concat CLS to max pool
        global_embedding = torch.cat([CLS, crystal_x], dim=1)
    if self.pool_strat == "max":
        # Set x to just crystal x
        global_embedding = crystal_x
    if self.pool_strat == "cls":
        # Set x to just cls
        global_embedding = CLS
</code></pre>
</blockquote>
<p>While my model works fine for “both” and “max” the use of “cls” causes a memory leak. crystal x and CLS are defined as</p>
<p>crystal_x = torch.max(x[1:],0,keepdim=False)[0]<br>
CLS = x[0]</p>
<p>I am extremely confused as to why these nearly identical options could cause a memory leak like this, more so confusing that concating CLS to global embedding works but directly pushing it forward does not. Is there something wrong with indexing x in the way I have? The memory leak grows on the end of every epoch, and it increases by memory usage at epoch 0 each time additively.</p>
How to suppress " SourceChangeWarning"?

username: josmi9966
cooked: <p>I am using pytorch to deploy models in a way where processes communicate over stdout/stderr pipes. I therefore want to suppress all messages coming from the pytorch backend library which are not important.</p>
<p>I am getting messages like “SourceChangeWarning: source code of class ‘mymoduleclasshere’ has changed. you can retrieve the original source code by accessing the object’s source attribute or set <code>torch.nn.Module.dump_patches = True</code> and use the patch tool to revert the changes.”</p>
<p>I know the source code has changed and I am fine with that, how can I suppress that message?</p>

username: ntomita
cooked: <p>The warning is implemented using python’s standard warnings library, thus<br>
<code>import warnings warnings.filterwarnings("ignore")</code><br>
should do the trick.</p>

username: Guru_Swaroop
cooked: <blockquote>
<p>import warnings<br>
warnings.filterwarnings(“ignore”)</p>
</blockquote>
<p>This is a great solution and suppresses all the warnings. However, is it possible to suppress only a specific Warning from PyTorch?</p>

username: ntomita
cooked: <p>Of course. You can do like this:</p>
<pre><code class="lang-auto">import warnings
from torch.serialization import SourceChangeWarning
warnings.filterwarnings("ignore", category=SourceChangeWarning)
</code></pre>
<p>As the example suggested, you need to identify a specific warning you want to suppress, which is usually indicated in original warning messages.</p>
Investigate other backend than cpu and cuda

username: czotti
cooked: <p>I’m currently trying to use different processing “backend” than the usual cpu and cuda.</p>
<p>This is my current pytorch configuration.</p>
<pre><code class="lang-auto">print(*torch.__config__.show().split("\n"), sep="\n")
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
</code></pre>
<p>I tried the <code>torch.utils.mkldnn.to_mkldnn</code> on my model and several functions are not present like <code>aten::cat</code>, <code>aten::leaky_relu</code> and it return an error message.</p>
<p>When I check the <code>torch.to</code> function I see several other types <code>cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu, xla</code> is their any way to use them ? and convert model to use these type ? I failed to find any tutorial or anything on this topic.</p>
<p>Thanks.</p>
Speed of cpu inference changes with parameter values

username: shomerj
cooked: <p>I have two models with the exact same architecture but different parameters. The parameters are stores as float32. When I time inference on each of these models I get roughly a 10x difference in speed. Could the actual values of the parameters affect inference time on a CPU?</p>
<p>Note: speeds are identical on GPU</p>

username: albanD
cooked: <p>Hi,</p>
<p>The only case that I know of is for pathologically small float numbers.<br>
You can try to do <code>torch.set_flush_denormal(True)</code> at the beginning of your script to see if it fixes it (doc here: <a href="https://pytorch.org/docs/stable/generated/torch.set_flush_denormal.html?highlight=denormal#torch.set_flush_denormal" class="inline-onebox">torch.set_flush_denormal — PyTorch 1.7.0 documentation</a>).</p>

username: shomerj
cooked: <p>Thanks for the response. That did not solve anything. I am quite perplexed by this.</p>

username: albanD
cooked: <p>Do you have a small code sample that reproduces this that we could run on colab by any chance?</p>

username: googlebot
cooked: <p>other possibilities:</p>
<ol>
<li>some indexing is done differently, one model has a lot more cache misses</li>
<li>rejection [random] sampling</li>
<li>some other operations with loops affected by values (some linear algebra routines, I think)</li>
<li>order of measurements matters, as memory allocations are resolved differently (if you re-use the same process)</li>
</ol>

username: shomerj
cooked: <p>What does it mean to have a lot more cache misses?</p>

username: googlebot
cooked: <p>I meant L2/L3 system ram cache</p>

username: shomerj
cooked: <p>I could not replicate it on google colab. I was able to get the speeds of both models to run the same. I think it was some type of caching issue. I have no real idea where to start to actually debug it.</p>
RuntimeError: 0 <= device && device < device_num INTERNAL ASSERT FAILED at /pytorch/c10/cuda/CUDACachingAllocator.cpp:879, please report a bug to PyTorch. Invalid device argument.

username: Shir_mani
cooked: <ul>
<li>
<p>The computer was able to run the code on the GPU</p>
</li>
<li>
<p>I did a computer restart</p>
</li>
<li>
<p>The computer cannot run the code and is recording the following error</p>
</li>
</ul>
<hr>
<p>RuntimeError Traceback (most recent call last)<br>
 in <br>
6 torch.cuda.empty_cache ()<br>
7<br>
----&gt; 8 print (torch.cuda.memory_allocated (0))</p>
<p>~ / anaconda3 / lib / python3.7 / site-packages / torch / cuda / memory.py in memory_allocated (device)<br>
276 details about GPU memory management.<br>
277 “” "<br>
-&gt; 278 return memory_stats (device = device) [“allocated_bytes.all.current”]<br>
279<br>
280</p>
<p>~ / anaconda3 / lib / python3.7 / site-packages / torch / cuda / memory.py in memory_stats (device)<br>
157 result.append ((prefix, obj))<br>
158<br>
-&gt; 159 stats = memory_stats_as_nested_dict (device = device)<br>
160 _recurse_add_to_result ("", stats)<br>
161 result.sort ()</p>
<p>~ / anaconda3 / lib / python3.7 / site-packages / torch / cuda / memory.py in memory_stats_as_nested_dict (device)<br>
167 "Returns the result of: func:<code> ~ torch.cuda.memory_stats</code> as a nested dictionary. “” "<br>
168 device = _get_device_index (device, optional = True)<br>
-&gt; 169 return torch._C._cuda_memoryStats (device)<br>
170<br>
171</p>
<p>RuntimeError: 0 &lt;= device &amp;&amp; device &lt;device_num INTERNAL ASSERT FAILED at /pytorch/c10/cuda/CUDACachingAllocator.cpp:879, please report a bug to PyTorch. Invalid device argument.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Could you give more context about what you where doing when it happened?<br>
Also could you run your code again with <code>TORCH_SHOW_CPP_STACKTRACES=1</code> env variable to get more informations where the error comes from?</p>
Can you do an optimization step by just setting param.grad?

username: Sam_Lerman
cooked: <p>For example, without a backwards pass:</p>
<pre><code class="lang-auto">for param in model.parameters():
       param.grad = 3.1415
optimization.step()
</code></pre>
<p>What about doing a backwards pass but then modifying the grads:</p>
<pre><code class="lang-auto">loss.backward()
for param in model.parameters():
       param.grad += 3.1415
optimization.step()
</code></pre>

username: eric.mitchell
cooked: <p>Yes, this works. By default, <code>autograd.backward()</code> computes the gradients without storing the computation graph for the gradients (in case you wanted to e.g. do a double backward). So making non-differentiable modifications to the gradients, if you aren’t planning on doing something like MAML or gradient penalty, should be fine.</p>
<p>If you look at the <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/sgd.py#L99" rel="noopener nofollow ugc">implementation</a> of e.g. <code>torch.optim.SGD</code>, you see it just makes an in-place update to the model parameter <code>p</code> with whatever is in <code>p.grad</code>.</p>
Semantic Segmentation weight initialization

username: Eyal
cooked: <p>Hello,</p>
<p>I’m working on a semantic segmentation task in point clouds, I have 9 classes, labeled from 0 to 8. when 0 is unlabeled points.</p>
<p>The classes are widely varied with the number of points, vegetation and ground are around 70% of the point cloud, and fence and light poles account for very few points. I understand that this affects the training process, so we should initialize the weights with respect to the proportion of the classes.<br>
I’d like to have more explanation regarding this topic.<br>
Thanks very much.</p>

username: Eyal
cooked: <p>Still have gotten no answer, I suppose it causes the IoU metric to go down while training.</p>

username: Usama_Hasan
cooked: <aside class="quote no-group" data-username="Eyal" data-post="1" data-topic="107989">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eyal/48/31008_2.png" class="avatar"> Eyal:</div>
<blockquote>
<p>I understand that this affects the training process, so we should initialize the weights with respect to the proportion of the classes.</p>
</blockquote>
</aside>
<p>I don’t think so this  a good idea, a better idea would be ground plane augmentation.</p>

username: Eyal
cooked: <p>What do you exactly mean by ground plane augmentation? never heard of it. Thanks!</p>
Torch.qr and torch.ones being created inside cpu instead of cuda?

username: lucastononrodrigues
cooked: <p>I was implementing a torch.qr inside a module of my creation with something like this:</p>
<p>self.q_transposed=torch.qr(nn.Parameter(torch.randn(self.m,self.n),requires_grad=False).T)[0].T</p>
<p>Even though the module device is cuda (I made sure of that), the parameter is in cpu for some reason.</p>
<p>The same thing is happening with torch.ones(): inside my forward method, I need to declare a torch.ones() tensor and then multiply it by another tensor (that is located in gpu) and, for some reason, this “ones” tensor goes directly to cpu.</p>
<p>The only solution I have found so far was to insert a device argument in the <strong>init</strong> and then add each parameter back to cuda when my matmul would not work. But I am afraid that this would compromise using resources such as nn.DataParallel, that are going to be useful for my application.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes all of the factory functions like <code>randn</code> or <code>ones</code> are independent of the nn.Module device.<br>
So in general, you want to provide a <code>device=</code> argument to these functions if you want the Tensor to be created on a special device.<br>
You can get the right device from any param or input of your Module.</p>
<p>Also, if you want Tensor attributes to be moved along with the parameters, you should register them as “buffers” on the Module with <code>self.register_buffer("q_transposed", the_tensor)</code> which you can then access as self.q_transposed.</p>
<p>Side not in your code sample, there is not need to wrap the Tensor into a Parameter here. Parameter are only important when you assign them to a nn.Module.</p>

username: lucastononrodrigues
cooked: <p>Right, thank you.<br>
However the torch.qr does not have a device argument, right?</p>

username: albanD
cooked: <p>The output device will be the same as the input. So you just need to make sure the randn in there is on the right device <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
LR-ASPP implementation mobilenet v3

username: henry_Kang
cooked: <aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1905.02244.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1905.02244.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1905.02244.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1905.02244.pdf" target="_blank" rel="noopener nofollow ugc">1905.02244.pdf</a></h3>

  <p class="filesize">522.98 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I am implementing the mobilenet v3 paper and face the error.<br>
In the paper, the figure 10 shows that after 16 stride they use the average pooling<br>
`</p>
<blockquote>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
class _LRASPP(nn.Module):<br>
“”“Lite R-ASPP”“”</p>
<pre><code>def __init__(self, in_channels, norm_layer, **kwargs):
    super(_LRASPP, self).__init__()
    out_channels = 128
    self.b0 = nn.Sequential(
        nn.Conv2d(in_channels, out_channels, 1, bias=False),
        norm_layer(out_channels),
        nn.ReLU(True)
    )
    self.b1 = nn.Sequential(
        nn.AvgPool2d(kernel_size=(49, 49), stride=(16, 20)),  # check it
        nn.Conv2d(in_channels, out_channels, 1, bias=False),
        nn.Sigmoid(),
    )

def forward(self, x):
    size = x.size()[2:]
    feat1 = self.b0(x)
    feat2 = self.b1(x)
    feat2 = F.interpolate(feat2, size, mode='bilinear', align_corners=True)
    x = feat1 * feat2  # check it
    return x
</code></pre>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
net = _LRASPP(in_channels=576,norm_layer=nn.BatchNorm2d)</p>
<pre><code>input_size=(1, 576,64, 32)
net=net.cuda()

x = torch.randn(input_size)
x=x.cuda()

net.eval()
for i in range(10):
    out = net(x)
print(out.shape)
</code></pre>
</blockquote>
<p>`<br>
However I face the error about<br>
RuntimeError: Given input size: (576x64x32). Calculated output size: (576x1x0). Output size is too small<br>
In the paper, they use the 1024 x 512 size input. Which mean, their final output is 64 32 after 16 stride.<br>
My question is how can we apply  the 49 x 49 average pooling?<br>
If I use the stride 8, of course I can apply it but the paper said they use 16 strides or 32 strides with or without LR ASPP. So It does not make any sense. Also no extra information is exists. Any idea for this?<br>
I tested the pytorch 1.0.0 and this code is working but 1.6 verstion this code has error.<br>
Still pooling operation has issus about it.</p>