akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to set longer timeout for DDP training?

Dazitu616: <p>I’m currently using DDP training on a large dataset. In evaluation, I only test the rank0 model for simplicity. It takes ~40min to run one eval epoch, and I set <code>dist.barrier()</code> in other threads to block the other models. However, since pytorch DDP has a default timeout of 30min, the training crashes everytime in the eval epoch.</p>
<p>According to the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="noopener nofollow ugc">document</a>, I can set <code>timeout</code> to a larger number when initing DDP. I’m confused about the statement that <em>this is applicable only if the environment variable <code>NCCL_BLOCKING_WAIT</code> or <code>NCCL_ASYNC_ERROR_HANDLING</code> is set to 1</em>. Does that mean in the code, I should write something like <code>os.environ['NCCL_ASYNC_ERROR_HANDLING'] = 1</code> in order to use a larger timeout value? <code>NCCL_BLOCKING_WAIT</code> and <code>NCCL_ASYNC_ERROR_HANDLING</code> which one is suitable for my use case? Thanks!</p>

pbelevich: <p><a class="mention" href="/u/rvarm1">@rvarm1</a> <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a> ?</p>

Dazitu616: <p>OK so I tried setting <code>timeout=timedelta(xxx)</code> in <code>init_process_group()</code> and that seems to work. Maybe we don’t need to manually set these environment variables</p>

Stephen-Hao: <p>Thank you very much! I  have solved the same problem.</p>
Loss is not Nan, but the gradients are

Sairaam_Venkatraman: <p>Hi all,<br>
I’ve found that in neural network, I’m coming across non-Nan losses with NaN grads. I’m using Adam with default parameters. Nothing fancy in my network. Has anyone come across such an issue? I’ve always known that NaN losses cause NaN gradients. But this is a bit odd.</p>

albanD: <p>Hi,</p>
<p>Many operations could give you <code>NaN</code> in the backward even with non-<code>NaN</code> values in the forward. For example, sqrt at <code>0</code>.</p>
<p>You will need to find where <code>NaN</code> appear in the backward to be sure.<br>
If you’re using master, you can use <a href="https://pytorch.org/docs/master/autograd.html#anomaly-detection">anomaly detection</a> to get that information.</p>

Sairaam_Venkatraman: <p>I don’t seem to have it. What is master, anyway?<br>
Also, while debugging, I’ve noticed gradients of the order of 1e-19. Could that be a problem?</p>

albanD: <p>What I call <code>master</code> is the version of pytorch from the current master branch on github. Not the releases.<br>
It depends what operations you are doing, 0/0 will give you NaN for example.</p>

hughperkins: <p>I’ve almost always had this problem as the result of takign the square root of 0.</p>
<p>Note that <code>.std()</code> implicitly takes the square root.</p>

hughperkins: <p>whooo, anomaly detection looks interesting <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Sairaam_Venkatraman: <p>Is there any known issue with softmax and small values?</p>

xuanyang: <p>If my pytorch version is 0.3, is there any method to check where causes the gradient NAN?Or, could you please give some examples?</p>

albanD: <p>To check the forward pass, you will need to add prints.<br>
To check the backward pass you will need to add hooks and prints.<br>
Unfortunately there is no easier way than doing it by hand in older versions.</p>

Guangyao_Li: <p>Hi~ Is it possible that loss is NaN while the gradients are not?</p>

ptrblck: <p>No, that should not be possible since the NaN loss value would be backpropagated and would create invalid gradients throughout the model. At least I wouldn’t know which operation can “recover” the gradient again and how it would work.</p>

Guangyao_Li: <p>Get it. Thanks for your patience!</p>
PyTorch + CUDA 11.6

yellowishlight: <p>Hello everyone,</p>
<p>As a follow-up to this question <a href="https://discuss.pytorch.org/t/pytorch-cuda-11-4/137900" class="inline-onebox">PyTorch + CUDA 11.4</a><br>
I have installed these Nvidia drivers version 510.60.02 along with Cuda 11.6.<br>
I’d like to install Pytorch in a conda virtual environment, and I’ve found in the Pytorch website that we couldn’t choose a stable version that relies on the latest versions of Cuda (the older version is 11.3) <a href="https://pytorch.org/get-started/locally" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>How can I install/build Pytorch with the latest version of Cuda in my conda virtual env ?</p>

ptrblck: <p>You could build PyTorch from source following <a href="https://github.com/pytorch/pytorch#from-source">these instructions</a> or install the nightly binaries with CUDA11.6 via:</p>
<pre><code class="lang-python">pip install torch --pre --extra-index-url https://download.pytorch.org/whl/nightly/cu116
</code></pre>

yellowishlight: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I installed it using the nightly binaries. Thanks a lot for your help !</p>

Phillip_J_Murphy: <p>Thank you so much. Also worked for torchvision.</p>

skoenman: <p>Just for future ref one can get all packages under that link… torchserve aswell.</p>

jianlong_Huang: <p>Hi, sir. I have one question: If I installed cuda 113 version pytorch, but my GPU is cuda 11.6. Is this mismatch of version affect the reproducibility of code. Since I set all random seeds and no deterministic algorithms used in code, but the training loss between different running times is still around 1%.</p>

ptrblck: <p>No, your locally installed libraries should not introduce non-determinism as they should not be used since the PyTorch binaries ship with their own CUDA runtime, cuDNN, cublas etc. and will use these.</p>
Out of memory error after certain epoch

mojzaar: <p>I reviewed the posts linked with this topic where folks encountered this error.</p>
<p>I eliminated all intermittent variables that may have grad graphs and used item() and detach() on all variables after the backward process was completed, then for testing and validation, I used torch.no grad() to not retain the grads during those processes.</p>
<p>I have a DataParallel GAN model with five 24 GB GPUs. The Generator consists of a UNet, 5 CNN layers for the Discriminator, a Wasserstein loss with gradient penalty as an adversarial loss, and MSE for the Generator. The input and reference images are both 512x512 pixels.</p>
<p>When I use nvidia-smi, I have 4 GB free on each GPU during training because I set the batch size to 16. There is even more free space upon validation (round 8 GB on each).</p>
<p>However, after a certain number of epochs, say 30ish, I receive an out of memory error, despite the fact that the available free GPU does not change significantly during training.</p>
<p>If I set the batch size to something unreasonably small, like 3 or 5, it will work (at least for more than 100 epochs), but it will take more than 20 to 30 minutes to complete a single epoch, which is unacceptable. I’d love it if you could share a solution to this problem with us so that many people like me can use it to have a sustainable training process.</p>
<p>I’m running Ubuntu 20 along with torch 1.10.1+cu113.</p>

ptrblck: <p>It’s hard to speculate what might be causing the issue, as your steps sounds reasonable and based on your description (“I have 4 GB free on each GPU during training”) no OOM should be raised at all.<br>
I would recommend to try to narrow down which operation is causing the OOM by checking the stacktrace and by adding debug print statements showing the current memory usage.</p>

mojzaar: <p>I figured out what the issue was.</p>
<p>I was calculating the MSSSIM of the results using the TorchMetrics library. I recognized that the MSSIME class will act as a loss function and will contain the gradient graphs that will cause the issue. I was able to address the problem by replacing the MSSIM class with mmssim functional.</p>
Cannot compile PyTorch under gcc 7, 8, 10, 11, or 12

Peter_Schmidt-Nielse: <p>Hello folks, I’m struggling to compile PyTorch. It doesn’t compile on my Ubuntu 22.04 machine with gcc version 10 or 11, due to crashes in gcc (I get <code>cc1plus: internal compiler error: Segmentation fault</code>), it doesn’t compile with gcc version 12 due to an Nvidia header rejecting gcc 12, but even if I comment out that check it still fails to compile with gcc 12, due to some warnings that are promoted to errors while compiling some MKL stuff (even when I set <code>USE_MKLDNN=0</code>). I’ve also tried compiling in the <code>pytorch/pytorch:1.12.1-cuda11.3-cudnn8-devel</code> and <code>pytorch/pytorch:latest</code> docker containers, and I can’t get it compiling there under gcc 7 or 8 either, following the exact instructions from the main README – again, I get <code>internal compiler error: Segmentation fault</code> crashes.</p>
<p>I’m struggling to see how one is supposed to be able to build PyTorch from source. Is there some docker container that it’s known to build in? I feel like I’ve tried every permutation of environments, and they all crash one way or another.</p>
<p>Thanks,</p>
<p>-Peter</p>

ptrblck: <aside class="quote no-group" data-username="Peter_Schmidt-Nielse" data-post="1" data-topic="160335">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peter_schmidt-nielse/48/46296_2.png" class="avatar"> Peter_Schmidt-Nielse:</div>
<blockquote>
<p>Is there some docker container that it’s known to build in?</p>
</blockquote>
</aside>
<p>These containers are building PyTorch:</p>
<aside class="quote no-group" data-username="Peter_Schmidt-Nielse" data-post="1" data-topic="160335">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peter_schmidt-nielse/48/46296_2.png" class="avatar"> Peter_Schmidt-Nielse:</div>
<blockquote>
<p>I’ve also tried compiling in the <code>pytorch/pytorch:1.12.1-cuda11.3-cudnn8-devel</code> and <code>pytorch/pytorch:latest</code> docker containers</p>
</blockquote>
</aside>
<p>I don’t know which base OS they are using but if it’s Ubuntu 22.04, they would be using gcc 9.4 (I think).<br>
The wheels are using a CentOS container but I don’t know which gcc version they are using.</p>
Is scaling down the loss the same as scaling down the gradient?

Ioan: <p>The output of the network has a certain loss, however, I want to scale down the loss before propagating. (multitask learning)</p>
<p>Is</p>
<pre><code class="lang-auto">
loss = loss * weight # weight can be 0.5 for example
loss.backward()
</code></pre>
<p>same as<br>
<code>logits.data.backward(grad * weight)</code></p>

ptrblck: <p>Yes, loss scaling should also scale the gradients as seen in this small example:</p>
<pre><code class="lang-python">model = models.resnet152()
x = torch.randn(2, 3, 224, 224)

loss1 = model(x).mean()

# standard
loss1.backward(retain_graph=True)
grads1 = [p.grad.clone() for p in model.parameters()]

# scale up
model.zero_grad()
loss2 = loss1 * 2
loss2.backward(retain_graph=True)
grads2 = [p.grad.clone() for p in model.parameters()]

# scale down
model.zero_grad()
loss3 = loss1 * 0.2
loss3.backward(retain_graph=True)
grads3 = [p.grad.clone() for p in model.parameters()]

# compare
for g1, g2, g3 in zip(grads1, grads2, grads3):
    print('g2.sum()/g1.sum() {}'.format(g2.sum() / g1.sum()))
    print('g3.sum()/g1.sum() {}'.format(g3.sum() / g1.sum()))
</code></pre>
Multiprocess Sample all identical - No Randomness

pmcvay: <p>When running a model and then sampling from the output distribution in multiple process with the multiprocessing module, every sample is the same even though they should be random. What is the reason for the sample being identical? A simple working example is:</p>
<pre><code class="lang-python">import multiprocessing
import torch
import torch.nn as nn

class test(multiprocessing.Process):
    def __init__(self, weights):
        multiprocessing.Process.__init__(self)
        self.m = nn.Linear(10, 10)
        self.m.load_state_dict(weights)
        
    def run(self):
        out = self.m(torch.ones(10))
        dist = Categorical(logits=out)
        print(dist.sample())

def main():
    orig_model = nn.Linear(10,10)
    weights = orig_model.state_dict()
    workers = [test(weights) for i in range(100)]
    for w in workers:
        w.start()

    for w in workers:
        w.join()
</code></pre>
<p>with output</p>
<pre><code class="lang-auto">tensor(2)
tensor(2)
...
tensor(2)
tensor(2)
</code></pre>

fduwjj: <p>Looks like you are override the weight from a global weight before spawning multiple processes. Or am I missing something here?</p>

pmcvay: <p>Yes, I override the weights so each worker has identical weights. But that shouldn’t affect the randomness. It just means every worker has the same distributions - not that every sample from the same distribution should be the same</p>

fduwjj: <p>Ah I see. Does use torch.manual_seed(self.rank) inside your run help here?</p>

pmcvay: <p>Inside the <code>run</code> function it helps. But inside <code>__init__</code> it does not help</p>
Difference of computational efficiency between Conv1D and Conv2D

Blankit: <p>Hi，<br>
The original network is constructed by Con1D, while Conv1D can’t be supported in edge devices. So I use Conv2D to implement Conv1D and change some operators’ parameters. Conv2D  took much more time to train for a epoch (9mins vs 2hs).</p>

ptrblck: <p>I don’t know which device and backend you are using, but e.g cuDNN uses the same approach of unsqueezing a spatial dimension as seen <a href="https://github.com/pytorch/pytorch/blob/7088a98fba3a5031a2afc293cbf25cec09f248a5/aten/src/ATen/native/Convolution.cpp#L1085-L1093">here</a> while e.g. MKL seems to natively support 1D convs.</p>
Can not reproduce the result even set all random seed

jianlong_Huang: <p>Problem: I’m implementing a GNN in pytorch 1.11 with cuda 113. I tried to set all random seeds using the below code, but still couldn’t get the same result on different epochs. At first, the console told that I used torch_scatter_mean, which is not a deterministic operation. Then I replaced the scatter_mean operation with a 100% deterministic operation, but it still doesn’t work.</p>
<p>Platform I used: windows 10<br>
Training on: cuda:0</p>
<p>seed_ = 42</p>
<p>def seed_torch(seed):<br>
random.seed(seed)<br>
os.environ[‘PYTHONHASHSEED’] = str(seed)  # 为了禁止hash随机化，使得实验可复现<br>
os.environ[“CUDA_LAUNCH_BLOCKING”] = “1”<br>
os.environ[“CUBLAS_WORKSPACE_CONFIG”] = “:16:8”<br>
np.random.seed(seed)<br>
torch.manual_seed(seed)<br>
torch.cuda.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed)  # if you are using multi-GPU.<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = True<br>
torch.use_deterministic_algorithms(True)</p>
<p>seed_torch(seed_)</p>
<p>The package I used:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13ebb2b123b659d6d42aa87d19d0846a073345f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/13ebb2b123b659d6d42aa87d19d0846a073345f6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13ebb2b123b659d6d42aa87d19d0846a073345f6.png" alt="image" data-base62-sha1="2Qe3gdSn8EL08uYaBnRUwlUDQDc" width="690" height="253" data-dominant-color="2D313A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">739×271 27.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
import time<br>
import os<br>
import numpy as np<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import torch.utils.data as Data<br>
import random<br>
from sklearn.cluster import KMeans<br>
from torch_geometric.nn import MetaLayer</p>
<p>from model import Model<br>
from dataset import trainDataset, valDataset, testDataset<br>
import argparse</p>
<p>import matplotlib.pyplot as plt</p>
<p>One more question: is it possible dataloader is not deterministic?<br>
train_loader = Data.DataLoader(train_dataset, batch_size=args.batch_size,<br>
shuffle=True, num_workers=0, pin_memory=True)<br>
val_loader = Data.DataLoader(val_dataset, batch_size=args.batch_size,<br>
shuffle=False, num_workers=0, pin_memory=True)</p>
<p>test_loader = Data.DataLoader(test_dataset, batch_size=args.batch_size,<br>
shuffle=False, num_workers=0, pin_memory=True)</p>
<p>I used a TransformerEncoderLayer, so I changed CUBLAS_WORKSPACE_CONFIG to :16:8:</p>

ptrblck: <p>Could you post a minimal, executable code snippet reproducing the issue so that we could debug it, please?</p>
Python self error

Angelina_Robert: <p>Hi, just wondering if any one could help me with the following code and its error</p>
<p>import torch</p>
<p>from myclass import myclass</p>
<p>def parse_option(self):<br>
print(‘Line 7 ‘)<br>
self.myclass = myclass<br>
parser = argparse.ArgumentParser(‘argument for training’)<br>
parser.add_argument(’–print_freq’, type=int, default=10, help=‘print frequency’)<br>
parser.add_argument(‘–save_freq’, type=int, default=50,  help=‘save frequency’)</p>
<pre><code>print('Line 15')
</code></pre>
<p>def main():<br>
print(‘Line 17’)<br>
opt = parse_option()</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
main()</p>
<p>Error:<br>
File “test_tensor.py”, line 22, in <br>
main()<br>
File “test_tensor.py”, line 18, in main<br>
opt = parse_option()<br>
TypeError: <strong>parse_option() missing 1 required positional argument: ‘self’</strong></p>

ptrblck: <p><code>def parse_option(self)</code> seems to be initialized as a class method, while you are defining it in a global namespace. Did you copy this method from a class? If so, remove the <code>self</code> argument and it should work.</p>
Speech commands dataset

s_n: <p>Hi,</p>
<p>Is there a repository or a code base for the SpeechCommands dataset? I found the below link but it doesn’t have the code for adding background noise etc.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/speech_command_classification_with_torchaudio_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/speech_command_classification_with_torchaudio_tutorial.html</a></p>

nivek: <p>I believe the Dataset <a href="https://github.com/pytorch/audio/blob/main/torchaudio/datasets/speechcommands.py" rel="noopener nofollow ugc">source code is here in <code>torchaudio</code></a>. It seems to exclude the ‘_background_noise_’ folder so you may need to make modification to include it.</p>

s_n: <p>Hi,</p>
<p>Thanks for the reply. I think the addition of background noise is not that straightforward because the size of the noise signal is not equal to the audio signal size. The source code for its implementation in TensorFlow is in the below link. So I was wondering if something similar is there in PyTorch as well or not.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/ARM-software/ML-KWS-for-MCU/blob/master/input_data.py">
  <header class="source">

      <a href="https://github.com/ARM-software/ML-KWS-for-MCU/blob/master/input_data.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/ARM-software/ML-KWS-for-MCU/blob/master/input_data.py" target="_blank" rel="noopener nofollow ugc">ARM-software/ML-KWS-for-MCU/blob/master/input_data.py</a></h4>


      <pre><code class="lang-py"># Copyright 2017 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Model definitions for simple speech recognition.

"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
</code></pre>



  This file has been truncated. <a href="https://github.com/ARM-software/ML-KWS-for-MCU/blob/master/input_data.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Embedding_bag optimizations for performance

Rishabh_Jain: <p>Hello,</p>
<p>I am trying to understand how much optimized are the embedding operators available in pytorch. I intend to learn about the optimizations for improving execution time on CPU/GPU. Like, the simple functionality is to do a gather (lookup on a big table across distant indices making it an irregular operation) and then apply reduce (addition of contiguous embeddings making it a regular  operation).</p>
<p>I understand that one optimization is the fusion of the gather and reduce which minimizes the generation of intermediate tensors. But, apart from that, are there other optimizations? like usage of AVX/SIMD, sorting the indices to exploit memory locality, etc.</p>
<p>Can someone indicate me on what optimizations are built on top of the basic/navie functionality and if possible point to the related references?</p>
<p>Thanks,<br>
Rishabh Jain</p>
Access the adaptive learning rates for different torch optimizers

Megh_Bhalerao: <p>Hello,<br>
Different optimization algorithms such as ADAM, Adagrad, RMSProp adapt their step size according to the gradients, for example adagrad accumulated the gradient L2 norm and the learning rate is scaled according to that, and RMSProp does an exponential moving average with a momentum parameter to accumulate the L2 norm of the gradients according to the torch.optim docs here - <a href="https://pytorch.org/docs/stable/optim.html" class="inline-onebox" rel="noopener nofollow ugc">torch.optim — PyTorch 1.12 documentation</a></p>
<p>I was wondering if there is any clean way of accessing these adaptive learning rates rather than me manually calculating it? Is there some instance variable that I can access externally from the optimizer object? If this is not present already, it would be great to have this.</p>
<p>Please do let me know and thanks for your time.<br>
Megh</p>

ptrblck: <p>You can check the <code>state_dict</code> of the optimizer and see the internal tracking stats:</p>
<pre><code class="lang-python">model = nn.Linear(2, 2, bias=True)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

out = model(torch.randn(1, 2))
out.mean().backward()
optimizer.step()

print(optimizer.state_dict())
# {'state': {0: {'step': tensor(1.), 'exp_avg': tensor([[-0.0394, -0.0382],
#         [-0.0394, -0.0382]]), 'exp_avg_sq': tensor([[0.0002, 0.0001],
#         [0.0002, 0.0001]])}, 1: {'step': tensor(1.), 'exp_avg': tensor([0.0500, 0.0500]), 'exp_avg_sq': tensor([0.0003, 0.0003])}}, 'param_groups': [{'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'maximize': False, 'foreach': None, 'capturable': False, 'params': [0, 1]}]}
</code></pre>
<p>Please don’t tag specific users, as it can discourage others to post an answer.</p>

Megh_Bhalerao: <p>Hi, thanks for this and my apologies for tagging specific users. I have edited my OP.</p>
Weighted random sampler pytorch

Prachi: <p>Can someone explain intuitively how a weighted random sampler works?</p>
<ol>
<li><code>list(WeightedRandomSampler([0.1, 0.9, 0.4, 0.7, 3.0, 0.6], 5, replacement=True))</code></li>
</ol>
<p>E.g. The above code says:<br>
If I have an image dataset, here I have 6 images and each has probs (weights).<br>
Now the sampler draws according to these weights meaning there are high chances to choose the image at index 1 then at index 3, 5, 2, 4, and 0 that too with replacement.</p>
<ol start="2">
<li>
<code>list(WeightedRandomSampler([0.9, 0.4, 0.05, 0.2, 0.3, 0.1], 5, replacement=False))</code><br>
Here are high chances that the sampler will choose images at the 0th index, 1st index,4th index, and so on but without replacement so that each image will get selected as we num_samples = 5.</li>
</ol>
<p>Thank you!!</p>

ptrblck: <p>I’m unsure where exactly you are stuck, but generally you can see the passed weights similar to probabilities, i.e. the higher, the more likely the sampler will pick this particular sample.<br>
The main difference is of course that the sum of all weights does not need to be <code>1.</code> as would be the case with probabilities.<br>
The <code>replacement</code> argument just defines if the sampler can re-draw the same sample again or if you would remove the sample from the “pool” after it was drawn.</p>

Prachi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your reply!</p>
<p>I have a CSV file of 100k rows and two columns = [‘ImageId’, ‘weight’], weights are in the range of [0,1], I want to make use of PyTorch’s weighted random sampler to sample images according to the associated weights. I don’t want to build an efficient sampler from scratch as I assume PyTorch’s APIs would save me time as I am desperately trying to achieve this. Any leading points would be really helpful.</p>
Using model functions in DistributedDataParallel

numpee: <p>I’ve noticed something peculiar in DDP training, but cannot seem to find any related posts.</p>
<p>I have an ordinary ConvNet, whose <code>forward()</code> is defined in the following manner:</p>
<pre><code class="lang-python">class MyModel(Module):
    ...
    def forward(self, x):
        mid = self.feature_extractor(x)
        return self.classifier(mid)

    def feature_extractor(self, x):
        # Some layers here

    def classifier(self, x):
        # Some layers here
</code></pre>
<p>In some cases, I want the mid-level features, and in others, I don’t want them at all. Also, sometimes I have mid-level features from elsewhere, and just want a forward pass through the classifier. Therefore, instead of adding a conditional to my <code>forward</code> method, I’ve split the <code>forward</code> into two sub-methods, so I can use the <code>feature_extractor</code> and <code>classifier</code> methods as needed.</p>
<p>It turns out, in DDP (maybe for DP as well?) there is a difference between training with:</p>
<ol>
<li>
<code>out = model(x)</code>, and</li>
<li><code>mid = model.module.feature_extractor(x); out = model.module.classifier(mid)</code></li>
</ol>
<p>In the case of (1), the model trains perfectly as expected. However, in the case of (2), the loss does not converge as efficiently as expected, and may not even converge to the same value as (1).</p>
<p>My question is: What would be the underlying mechanism that makes a difference between (1) and (2)? Could it be that (2) does not perform gradient synchronization across processes? Or perhaps <code>nn.SyncBatchNorm</code> is not working as expected?</p>
<p>Thanks</p>

ptrblck: <p>Yes, the second approach will not work with <code>DDP</code> since you are explicitly calling the underlying model placed on the current device. <code>DDP</code> will not sync the gradient as it depends on its <code>__call__</code> method being invoked via a direct <code>model(x)</code> call.</p>

numpee: <p>I see, thanks for the reply. Would this apply to <code>DataParallel</code> as well (perhaps gradients are not reduced to <code>gpu:0</code>?)</p>

ptrblck: <p>Yes, <code>nn.DataParallel</code> also depends on its <code>forward</code> method as the data parallelization logic is used there as seen <a href="https://github.com/pytorch/pytorch/blob/eebdcb5a2ef6a117a608b9ca5ca1eb2fd4f72fbd/torch/nn/parallel/data_parallel.py#L150-L172">here</a>.</p>
Mask Position Embedding without break the graph

Olii: <p>Hi,</p>
<p>I have a simple learnable positional embedding and want to mask based on the pad index. However, it breaks the graph when I do <code>expand_as</code>. Should I just add <code>clone()</code> after this?</p>
<pre><code class="lang-python">pos_embed = nn.Embedding(n_position, dim)

def forward(embeds, entities):
    # batch, n, dim
    seq_len = embeds.shape[1]
    mask_idx = (entities == 0).nonzero(as_tuple=True)

    positions = torch.arange(seq_len, dtype=torch.long,  device=entity_embeds.device).unsqueeze(0)
    pos_embeds = self.pos_embed(positions).expand_as(embeds).clone()
    # this would break the embeddings when I do in place 
    # pos_embeds = self.pos_embed(positions).expand_as(embeds)
    
    
    # mask based on the index
    pos_embeds[mask_idx] = 0
    return embeds + pos_embed

</code></pre>
<p>Thanks</p>
Get two samples of different category from one dataset

EEQ: <p>I’m trying to define a <code>Dataset</code>/<code>Dataloader</code> for an image style transfer network. I have a dataset of images grouped by styles, and I want each sample from this dataset to consist of two images, one for style, the other for content. My first idea was to implement a <code>Dataset</code> with something like this in <code>__init__</code>:</p>
<pre><code class="lang-auto">n = len(images)
itr_style = random.shuffle([i for i in range(n)]))
itr_content = random.shuffle([i for i in range(n)]))
</code></pre>
<p>and this in <code>__getitem__</code>:</p>
<pre><code class="lang-auto">return (images[itr_style[index]], images[itr_content[index]])
</code></pre>
<p>Which is probably not the most efficient implementation, and I also need to make sure that:</p>
<ol>
<li>The two images don’t come from the same style</li>
<li>The dataset re-shuffles every epoch</li>
</ol>
<p>What is the best way to implement this <code>Dataset</code>?</p>

nivek: <p>If I understand correctly, you should be able to compute all valid combinations of indices <code>(i, j)</code> ahead of time, where <code>i</code> is the index for an image for style and <code>j</code> is the index for an image for content. Basically, you can do something like:</p>
<pre><code class="lang-python">valid_combinations = []
for i in range(n):
    for j in range(n):
        if _:  # images don't come from the same style
            valid_combinations.append((i, j))
</code></pre>
<p>Your <code>__getitem__</code> can read the indices from <code>valid_combinations</code> and then get the corresponding images.</p>
PyTorch nn.Embedding takes too much memory

Larisa_Biriescu: <p>Hello, I’m implementing a deep neural network and for that I use torch Embedding in order to encode the input data. The problem is that the input id’s go up to 120 000 and when I ran the model, it requires me a lot of GB RAM. Do you have any idea how can i solve this problem?</p>

nivek: <p>How are you loading your data? Is the entirety of the input data loaded into memory ahead of time?</p>
<p>If so, you can consider loading them lazily or use Iterable-style Dataset or <a href="https://github.com/pytorch/data" rel="noopener nofollow ugc">IterDataPipe from TorchData</a>.</p>

ptrblck: <p>It seems you are concerned about the size of this particular embedding layer.<br>
I don’t know how large the <code>embedding_dim</code> is, but for <code>num_embeddings=120000</code> and <code>embedding_dim=1024</code>, you would only use ~470MB:</p>
<pre><code class="lang-python">print(torch.cuda.memory_allocated()/1024**2)
# 0.0

emb = nn.Embedding(120000, 1024).cuda()
print(torch.cuda.memory_allocated()/1024**2)
# 468.75
</code></pre>

Larisa_Biriescu: <p>Thank you guys for your answers! I figured out where the problem was located: in some place a defined 120 000 *10 000 embedding layer - that generated the OOM error</p>
Reinitiate the weights and biases

sunny1: <p>I have a MLP network and i want to use the same MLP with different seed value. I want to know if i can reinitiate the weights and biases values as it should be in start while calling the MLP in loop.</p>

tom: <p>Many of the standard PyTorch modules have a <code>.reset_parameters()</code> method performing the (re-)initialization of that module (but not the submodules, so you would need to traverse with <code>.apply(fn)</code> on the root module). But this is a convention and not enforced.<br>
Another method could be a for loop over <code>.named_parameters()</code> and set the parameter in a <code>.no_grad</code> block. This is particularly useful if you have your own ideas about how to initialize weights (you should!).</p>
<p>Best regards</p>
<p>Thomas</p>
Is there a gold standard of how much percentage of the data should be used as the final blind test?

SouthTorch: <p>And the rest be used for training.<br>
How did you guys decide this? Thanks.</p>

tom: <p>There is no golden recipe, but a train/dev/test split of 80%/10%/10% is not uncommon.</p>
<p>Note that the test performance is a random variable, so the reliability of the estimate strongly depends on the sample size. On the other hand, if 10% is small for a test set, chances are that the 80% are small for a training set, too.<br>
Depending on your data, you might want to stratify the splitting.</p>
<p>Best regards</p>
<p>Thomas</p>
TORCH.NN.FUNCTIONAL.GRID_SAMPLE invariant to sign of the grid

Henar_Dominguez: <p>Hello!</p>
<p>I’ve been using TORCH.NN.FUNCTIONAL.GRID_SAMPLE to warp an optical flow (vector field) over an image, only rotating the image by the center, and I realized that the function returns the same result if I pass it the optical_flow or the -optical_flow. Is this a normal behaviour?</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">import torch.nn.functional as Fu
transformed = Fu.grid_sample(input_image, optical_flow, align_corners=True, mode="bilinear")
</code></pre>
<p>Thanks!</p>

tom: <p>When you say “optical flow” and “vector field”, are you entering displacements or are you entering coordinates in the -1…1 range? If not, you might you need to add the “current position” and/or scale to the input convention of <code>grid_sample</code>.</p>
<p>I’m using this example and see a difference when I change the sign of the coordinate field.</p>
<pre><code class="lang-python">ls = torch.linspace(-1, 1, 50)
y = ls[:, None].expand(50, 50)
x = ls[None, :].expand(50, 50)
coords = torch.stack([y, x], -1)[None]
inp = torch.arange(20*20, dtype=torch.float).view(1, 1, 20, 20)
out = torch.nn.functional.grid_sample(inp, -coords)  # flip sign on coords and compare
from matplotlib import pyplot
pyplot.imshow(out[0, 0])
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

Henar_Dominguez: <p>ok thank you very much!</p>
<p>In your example it works as I expected, so I will look for a minimum reproducible example of my exact problem and I will update the post when I have it!</p>
<p>Again thank you for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Loading data for multilabel classification, too large for memory

Robin_Muller: <p>I have one folder containing ~30k images. There are 19 classes, and an image can belong to n of them (multilabel classification). Therefore, i cant use the <code>ImageFolder</code> function. I also need the name of each image-file, since the filename contains the ID of the image and there is a corresponding Dataframe including the classes of each ID. What is the most efficient way loading this data?</p>

Robin_Muller: <p>I’ve found a solution by implementing your own version of <code>torch.utils.data.Dataset</code>. A detailed tutorial <a href="https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel" rel="noopener nofollow ugc">here</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Is it possible to regularize gradients without multiple graph traversal?

Soudini: <p>I am currently trying to regularize the training of a network using the norm of the gradients of one of the output relative to one of its inputs.</p>
<p>The issue I am facing currently is that as I have to compute the gradients in the loss, I have to call <code>autograd.grad</code> which triggers one graph traversal and then call <code>loss.backward()</code> to update the weights of my network.</p>
<p>Currently, this doubles the training time because I spend a lot of time in the backward. Is there a better way to do it ?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/soudini">@Soudini</a>,</p>
<p>You could create a custom optimiser and re-use those gradient terms within the <code>optim</code> itself. But do share a minimal reproducible example to highlight your problem!</p>

Soudini: <p>Hi, thank you for the response.</p>
<p>What I want to do is something like this:</p>
<pre><code class="lang-auto">inputs = torch.rand(10, 10)
pred = model(inputs)

# Here is my first graph traversal
grad = grad(pred.sum(), inputs, retain_graph=True)[0][:,:5] #let's say that I only need the gradients relative to the 5 first inputs

# Here is my second graph traversal
loss = grad.norm(dim=1).sum() + (pred - target).norm() 
loss.backward()
optim.step()
</code></pre>
<p>Currently this takes around twice the time to perform compared to not regularizing the gradient.</p>
<p>I will also try your suggestion and come back to you.</p>
Accessing Python Dictionary

Angelina_Robert: <p>Hi,<br>
Just wondering if any one could help with the following error:<br>
ratio=6<br>
label_sample_dict= {0: 61, 1: 28, 2: 21, 3: 6, 4: 4, 5: 4, 6: 1, 7: 3}<br>
for label, samples in label_sample_dict:<br>
self.samp_queue[label] = [Queue(maxsize=samples*ratio), samples]</p>
<p>Error:    for label, samples in label_sample_dict:<br>
TypeError: cannot unpack non-iterable int object</p>

ptrblck: <p>Since <code>label_sample_dict</code> is a Python <code>dict</code> iterating it should return the <code>key</code> while you are trying to get two return values. I guess the unpacking then fails. Try to use <code>for key in label_sample_dict</code> and use the <code>key</code> to access the <code>value</code>.</p>
PyTorch on m1 max using anaconda

Kishor_holmes: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a2cc2cfe1f062b23c72509d5de1f9faad6107bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a2cc2cfe1f062b23c72509d5de1f9faad6107bf" title="Screen Shot 2022-08-29 at 4.11.27 pm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a2cc2cfe1f062b23c72509d5de1f9faad6107bf_2_690x406.png" alt="Screen Shot 2022-08-29 at 4.11.27 pm" data-base62-sha1="615X87AZws6eLbJNc0zn9SOUocn" width="690" height="406" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a2cc2cfe1f062b23c72509d5de1f9faad6107bf_2_690x406.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a2cc2cfe1f062b23c72509d5de1f9faad6107bf_2_1035x609.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a2cc2cfe1f062b23c72509d5de1f9faad6107bf.png 2x" data-dominant-color="2B353E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-08-29 at 4.11.27 pm</span><span class="informations">1246×734 285 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My PyTorch installation on an m1 max machine worked fine upon installation. However, when I restarted anaconda, I received the following error. Assume there is some installation conflict. Any help would be greatly appreciated.</p>
Word Question Processing

penguin-number-123: <p>I am trying to be able to parse a word question e.g. " John needs 5 minutes to walk back home from school. How far is his school from his home if he walks at a constant speed of 4km/h?"<br>
However, I am not sure how to do this.</p>
<p>An alternative method I have thought of is to directly use the question as the input, then put the equation as the output, which can then be evaluated by some other code. Unfortunately, I do not know how to have the output be not fixed. Any help is appreciated; thanks!</p>
How is this an inplace operation error?

Ananyapam_De: <p>I am running into this problem again and again:</p>
<pre><code class="lang-auto">Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>The code as shown by the backtrace is the following-</p>
<pre><code class="lang-auto">v = self.id_tensor_list(self.d)
S_n = 0
for i in range(self.d):
    S_n = S_n + torch.mul(torch.diag(v[i]), torch.exp(self.phi[i][2] * torch.log(torch.exp(self.phi[i][3]) + self.Y[n][i])))
return S_n
</code></pre>
<p>The backtrace tells me that the line</p>
<pre><code class="lang-auto">S_n = S_n + torch.mul(torch.diag(v[i]), torch.exp(self.phi[i][2] * torch.log(torch.exp(self.phi[i][3]) + self.Y[n][i]))) 
</code></pre>
<p>operation is the issue, but I don’t understand how is this an inplace operation?</p>
<p>v is a list of tensors, Y is a tensor, phi is a tensor with <code>requires_grad=true</code></p>
<p>How do I fix this issue?</p>

srishti-git1110: <p>Can you please post some code that reproduces the error?<br>
From this piece of code, I cannot really figure out where the in-place op is taking place as all functions you used, viz. exp, log, mul, diag return new tensors only.</p>
<p>I tried reproducing the error using a very similar code, but it ran error free meaning I wasn’t able to reproduce what you are trying to do.</p>
How to Add 0-Value Columns/Rows to Tensor

zhanwenchen: <p>Hi all,</p>
<p>This is related to <a href="https://discuss.pytorch.org/t/is-there-a-way-to-insert-a-tensor-into-an-existing-tensor/14642" class="inline-onebox">Is there a way to insert a tensor into an existing tensor?</a> but vectorized.</p>
<p>I want to add additional dummy categories to an object detector. Suppose the original vision model predicts 151 categories. How would I add 22 new 0 categories with a known order (they can be in the middle of the 151 so it wouldn’t be left/right-padding).</p>
<pre><code class="lang-python">obj_dists = torch.rand(650, 151) # suppose there are 650 RoIs in this image.
mapping_151_177 = torch.randint(151) # index of the 177 corresponding to the 
mapping_177_151 = torch.randint(177) # index of the 177 corresponding to the 
</code></pre>
<p>Now my additional question is whether this is a dumb idea to start with, even if it’s feasible. Does this affect the gradient flow of the original data, for example? Or does this totally calls into question the losses?</p>

ptrblck: <p>If I understand your use case correctly, you would like to add new class ids into the targets as pre-defined locations (which is not necessarily at the beginning or end of the current target classes).<br>
If so, you would of course change the entire datset and would most likely need to retrain the model.<br>
The class correspondence between the model outputs and the target will be broken unless you also “shift” the model output logits by the same amount. In any case, adding new classes to the “right” of the already used classes sounds like an easier approach, so could you explain why you want to perform this remapping?</p>
Training loss remains stable!

goksu: <p>Hello!</p>
<p>I am working on the MS-COCO dataset for the multi-label classification task. While training my neural network the model does not learn anything. I mean training loss values are nearly similar and training loss remains stable (e.g. loss value min. ~ 0.68, max. ~0.71 ) after many iterations. I use the  ResNet18 model with BCELogitLoss and AdamW optimizer, and no data augmentation. I tried many things such as change learning rate, weight_decay, batch size, data augmentation, different model architecture, and different loss, etc. But there are not any changes at all. Do you have any ideas or suggestions?</p>
<p>Thank you very much in advance!</p>
Am I using the transfer learning using TIMM correctly

bing: <p>Hi,<br>
I am trying to do a binary image classification.<br>
I am using model zoo of <a href="https://github.com/rwightman/pytorch-image-models" rel="noopener nofollow ugc">timm</a>.<br>
I just want to confirm whether I am doing the transfer learning correctly or not.<br>
Below is my model loading code -</p>
<pre><code class="lang-auto">import timm
model_transfer = timm.create_model('convnext_tiny_in22k', pretrained=True,num_classes=2) # noqa

</code></pre>
<p>Then passing the model to run on the dataset<br>
Please let me know of any other suggestions also, if any.<br>
Thanks.</p>

InnovArul: <p>It looks correct to me.<br>
If it is binary image classification, probably you can pass <code>in_chans=1</code> to <code>timm.create_model()</code>. You can decide based on your use case.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://timm.fast.ai/models#How-is-timm-able-to-use-pretrained-weights-and-handle-images-that-are-not-3-channel-RGB-images">
  <header class="source">
      <img src="https://timm.fast.ai/assets/images/favicon.ico" class="site-icon" width="16" height="16">

      <a href="https://timm.fast.ai/models#How-is-timm-able-to-use-pretrained-weights-and-handle-images-that-are-not-3-channel-RGB-images" target="_blank" rel="noopener">timm.fast.ai</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://timm.fast.ai/models#How-is-timm-able-to-use-pretrained-weights-and-handle-images-that-are-not-3-channel-RGB-images" target="_blank" rel="noopener">Models API and Pretrained weights | timmdocs</a></h3>

  <p>Documentation for timm library created by Ross Wightman</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


bing: <p>Images are RGB, I really don’t think I’ll need to change the in_chans though.</p>