akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Quantized CNN model gives the wrong output on Android

username: komeiji_satori
cooked: <p>I trained a CNN image deblurring model and exported a .ptl model following the tutorial. It worked well on my demo app (can give a sharp output for blurred image).</p>
<p>After that I used quant aware training with qnnpack backend. The output is still good when I reload the quantized .ptl model using <code>torch.jit.load</code> in Python. However, the output comes bad when I use it on my app. There is no running errors but the result is different. I did not change any java code in my app compared with the model before quantization.</p>
<p>I’ve optimized the model like this:</p>
<pre><code class="lang-auto">script_module = torch.jit.script(model)
optimized_scripted_module = optimize_for_mobile(script_module )
optimized_scripted_module._save_for_lite_interpreter("mobile_classification.ptl")
</code></pre>
<p>Library used for Pytorch Android :</p>
<pre><code class="lang-auto">implementation 'org.pytorch:pytorch_android_lite:1.9.0'
implementation 'org.pytorch:pytorch_android_torchvision_lite:1.9.0'
</code></pre>
Requires_grad of list[0] Disappears

username: Glory_Chen
cooked: <pre data-code-wrap="python"><code class="lang-python">def f():
    x = []
    for i in range(3):
        x.append(torch.tensor(i * 1.).clone().requires_grad_(True))
    print(x)
</code></pre>
<p>Expected results:</p>
<pre data-code-wrap="bash"><code class="lang-bash">[tensor(0., requires_grad=True), tensor(1., requires_grad=True), tensor(2., requires_grad=True)]
</code></pre>
<p>This function can work well in the terminal. However, in my project code (in the same env), <code>x[0].requires_grad</code> is always <code>False</code> (even if I try to set it to <code>True</code> afterward) while others are okay. It is quite mysterious. Does anyone know the potential causes? Thanks.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a> et al.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7ef026033cc6e57811cf5e0a119e7729ee802214.png" alt="image" data-base62-sha1="i6WFYwPQ0EKEgDKBv29W37fYuQk" width="623" height="151"></p>

username: Glory_Chen
cooked: <p>More mysterious, when I change the name from <code>x</code> to <code>abc</code>, the problem is solved…</p>

username: ptrblck
cooked: <p>I get the expected results running your code, but are you able to reproduce the issue by running your code in a new script file?</p>

username: Glory_Chen
cooked: <p>I insert the snippet at the beginning of my project, and only using the name <code>x</code> will get the problem. Sorry. I may not be able to compress it for reproducibility. But do you have any insights on it?</p>
Issue with installation pytorch 1.5.0/1.6.0

username: Techniche_India
cooked: <p>i want to install pytorch 1.5.0 or 1.6.0 for making engine with <a href="https://github.com/Tianxiaomo/pytorch-YOLOv4" rel="noopener nofollow ugc">pytorch_yolo</a>.</p>
<p>Please help me with installation</p>

username: ptrblck
cooked: <p>You can check <a href="https://pytorch.org/get-started/previous-versions/">these install instructions</a> to get information about how to install older PyTorch releases.</p>

username: Techniche_India
cooked: <p>I have tried but i am getting error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/858be90ba1453bf74f5a592c10ae4abc5f29863f.png" data-download-href="https://discuss.pytorch.org/uploads/default/858be90ba1453bf74f5a592c10ae4abc5f29863f" title="Screenshot from 2022-03-25 12-39-10"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/858be90ba1453bf74f5a592c10ae4abc5f29863f_2_690x43.png" alt="Screenshot from 2022-03-25 12-39-10" data-base62-sha1="j3pg7JGAEjVgv0q9HQ09Lw473kj" width="690" height="43" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/858be90ba1453bf74f5a592c10ae4abc5f29863f_2_690x43.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/858be90ba1453bf74f5a592c10ae4abc5f29863f_2_1035x64.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/858be90ba1453bf74f5a592c10ae4abc5f29863f_2_1380x86.png 2x" data-dominant-color="370F27"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-03-25 12-39-10</span><span class="informations">1835×117 26.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please help me</p>

username: ptrblck
cooked: <p>The command is working for me:</p>
<pre><code class="lang-python">pip install torch==1.5.0 torchvision==0.6.0
Collecting torch==1.5.0
  Downloading torch-1.5.0-cp36-cp36m-manylinux1_x86_64.whl (752.0 MB)
     |████████████████████████████████| 752.0 MB 13 kB/s 
Collecting torchvision==0.6.0
  Downloading torchvision-0.6.0-cp36-cp36m-manylinux1_x86_64.whl (6.6 MB)
     |████████████████████████████████| 6.6 MB 7.1 MB/s 
Collecting numpy
  Downloading numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)
     |████████████████████████████████| 14.8 MB 21.2 MB/s 
Collecting future
  Using cached future-0.18.2.tar.gz (829 kB)
Collecting pillow&gt;=4.1.1
  Downloading Pillow-8.4.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 12.9 MB/s 
Building wheels for collected packages: future
  Building wheel for future (setup.py) ... done
</code></pre>
<p>so maybe your <code>pip</code> is too old? If so, try to download the wheel manually and install it locally.</p>
<p>Btw. I would generally recommend to use the latest stable or nightly release to get the latest features and bug fixes.</p>
Information about host self duration in torch profiler

username: gouthamvgk
cooked: <p>I am using torch.profiler.profile to benchmark my training. While exploring the operator wise runtime in tensorboard it seems that each operator spends a considerable amount of time in host compared to the device time. Attached is the screenshot of one such instance<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d690a634a2ca3e48c81d8ea37188b5ca848f22b.png" data-download-href="https://discuss.pytorch.org/uploads/default/2d690a634a2ca3e48c81d8ea37188b5ca848f22b" title="Screenshot 2022-03-25 at 12.33.05 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d690a634a2ca3e48c81d8ea37188b5ca848f22b_2_690x94.png" alt="Screenshot 2022-03-25 at 12.33.05 PM" data-base62-sha1="6tIx8SVauteoO9x6miKPr7zwUnF" width="690" height="94" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d690a634a2ca3e48c81d8ea37188b5ca848f22b_2_690x94.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d690a634a2ca3e48c81d8ea37188b5ca848f22b_2_1035x141.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d690a634a2ca3e48c81d8ea37188b5ca848f22b.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-03-25 at 12.33.05 PM</span><span class="informations">1109×152 11.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Here I am analyzing the Device self duration and Host Self Duration. It seems the operator is spending close to 30% of GPU time in the host (For conv_backward_weight GPU time is 105 ms while host time is 30 ms). This seems to be a considerable bottleneck for training. Can I know what kind of host operations are involved in aten::cudnn_convolution_backward_weight (or other similar operators) that leads to such high values of host time?. Also, does this host operation run in parallel to the GPU kernel execution, or do they run synchronously?</p>
Implementing Bayesian CNN

username: MERAH_Samia
cooked: <p>Hi everyone,<br>
I am trying to implement Bayesian CNN, and am searching for people who have worked on this topic, I really need some help (especially tutorials)?<br>
Thank you,</p>

username: Shisho_Sama
cooked: <p>Hi,<br>
whats your problem? have you searched the papers in this field?<br>
there was this repo that I had used a year ago : <a href="https://github.com/kumar-shridhar/PyTorch-BayesianCNN" rel="nofollow noopener">https://github.com/kumar-shridhar/PyTorch-BayesianCNN</a><br>
give that a try and it should get you going.<br>
Update :<br>
strongly recommend you to see : <a href="https://discuss.pytorch.org/t/implementing-a-bayesian-cnn-in-pytorch/13547/12" class="inline-onebox">Implementing a Bayesian CNN in PyTorch</a></p>

username: MERAH_Samia
cooked: <p>Hi,<br>
I found it complicated,I am searching for an approach to implement Bayesian Deep learning, i found two methods either by bayes by backprop or by dropout, I’ve read that Optimising any neural network with dropout is equivalent to a form of approximate Bayesian inference and a network trained with dropout already is a Bayesian neural network,<br>
Could you please confirm these statements if you have any idea ?</p>

username: blessen
cooked: <aside class="quote no-group" data-username="MERAH_Samia" data-post="3" data-topic="88689">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/merah_samia/48/53909_2.png" class="avatar"> MERAH_Samia:</div>
<blockquote>
<p>to a form of approximate Bayesian inference and a network trained with dropout already is a Bayesian neural network,</p>
</blockquote>
</aside>
<p>Hi, regarding your two statements -</p>
<ol>
<li>Yes as per Yarin Gal et al <a href="https://arxiv.org/abs/1506.02142" class="inline-onebox" rel="noopener nofollow ugc">[1506.02142] Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning</a>, you could interpret training a neural net with dropout as conducting Bayesian inference.</li>
<li>A Bayesian neural net is one that has a distribution over it’s parameters. Using dropout allows for the effective weights to appear as if sampled from a weight distribution. If you were to remove the dropout layer, then you’d have point estimates which would no longer correspond to a bayesian network.</li>
</ol>

username: MERAH_Samia
cooked: <p>Yes I see<br>
Thank you so much</p>

username: MERAH_Samia
cooked: <p>Okay, I see<br>
Thank you so much</p>
Slightly different inference results

username: Llama2
cooked: <p>I trained my model with pytorch 1.7 but recently upgraded to pytorch 1.11 and did an inference with the same code and exact same data. I get predictions that are off (+/-) by 0.02 for my binary classifier. Any reason why?</p>

username: Harish_S
cooked: <p>Please share the code</p>
Reproducing Pytorch Model Parameters

username: csl
cooked: <p>I’m pretty new to Pytorch but I was trying to run two consecutive runs of the same model by loading a checkpoint at epoch 0. Is it normal for the model parameter outputs to differ between these two runs? Is it possible the optimizer I am using (AdamW) may create a difference in parameters between iterations?</p>

username: Matias_Vasquez
cooked: <p>You can try setting a seed for the random parameters.</p>
<p>Many (if not all) layers rely on random initialization.</p>
<p>If you do</p>
<pre><code class="lang-python">torch.manual_seed(some_number)
</code></pre>
<p>Then it should be fine</p>
<p><a href="https://pytorch.org/docs/stable/notes/randomness.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/randomness.html</a></p>

username: csl
cooked: <p>Thanks for your reply- would you recommend doing this even with checkpoint loading enabled? I was under the impression that loading checkpoints would make sure the initial random initialization was the same for both runs.</p>

username: ptrblck
cooked: <p>I don’t know what you’ve stored in the “checkpoint”, but usually the <code>state_dict</code>s are stored only without any information about the seeds etc. In any case, you would still need to reseed the code as mentioned by <a class="mention" href="/u/matias_vasquez">@Matias_Vasquez</a> as loading the <code>state_dict</code>s alone won’t seed the code.</p>

username: csl
cooked: <p>Great, thanks to both of you for your input! I’ll try that out.</p>

username: csl
cooked: <p>It turns out a few of my model layers were non-deterministic but I was able to track them down using the line <code>torch.use_deterministic_algorithms(True)</code></p>
<p>Thanks <a class="mention" href="/u/matias_vasquez">@Matias_Vasquez</a> for the helpful link on reproducibility <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face.png?v=12" title=":smiling_face:" class="emoji" alt=":smiling_face:" loading="lazy" width="20" height="20"></p>
Why get_params() is a staticmethod in transforms?

username: kuzand
cooked: <p>I was checking the different transforms in <a href="https://github.com/pytorch/vision/blob/main/torchvision/transforms/transforms.py" class="inline-onebox" rel="noopener nofollow ugc">vision/transforms.py at main · pytorch/vision · GitHub</a><br>
in order to see how to implement my custom transform. The <code>get_params()</code> method of different transform classes are marked as <code>staticmethod</code>. What is the reason for this?</p>

username: ptrblck
cooked: <p>The <code>@staticmethod</code> decorator allows you to get the parameters without creating an object, which might be more convenient in some use cases. You could still call the <code>get_params</code> method on the object directly:</p>
<pre><code class="lang-python">transform = transforms.RandomCrop(size=(10, 10))

x = torch.randn(1, 3, 24, 24)
params = transforms.RandomCrop.get_params(x, (10, 10))
params = transform.get_params(x, (10, 10))
</code></pre>
How to add noise to audio using pytorchaudio if sample rates are not same?

username: ADONAI_TZEVAOT
cooked: <p>I have 2 audio files, Speech.wav and noise.wav. I want to merge the 2 audio files into one such that the noise.wav acts as a background noise for the speech audio. But the sample rates of both the files are different. the noise file has a sample rate of 8000 and the speech file has a sample rate of 48000. If I reduce the sample rate of speech file, its losing its quality. I want to add the noise to speech file without changing the speech file’s sample rate. How do I do it?</p>
<p>Please help</p>

username: ptrblck
cooked: <p>I think you could load the noise file and resample it via <code>librosa.resample</code> specifying the new sampling rate of 48k before blending these audio files.</p>
Loss function not implemented on pytorch

username: yuri
cooked: <p>I had to detach my model’s output to calculate the loss value. This is because the loss function is not implemented on PyTorch and therefore it accepts no tensors but  NumPy arrays instead.<br>
the problem now is that the loss value is detached from the computational graph and therefore no backpropagation could be performed.<br>
<strong>ps</strong>: the loss function is scipy.optimize.linear_sum_assignment</p>

username: ptrblck
cooked: <p>You could write a custom <code>autograd.Function</code> and implement the <code>backward</code> function in addition to the <code>forward</code> so that the computation graph wouldn’t break.<br>
<a href="https://pytorch.org/tutorials/beginner/examples_autograd/polynomial_custom_function.html">This tutorial</a> gives you an example.</p>
Is it possible to assign different numbers of workers across hosts?

username: zizhao.mo
cooked: <p>Hi,</p>
<p>I have two hosts where one has two GPUs and another one has only one GPU. I am wondering if torchrun can allow me to assign a different number of workers to hosts? Because according to the document, all the launching scripts are required to have the same nproc_per_node.</p>

username: fduwjj
cooked: <aside class="quote no-group" data-username="zizhao.mo" data-post="1" data-topic="147342">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/0ea827/48.png" class="avatar"> zizhao.mo:</div>
<blockquote>
<p>torchrun</p>
</blockquote>
</aside>
<p>Thanks for your question. I didn’t find it in the wiki. Maybe <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> can help answer this question?</p>

username: Kiuk_Chung
cooked: <p>Should work with torchrun:</p>
<pre><code class="lang-auto"># on host0 (2x gpus)
$ torchrun --rdzv_backend c10d --rdzv_endpoint $host0_hostname:$port --nnodes 2 --nproc_per_node 2 

# on host1 (1x gpu)
$ torchrun --rdzv_backend c10d --rdzv_endpoint $host0_hostname:$port --nnodes 2 --nproc_per_node 1 
</code></pre>
<p>Just make sure you set the cuda device using the <code>LOCAL_RANK</code> environment variable in your trainer</p>
<pre><code class="lang-auto">torch.cuda.set_device(int(os.environ["LOCAL_RANK"])
</code></pre>
<p>Also, side note, with heterogeneous nodes, performance could be bottlenecked by the node with fewer resources. I say “could” because it depends on various factors like the hardware topology, network topology, existence of RDMA, collective operation patterns etc. Typically we recommend homogeneous nodes for DDP.</p>
Dist.all_gather_object hangs

username: zhangyuygss
cooked: <p>I’m using dist.all_gather_object (PyTorch version 1.8) to collect sample ids from all GPUs:</p>
<pre><code class="lang-auto">for batch in dataloader:
    video_sns = batch["video_ids"]
    logits = model(batch)
    group_gather_vdnames = [None for _ in range(envs['nGPU'])]
    group_gather_logits = [torch.zeros_like(logits) for _ in range(envs['nGPU'])]
    dist.all_gather(group_gather_logits, logits)
    dist.all_gather_object(group_gather_vdnames, video_sns)
</code></pre>
<p>The line <code>dist.all_gather(group_gather_logits, logits)</code> works properly,<br>
but program hangs at line <code>dist.all_gather_object(group_gather_vdnames, video_sns)</code>.</p>
<p>I wonder why the program hangs at <code>dist.all_gather_object()</code>, how can I fix it ?</p>
<p>EXTRA INFO:<br>
I run my ddp code on a local machine with multiple GPUs. The start script is:</p>
<pre><code class="lang-auto">export NUM_NODES=1
export NUM_GPUS_PER_NODE=2
export NODE_RANK=0
export WORLD_SIZE=$(($NUM_NODES * $NUM_GPUS_PER_NODE))

python -m torch.distributed.launch \
       --nproc_per_node=$NUM_GPUS_PER_NODE \
       --nnodes=$NUM_NODES \
       --node_rank $NODE_RANK \
       main.py \
       --my_args
</code></pre>

username: zhangyuygss
cooked: <p>Turns out we need to set the device id manually as mentioned in the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather_object" rel="noopener nofollow ugc">docstring</a> of <code>dist.all_gather_object()</code> API.</p>
<p>Adding</p>
<pre><code class="lang-auto">torch.cuda.set_device(envs['LRANK']) # my local gpu_id
</code></pre>
<p>and the codes work.</p>
<p>I always thought the GPU ID is set automatically by PyTorch dist, turns out it’s not.</p>

username: fduwjj
cooked: <p>Thanks for you question, glad to know you solved it yourself. And I guess we do expect users to set cuda device manually. Will confirm this and get back to you.</p>
Custom dataset has no attribute "data"

username: Ashley_Gao
cooked: <p>Hello,</p>
<p>I have created a custom dataset using the ImageFolder:</p>
<pre><code class="lang-auto">dataset_fake = datasets.ImageFolder(root=dataroot_fake,
                           transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))
</code></pre>
<p>And when I try to access its data field by doing</p>
<pre><code class="lang-auto">dataset_fake.data
</code></pre>
<p>It gives me the error</p>
<pre><code class="lang-auto">AttributeError:
'ImageFolder' object has no attribute 'data'
</code></pre>
<p>Any help would be appreciated. Thank you so much!</p>

username: Matias_Vasquez
cooked: <p>If you want to access a specific image you can do it so</p>
<pre><code class="lang-python">img, lbl = dataset_fake[position_you_want]
</code></pre>
<p>If you only want to print the summary then you can use</p>
<pre><code class="lang-python">print(dataset_fake)
</code></pre>
<p>Or if you want the name of the image you are using you can do something like</p>
<pre><code class="lang-python">print(dataset_fake.samples)
</code></pre>

username: Ashley_Gao
cooked: <p>Thank you very much! This solved my problem!</p>
Adding a pre-trainned weights to some channels of a CNN while keeping it frozen and learning on rest of the channels

username: Shaktiman2051
cooked: <p>Hi, I have trained a Unet for denoising application. I wanted to get the residual noise in the denoised image. For this purpose, I wanted to add few trainable channels in the upsampling CNN’s of the trained Unet keeping the weight frozen. As far as my understanding, we can make a whole layer frozen but how to keep few channels frozen and let the other channels of the same layer have gradients. I am also not sure if I have to define a big CNN which can be filled with both pre-trained weights and new weights or I have to concatenate these channels in forward() function.</p>

username: Matias_Vasquez
cooked: <p>You can do something like this</p>
<pre><code class="lang-python">for param in model.parameters():
    param.requires_grad = False
</code></pre>
<p>You would only set it to true for the layers where you actually want to train.</p>
Super slow when Initialize MultivariateNormal with covariance matrix

username: potatolige
cooked: <p>Hi,</p>
<p>I am new to pytorch and want to sample batch data from MultivariantNormal distribution because I would like to predict the correlation between my out put.</p>
<p>Previously I use the scale_tril (Cholesky decomposition) to generate the MultivariantNormal and worked quite good. However now I could not directly form up the Cholesky decomposition from my algorithm but instead get the full covariance matrix. I feed it to MultivariantNormal but the initialization of this MultivariantNormal would cause a super slow computation speed (because it has to firstly compute the Cholesky Decomposition from full covariance matrix).</p>
<p>So my question is how can I overcome this computation catastropy?</p>
<p>Thank you in advance,</p>
<p>Best,<br>
Bruce.</p>

username: potatolige
cooked: <p>Soluion: turn the validation term to False will get avoid the eigen value positive checking, and will speed up the computation (x100)…</p>
Can I use weighted sample based dataloader for performing regression tasks

username: sparshgarg23
cooked: <p>I have used weighted sample dataloader for performing classification task where the objective of the model is to determine which class does the image belong.</p>
<p>I also have another model,which given an image predicts it’s age weight and body tone.The dataloader for this model uses random sampling.Compared to the previous model,its performance isn’t that great,so I was wondering can I use weighted sample based dataloder to perform a regression task,and if yes then what changes would I have to make in my following code shown below.</p>
<pre><code class="lang-auto">Classification task
Weighted dataloader sampling for my dataset
contains apprx 5000 images belonging to  8 classes

def obtain_class_weights(img_dataset):
  target_list=torch.tensor(img_dataset.targets)
  class_count=[i for i in get_class_distribution(img_dataset).values()]
  class_weights=1./torch.tensor(class_count,dtype=torch.float)
  print(class_weights)
  class_weights_all=class_weights[target_list]
  return class_weights_all
train_class_weights=obtain_class_weights(train_dataset)
train_weighted_sampler = WeightedRandomSampler(
    weights=train_class_weights,
    num_samples=len(train_class_weights),
    replacement=True
)
trainloader=DataLoader(train_dataset,batch_size=16,shuffle=False,sampler=train_weighted_sampler,drop_last=True)
Each batch consists of image and its label which belongs to one of the eight classes
</code></pre>
<p>Data loader used for regression purpose.</p>
<pre><code class="lang-auto">train_indices,val_indices=indices[split:],indices[:split]
train_sampler=SubsetRandomSampler(train_indices)
train_loader=DataLoader(dataset,batch_size=16,sampler=train_sampler,num_workers=1)
here each batch consists of image and a label of dimension 3x1
</code></pre>

username: suraj.pt
cooked: <p>Hi <a class="mention" href="/u/sparshgarg23">@sparshgarg23</a>  that’s an interesting question. In classification, weighted sampling helps normalize overrepresentation of discrete classes; not sure how that strategy works for regression where the targets are continuous.</p>
<p>Maybe you can try using the inverse of the target density instead of counts?</p>

username: sparshgarg23
cooked: <p>thanks any tips on how to obtain the target density?<br>
I found this article about transformed regressors in sklearn,but I am not sure on how to integrate it with pytorch.Any tips or suggestions would be welcome.</p>

username: suraj.pt
cooked: <p>I am not an expert, but you could try KernelDensityEstimator in sklearn. AFAIK you can fit it to your targets, get densities for your target domain and use their inverse to weight your samples. All this logic can be added in your <code>obtain_class_weights()</code>. Hope this helps!</p>
Sparse mean pooling while ignoring padded values

username: matthewleigh
cooked: <p>Hi everyone,</p>
<p>I am using a torch.sparse_coo_tensor and I was hoping to do mean reduction along a dimension where the zeroed values are ignored.</p>
<p>The sparse library supports summation and even softmax, but no way to do mean.</p>
<p>Is there a simple trick to getting the mean to work with a sparse tensor using the index values?</p>
<p>Thanks,<br>
Matt</p>
Analysis of gradient transfer overhead in distributed training

username: zizhao.mo
cooked: <p>Hi,</p>
<p>Is there any method to analyze the transfer process of distributed training in Pytorch? I want to predict the networking overhead for each mini-batch, given the known bandwidth.</p>
<p>Thanks</p>

username: wanchaol
cooked: <p>Hi <a class="mention" href="/u/zizhao.mo">@zizhao.mo</a> thanks for posting!</p>
<p>Since PyTorch 1.9, we released PyTorch profiler with distributed training view support, which could help you analyze the time/memory consumed by the training job, you can refer to this section <a href="https://pytorch.org/blog/pytorch-profiler-1.9-released/#distributed-training" class="inline-onebox" rel="noopener nofollow ugc">What’s New in PyTorch Profiler 1.9? | PyTorch</a></p>

username: rvarm1
cooked: <p>As <a class="mention" href="/u/wanchaol">@wanchaol</a> mentioned, <a href="https://pytorch.org/blog/pytorch-profiler-1.9-released/#distributed-training" class="inline-onebox" rel="noopener nofollow ugc">What’s New in PyTorch Profiler 1.9? | PyTorch</a> will give you insights regarding synchronization/communication overview.</p>
<p>Although, could you describe your use case a little more? What specifically are you predicting with regard to “networking overhead”, do you mean per-worker communication latency?</p>

username: zizhao.mo
cooked: <p>Thank you very much.</p>

username: zizhao.mo
cooked: <p>Hi Rohan,</p>
<p>Thanks for your kindly response. The networking overhead I mentioned is the total time spent on aggregating the gradient. Actually, I am writing a scheduler on top of PyTorch, to achieve the best efficiency (execute as many epochs as possible per unit time). In this case, I can determine how many workers I should run (maybe cross hosts). So profiling the gradient aggregation overhead is important for me, since this can help me determine if I should create a new worker or not for jobs.</p>
Quantize the CRNN model

username: huoge
cooked: <p>I read the quantization paper in pytorch website and realize the post dynamic quantization and static quantization.dynamic quantization is good for LSTM and Linear, and static quantization is good for CNNs, I wanna ask: when I use the CRNN model，the model  is like: CNN + LSTM + Linear, what is the best way to quantize my model, or is there some tricks to mix the two quantization methods?</p>
<p>I’d appreciate if anybody can help me! Thanks in advance!</p>

username: hx89
cooked: <p>I think it’s possible, you may apply static quantization to the CNN part of the model and dynamic quantization on LSTM + Linear part of the model, since both of them will have float data in the input and output, the combined model should work.</p>

username: wizardk
cooked: <p>1.fix rnn and linear layers, quantize cnn layers (post-training static quantization)<br>
2.fix rnn and linear layers, quantize cnn layers (quantization-aware training, this step is optional)<br>
3.fix quantized cnn layers, quantize rnn and linear layers(post-training dynamic quantization)</p>

username: THU-cui
cooked: <p>Hi, how to fix rnn and linear layers when quantize cnn layers?</p>

username: hx89
cooked: <p>Quantization is controlled by the qconfig, so when quantize cnn layers you can remove the qconfig of rnn layer, this way rnn layer will not be quantized.</p>

username: THU-cui
cooked: <p>Thk, I will try it soon!</p>

username: weilu_yang
cooked: <p>do you a demo on how to quantize different layers at different time? thanks!</p>

username: supriyar
cooked: <p>You can control the layers to quantize by specifying quant/dequant stubs around the layers. For more details you can refer to the tutorial in <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>

username: papcharis
cooked: <p>I am trying to do the exact same thing. Did you figure out how to do the quantization in a CNN-LSTM hybrid model?</p>
Small Learning rate changes considerable PSNR-SSIM value

username: cbd
cooked: <p>I run the code for 40 epochs. For 20 epochs, learning rate is constant and it linearly decrease for remaining epochs. During the linearly decrease mode, even if lr is in range of 10^-5, there is considerable change in validation image PSNR value around <strong>1</strong> during the last epochs. I think for small value of lr, PSNR should converge to near value of the previous epoch PSNR value.</p>
Error: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

username: rishabh10gpt
cooked: <p>My model code:</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        self.resnet34 = models.resnet34(pretrained=True)
        self.fc = nn.Sequential(
            nn.Linear(2000, 1024),
            nn.Linear(1024, 1),
            nn.Sigmoid()
        )
    def forward(self, x1, x2):
        x1 = self.resnet34(x1)
        x2 = self.resnet34(x2)
        output = torch.cat((x1,x2),1)
        output = self.fc(output)
        return output
</code></pre>
<p>‘’’<br>
Please help me solving this issue.</p>

username: ptrblck
cooked: <p>Your model itself is working fine as seen here:</p>
<pre><code class="lang-python">model = Classifier()
x1 = torch.randn(1, 3, 224, 224)
x2 = torch.randn(1, 3, 224, 224)
out = model(x1, x2)
out.mean().backward()
</code></pre>
<p>Could you post a minimal, executable code snippet which would reproduce the issue or check what the difference between my working example and your code could be?</p>

username: rishabh10gpt
cooked: <p>Thanks for the reply <a class="mention" href="/u/ptrblck">@ptrblck</a>  i was taking <code>torch.max(outputs.data,1)</code> before passing the output to calculate the loss. Now it is resolved.</p>
What's the proper order of "model.to(device)" and "optim(model.parameters())", thanks

username: ingYe_Laka
cooked: <p>what’s the proper order of “model.to(device)” and “optim(model.parameters())”, thanks</p>

username: suraj.pt
cooked: <ol>
<li><code>model.to('cuda')</code></li>
<li><code>opt(model.parameters(), ...)</code></li>
</ol>
<p>From the <a href="https://pytorch.org/docs/stable/optim.html">docs</a>:</p>
<blockquote>
<p>If you need to move a model to GPU via <code>.cuda()</code>, please do so before constructing optimizers for it. Parameters of a model after <code>.cuda()</code> will be different objects with those before the call.</p>
</blockquote>

username: ingYe_Laka
cooked: <p>Thanks! I also want to know what the consequences will be if I don’t do so?</p>

username: ptrblck
cooked: <p>In the worst case the optimizer might not update the passed parameters, so I would recommend to stick to the docs.</p>
Large memory cost of grouped matrix product for ndim > 3

username: mgreenig
cooked: <p>I’ve been seeing some strange behaviour on the GPU with <code>torch.matmul</code> for arguments with more than three dimensions. The following throws an out-of-memory error for an attempted allocation of 372 Gb (!).</p>
<pre><code class="lang-auto">y = torch.randn(1000000, 5, 1, 100).to(torch.device('cuda'))
x = torch.randn(5, 100, 200).to(torch.device('cuda'))
# out-of-memory
z = torch.matmul(y, x) 
# reducing number of dimensions solves the problem
z2 = torch.matmul(y.reshape((1000000, 1, 500)), x.reshape((500, 200))) 
</code></pre>
<p>Note I am on quite a large GPU (48Gb) so perhaps z2 might throw OOM for some people - if so, maybe trying reducing the first dimension of y.</p>
<p>Is this expected behaviour? I am trying to build a multi-head network so having that extra “head” dimension of 5 is helpful.</p>

username: KFrank
cooked: <p>Hi Mgreenig!</p>
<aside class="quote no-group" data-username="mgreenig" data-post="1" data-topic="147257" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mgreenig/48/47875_2.png" class="avatar"> mgreenig:</div>
<blockquote>
<p>The following throws an out-of-memory error for an attempted allocation of 372 Gb (!).</p>
</blockquote>
</aside>
<p>“372 Gb” does seem to be unreasonably large.</p>
<p>But leaving that aside …</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">z = torch.matmul(y, x) 
# reducing number of dimensions solves the problem
z2 = torch.matmul(y.reshape((1000000, 1, 500)), x.reshape((500, 200))) 
</code></pre>
<p>Note I am on quite a large GPU (48Gb)</p>
</blockquote>
</aside>
<p>The shape of <code>z</code> will be <code>[1000000, 5, 1, 200]</code>.  So you do need gigabytes<br>
of memory just for <code>z</code>.  Depending what else is already on your gpu, this<br>
could reasonably push you over the edge and cause an out-of-memory<br>
error, even with 48 GB.</p>
<p>Furthermore, the shape of <code>z2</code> will be <code>[1000000, 1, 200]</code>, so one-fifth<br>
the size of <code>z</code>.  That could conceivably be enough less to not cause the<br>
out-of-memory error.</p>
<p>Best.</p>
<p>K. Frank</p>

username: ptrblck
cooked: <p>The “372GB” sound indeed too large so are you sure it’s not 3.72GB which is the expected memory requirement for the output?</p>
<pre><code class="lang-python">(1000000 * 5 * 1 * 200 * 4) / 1024**3
# 3.725
</code></pre>
Resnet transfer learning on sequences of data

username: Brennan
cooked: <p>Hello!</p>
<p>I am trying to use the pretrained model resnet to perform classification on sequences of data. This sequence of data is a mel spectrogram of an audio waveform over time (speech).</p>
<p>It is important that I preserve the shape of the output from the model so that I can apply CTC loss to the predictions: <a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CTCLoss — PyTorch 1.11.0 documentation</a></p>
<p>When passing the spectrogram through the model the tensor returned is of shape [batch_size, num_classes] but I need it to return [batch_size, input_length, num_classes]</p>
<p>I current have a setup like the following:</p>
<pre><code class="lang-auto">        # 1 channel input, 3 channel output
        self.conv1 = nn.Conv2d(1, 3, 2)

        backbone = models.resnet50(pretrained=True)
        num_filters = backbone.fc.in_features
        layers = list(backbone.children())[:-1]
        self.feature_extractor = nn.Sequential(*layers)

        num_target_classes = 28
        self.classifier = nn.Linear(num_filters, num_target_classes)
</code></pre>
<p>thanks!</p>

username: ptrblck
cooked: <p>In the standard ResNet the output activation of the adaptive pooling layer is flattened in the <code>forward</code> method into a 2-dimensional tensor and passed to the linear layer. It should be possible to create a 3-dimensional tensor and pass it to <code>self.fc</code>.</p>
Why sometimes the value of loss rises first and then falls? It confuse me

username: ingYe_Laka
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a75c294094fb03a85d6d0360af236fc18e7063d.png" alt="截屏2022-03-24 15.06.55" data-base62-sha1="cUfgS47T4lFYeRna0gUX0vqmaHr" width="686" height="490"></p>
Leaf variable has been moved into the graph interior: extending mse_loss with a gradient penalty

username: tmaric
cooked: <p>I am trying to extend the <code>mse_loss</code> with an <code>mse_loss</code> with respect to a gradient penalty.</p>
<pre><code class="lang-auto">    // - Initialize training data 
    trainingPoints = torch::zeros({static_cast&lt;long&gt;(trainingLabels.size()),3},
                                   torch::requires_grad());
    trainingValues = torch::zeros({static_cast&lt;long&gt;(trainingLabels.size()),1});

    // Data initialization ...

    torch::Tensor trainingPrediction = torch::zeros_like(
        trainingValues, 
        torch::requires_grad()
    );
    torch::Tensor trainingMse = torch::zeros_like(trainingValues);

    // Solver setup ...

    for (; epoch &lt;= maxIterations; ++epoch) 
    {
        // Training
        optimizer.zero_grad();

        trainingPrediction = nn-&gt;forward(trainingPoints);
        auto predGrad = torch::autograd::grad(
           {trainingPrediction},
           {trainingPoints},
           {torch::ones_like(trainingValues)},
           true
        );
        trainingMse = mse_loss(trainingPrediction, trainingValues);
        // TODO: replace the current mse_loss with something like this, but adapt the weights
        /*trainingMse = 0.5*mse_loss(trainingPrediction, trainingValues)*/
         /*+ 0.5*mse_loss(at::norm(predGrad[0]), torch::ones_like(trainingValues)); */

        trainingMse.backward(); 
        optimizer.step();
</code></pre>
<p>This gives me an error:</p>
<blockquote>
<p>terminate called after throwing an instance of ‘std::logic_error’<br>
what():  leaf variable has been moved into the graph interior</p>
</blockquote>
<p>in <code>trainingMse.backward()</code>. I tried stepping through in <code>gdb</code> after adding a breakpoint there, this was a dead end.</p>
<p>The <code>trainingMse</code> is a leaf variable, does <code>trainingMse.backward()</code> represent an in-place change of <code>trainingMse</code> as described <a href="https://discuss.pytorch.org/t/leaf-variable-was-used-in-an-inplace-operation/308/2">in this post?</a></p>
<p>How to solve this?</p>
Expand a tensor with zeros

username: glennres
cooked: <p>Given an image tensor with a shape of:<br>
<code>(1,3,640,480)</code></p>
<p>I want to expand the image tensor to a shape of:<br>
<code>(1,3,640,640)</code></p>
<p>I want to fill the newly added space with <code>zeroes</code>.</p>
<p>Here’s an example of the desired result:</p>
<pre><code class="lang-python"># Given tensor shape (2,2)
&gt;&gt;&gt; img = torch.arange(100, 104).view(2,2)
tensor([[100, 101],
        [102, 103]])

&gt;&gt;&gt; dst = torch.zeros(3,3).long()
tensor([[0, 0, 0],
        [0, 0, 0],
        [0, 0, 0]])

# Desired result...
# Make image shape (3,3)
&gt;&gt;&gt; dst.put_(torch.tensor((0,1,3,4)), img)
tensor([[100, 101,   0],
        [102, 103,   0],
        [  0,   0,   0]])   # &lt; zeros
#                    ^ zeros

# Notice the image is now expanded 
# with zeros along the right and bottom.
</code></pre>
<p>How do I expand an image shape <code>(1,3,640,480)</code> into a shape <code>(1,3,640,640)</code>?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
<hr>
<h1>
<a name="solution-1" class="anchor" href="#solution-1"></a>Solution:</h1>
<pre><code class="lang-python">dst[..., :640, :480] = img
</code></pre>
<hr>
<h1>
<a name="new-problem-2" class="anchor" href="#new-problem-2"></a>New Problem:</h1>
<p>I ran the above expression through <code>torch.jit.trace</code> and discovered that tracing fails. Is there a torch function that can achieve the desired result (expand zeros on a tensor) while complying with <code>torch.jit.trace</code>?</p>
Create a custom predict function

username: Anime_Drive
cooked: <p>Hello,<br>
I am training a model using pytorch and i want to use the weights learned to create a custom predict function. I am using a custom module for matrix multiplication i want to test the module for prediction, the module does not support tensors, so can is it possble to get the weights as numpy arrays. ?</p>

username: ptrblck
cooked: <p>Yes, you can access the <code>.weight</code> attribute and transform them to numpy arrays e.g. via:</p>
<pre><code class="lang-python">arr = model.layer.weight.cpu().numpy()
</code></pre>
Unroll for loops in forward pass for GPU

username: luca7
cooked: <p>I have an input tensor of size (batch_size, X, Y) and need to pass it though the forward step of my custom model.<br>
At high level in the forward step:<br>
I loop over each batch and send the inner tensor of shape (X, Y) to another model that gives me something of shape (X,Z).<br>
Then I need to do the average over X and assign this result to each batch to get a final tensor of shape (batch_size, Z).</p>
<p>I can do it with for-loop, but I think it might be inefficient to run on a GPU (also, I am getting some CUDA error right now). A naive-example to better explain the problem.</p>
<pre><code class="lang-auto">def forward(self, in_tensor)
    """ in_tensor something like
    in_tensor = tensor([[[ 1.,  2.,  3.],
                                    [ 4.,  5.,  6.]],

                                    [[ 1.,  2.,  3.],
                                    [ 4.,  5.,  6.]]], dtype=torch.float) # shape [2,2,3]
    """
    res = []
    for batch, el in enumerate(in_tensor):
        # el has shape [2,3]
        tmp_res = other_model(el) # other model return something with 10 features for each input
        # tmp res has shape (2, 10)
        res.append(torch.mean(tmp_res, dim=0))
    res = torch.stack(res, dim=0)
    ...
</code></pre>
<p>I am wondering if there is a way to avoid the for-loop.<br>
Thanks</p>

username: beels
cooked: <p>Did you ever figure out a way? I was looking into something like this and found TorchScript. Maybe putting it through the JIT will let it unroll the for loop, but I have not tested it at all and am just proposing a solution because I am in the same spot.</p>
<p>If you found a solution without using JIT, would very much like to know!</p>
Focal loss NLP/text data pytorch - improving results

username: ni_tempe
cooked: <p>I have a NLP/text data classification problem where there is a very skewed distribution - <code>class 0 - 98%, class 1 - 2%</code><br>
For my training and validation data I am doing oversampling and my class distribution is <code>class 0 - 55%, class 1 - 45%</code>.<br>
The test data has skewed distribution</p>
<p>i built a model using <code>nn.BCEWithLogitsLoss(pos_weight=tensor(1.2579, device='cuda:0'))</code> . <code>pos_weight</code> was calculated using <code>55/45</code> (class distribution in training data.)</p>
<p>and on my class 1 of test data I got <code>f1</code> performance of <code>0.07</code>,<br>
<code>true negatives, false positives, false negative, true positive = (28809, 13258, 537, 495)</code></p>
<p>I changed to focal loss using below code and my performance didnt improve a lot. <code>f1</code> on class 1 of test data is still same and<br>
<code>true negatives, false positives, false negative, true positive = (32527, 9540, 640, 392)</code></p>
<p><code>kornia.losses.binary_focal_loss_with_logits(probssss, labelsss,alpha=0.25,gamma=2.0,reduction='mean')</code></p>
<ol>
<li>are my alpha and gamma parameters wrong? Are there any specific values that I should try? I could try to tune them but it might take a lot of time and resources. therefore I am looking for recommendations</li>
<li>for my <code>nn.BCEWithLogitsLoss(pos_weight=tensor(1.2579, device='cuda:0'))</code> should I use any other value for <code>pos_weight</code>? Please remember that my goal is to get maximum <code>f1</code> performance for test data class 1</li>
</ol>
Choosing a formula for calculating Dice metrics

username: grivkin00
cooked: <p>I am solving the problem of binary segmentation of medical images and plan to use Dice as a quality metric. The problem is this: different articles use two different formulas to calculate this ratio.<br>
This one:</p>
<pre><code class="lang-auto">class DiceMetric1(nn.Module):
    def __init__(self, weight=None, size_average=True):
        super(DiceMetric1, self).__init__()

    def forward(self, inputs, targets, smooth=1e-6):
    
        inputs = inputs.view(-1)
        targets = targets.view(-1)
    
        intersection = (inputs * targets).sum()                            
        dice = (2.*intersection + smooth)/(torch.pow(inputs, 2).sum() + torch.pow(targets,2).sum() + smooth)  
    
       return dice
</code></pre>
<p>Or like this:</p>
<pre><code class="lang-auto">class DiceMetric2(nn.Module):
    def __init__(self, weight=None, size_average=True):
        super(DiceMetric2, self).__init__()

    def forward(self, inputs, targets, smooth=1e-6):
    
        inputs = inputs.view(-1)
        targets = targets.view(-1)
    
        intersection = (inputs * targets).sum()                            
        dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)  
    
        return dice
</code></pre>
<p>Experiments have shown that the value calculated by the first formula is usually greater than by the second. Which one should be used?</p>
<p>The second question is that in my dataset there are images in the mask of which there are no positive values ​​at all, but only the background (in the mask it corresponds to the value 0). For such images, the metric value is displayed very small, even if the probabilities obtained by the network are close to zero, which does not look very logical, because the network correctly understood that the background is in the picture. But this problem is solved by setting the value smooth = 1, which is the default value I saw in half of the implementations of Dice (in the other half, smooth = 1e-6). Which smooth should you use?</p>
<p>I’ve tried every possible combination, but I don’t know which one to choose.</p>