akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use VOCDetection in torchvision.datasets?

_moonlight: <p>Hi all, I’m new to object detection.<br>
Now I want to write a simple demo using <code>torchvision.models.detection.fasterrcnn_resnet50_fpn</code> on voc dataset, but I always get failure on building voc dataset, could anyone show me a simple pipeline?<br>
I found a <code>VOCDetection</code> class in <code>torchvision.datasets</code>, but it doesn’t work.</p>
Pytorch cpp module cannot include header files

Smith_David: <p>by vscode remote ssh, access pytorch Module.cpp but always a few include sqyiggles looks like</p>
<pre><code class="lang-auto">...
[{
	"resource": "/output/workspace/pytorch/torch/csrc/cuda/Module.cpp",
	"owner": "C/C++: IntelliSense",
	"code": "1696",
	"severity": 8,
	"message": "cannot open source file \"Aten/native/cudnn/Macros.h\"",
	"source": "C/C++",
	"startLineNumber": 5,
	"startColumn": 1,
	"endLineNumber": 5,
	"endColumn": 38
},{
	"resource": "/output/workspace/pytorch/torch/csrc/cuda/Module.cpp",
	"owner": "C/C++: IntelliSense",
	"code": "1696",
	"severity": 8,
	"message": "#include errors detected. Consider updating your compile_commands.json or includePath. Squiggles are disabled for this translation unit (/output/workspace/pytorch/torch/csrc/cuda/Module.cpp).",
	"source": "C/C++",
	"startLineNumber": 12,
	"startColumn": 1,
	"endLineNumber": 12,
	"endColumn": 35
},{
	"resource": "/output/workspace/pytorch/torch/csrc/cuda/Module.cpp",
	"owner": "C/C++: IntelliSense",
	"code": "1696",
	"severity": 8,
	"message": "cannot open source file \"Aten/cuda/CUDAContext.h\"",
	"source": "C/C++",
	"startLineNumber": 12,
	"startColumn": 1,
	"endLineNumber": 12,
	"endColumn": 35
},{
	"resource": "/output/workspace/pytorch/torch/csrc/cuda/Module.cpp",
	"owner": "C/C++: IntelliSense",
	"code": "1696",
	"severity": 8,
	"message": "cannot open source file \"Aten/cuda/CachingHostAllocator.h\"",
	"source": "C/C++",
	"startLineNumber": 13,
	"startColumn": 1,
	"endLineNumber": 13,
	"endColumn": 44
}]
...
</code></pre>
<p>a few addition on include path seems not work, any problems like setup or permission?</p>
Allocate GPU and CPU Memory When Load Model on CUDA

Kaan_Kucuk: <p>While installing pytorch models with the gpu option, I see about 4 GB usage in cpu ram. It already uses 2.5 GB from my gpu memory. I could never understand the reason. My codes and ram information are below.</p>
<pre><code class="lang-auto"># Import Libraries
import torch
import os
from torchvision.models.segmentation import deeplabv3_resnet50
from flask import Flask
import torch.backends.cudnn as cudnn


os.environ['LRU_CACHE_CAPACITY'] = '1'

# Initialize Flask
app = Flask(__name__)

device = 'cuda' if torch.cuda.is_available() else 'cpu'

def load_deskew_model(num_classes=2, device='cuda', model_path=None):
    model = deeplabv3_resnet50(num_classes=num_classes, aux_loss=True)
    model.to(device)
    checkpoint_path = os.path.join(os.getcwd(), model_path)
    checkpoints = torch.load(checkpoint_path, map_location=lambda storage, loc: storage.cuda(0))
    model.load_state_dict(checkpoints, strict=False)
    return model
</code></pre>
<p>±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 470.129.06   Driver Version: 470.129.06   CUDA Version: 11.4     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  NVIDIA GeForce …  Off  | 00000000:01:00.0 Off |                  N/A |<br>
| 33%   29C    P2    44W / 220W |   2242MiB /  7982MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
|    0   N/A  N/A      1087      G   /usr/lib/xorg/Xorg                  9MiB |<br>
|    0   N/A  N/A      1249      G   /usr/bin/gnome-shell                3MiB |<br>
|    0   N/A  N/A   3622956      C   python3                          2225MiB |<br>
±----------------------------------------------------------------------------+</p>
I am new in python can anyone help me how can I save the prediction mask as image I want to save y_batch_pred as image

Aya_Elfatyany: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c765d7b56879e29d2b2b9c941bff280ac8eec5b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/c765d7b56879e29d2b2b9c941bff280ac8eec5b5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c765d7b56879e29d2b2b9c941bff280ac8eec5b5.png" alt="image" data-base62-sha1="srXdelj9uZoiXaQpLMHOmIqmOVL" width="690" height="367" data-dominant-color="1E1F1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1031×549 23.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on your screenshot it seems you are using TensorFlow, not PyTorch, so I would recommend posting this question in the TensorFlow discussion board.</p>

Aya_Elfatyany: <p>yeah I am using tensorflow ,Thanks for replying</p>
Zeroing out gradient in word embeddings tutorial

codingGoose: <p>I noticed that in the <a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html" rel="noopener nofollow ugc">Word Embeddings tutorial</a>, the N-Gram  Language Modelling example zeroes out the gradient with the following line of code:</p>
<pre><code class="lang-auto"># Step 2. Recall that torch *accumulates* gradients. Before passing in a
# new instance, you need to zero out the gradients from the old
# instance
model.zero_grad()
</code></pre>
<p>My understanding is that this line should be <code>optimizer.zero_grad()</code>. Why is this case different?</p>
<p>Thanks</p>

ptrblck: <p><code>optimizer.zero_grad()</code> is equal to <code>model.zero_grad()</code> if all parameters were passed to the optimizer, which is also the case in this tutorial.</p>
Getting error in backward function

nkkaushal: <p>I am trying to implement <a href="https://github.com/VIPL-Audio-Visual-Speech-Understanding/LipNet-PyTorch/tree/master" rel="noopener nofollow ugc">LipNet</a> but after few iterations I am getting nan as my loss, I checked my input and output from the model but there isnt any nan value but after loss.backward() I am getting nan values in my gradient and assert statement is getting failed. Below is the code snippet, can someone help me out with this??</p>
<h4><a name="y-is-output-from-model-with-shape-batch_size-time_steps-number-of-classes-1" class="anchor" href="#y-is-output-from-model-with-shape-batch_size-time_steps-number-of-classes-1"></a>y is output from model with shape (Batch_size, time_steps, number of classes)</h4>
<p>loss = nn.CTCLoss(y.transpose(0, 1).log_softmax(-1), txt, vid_len.view(-1), txt_len.view(-1))<br>
loss.backward()<br>
nn.utils.clip_grad_norm_(net.parameters(), 1.0)</p>
<p>assert not any(torch.isnan(p.grad).any() for p in net.parameters()), “NaN values found in the gradients after the backward pass.”</p>

ptrblck: <p>Infs and NaNs in <code>CTCLoss</code> were discussed a few times already in other posts so take a look at e.g. <a href="https://discuss.pytorch.org/t/ctcloss-performance-of-pytorch-1-0-0/27524/11">this one</a> and check if your use case might run into the same issue.</p>
Translation keras to pytorch - resahping unmatched tensor

berlin: <p>I am trying to reproduce the <a href="https://github.com/NAMlab/DeepCRE/blob/main/src/python/train_low_high_expressed_multi_species_models.py" rel="noopener nofollow ugc">code</a> of keras to pytorch. However I am struggling to reproduce the result.</p>
<pre><code>from typing import List
class DNA_CNN_test2(nn.Module): # deepcre model  
    def __init__(self,
                 seq_len: int =1000,
                 kernel_size: int = 8,
                 p = 0.25): # drop out value 
        super().__init__()
        self.seq_len = seq_len
        # CNN module
        self.conv_net = nn.Sequential()
        self.model = nn.Sequential(
            nn.Conv1d(4,64,kernel_size=kernel_size, padding='same'),
            nn.ReLU(),
            nn.Conv1d(64,64,kernel_size=kernel_size, padding='same'),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=8),
            nn.Dropout(p),
            nn.Conv1d(64,128,kernel_size=kernel_size, padding='same'),
            nn.ReLU(),
            nn.Conv1d(128,128,kernel_size=kernel_size, padding='same'),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=8),
            nn.Dropout(p),
            nn.Conv1d(128,64,kernel_size=kernel_size, padding='same'),
            nn.ReLU(),
            nn.Conv1d(64,64,kernel_size=kernel_size, padding='same'),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=8),
            nn.Dropout(p),
            nn.Flatten(),
        
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Dropout(p),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 1)
        )#.to(device)
         
    def forward(self, xb: torch.Tensor):
        """Forward pass."""
        xb = xb.permute(0, 2, 1) 
        out = self.conv_net(xb)
        return out 
</code></pre>
<p>I am following all the order of the original code however the code give me an error, which I can locate. Here I am using 512 batch size. My input is one hot encoded DNA sequence (1000bp) and its  corresponding transcript value (numeric). What have I missed?  Meanwhile, I have used used to unsqueze() the matrix, but did not help till now.</p>
<pre><code>The size of tensor a (4) must match the size of tensor b (512) at non-singleton dimension 1
</code></pre>
Libtorch <torch/torch.h> not found

ali2210: <p><a href="https://drive.google.com/file/d/1kjSYlhF7d6TSoDf4WYGQyiWr8Pw0F-1e/view?usp=sharing" rel="noopener nofollow ugc">error</a>.</p>
<p>I’m working on rust bert module for personal application. I had read documentation and try how to configure libtorch in your machine. crates/rust-bert.<br>
Then a execute cargo run.<br>
So I searched over internet and I found few solutions . One is to generated cmake files in your libtorch and recompile.</p>
<p><a href="https://drive.google.com/file/d/1_jXkhAJBILE8BTKHp26BgJwgFpy6kIW8/view?usp=sharing" rel="noopener nofollow ugc">libtorch after adding cmake</a> . Still error persist . Kindly help me to resolve the error as soon as possible. Snapshots are shared</p>

PatrickV: <p>Be sure to include 2 directories in your include path.<br>
Those paths depend on where you downloaded Libtorch.<br>
For me, I downloaded the pytorch sources into “PytonSources”, so the paths I include are:</p>
<pre><code class="lang-auto">~/PythonSources/pytorch/torch/include
~/PythonSources/pytorch/torch/include/torch/csrc/api/include
</code></pre>
CelebA accuracy for multiclass classification

Shashank_Priyadarshi: <p>Hi,<br>
So I was doing multi-class classification in the CelebA dataset (40 classes/attributes) using the crossentropy loss using a batch size of 64. So my target is [64, 40] and my output is also [64, 40] (using the ConvNeXt model). So for calculating accuracy, I am doing something like -</p>
<pre><code class="lang-auto">for i, d in enumerate(output, 0):
    acc_ = pred_acc(torch.Tensor.cpu(targets[i]), torch.Tensor.cpu(d))
    acc.append(acc_)
class_acc = np.asarray(acc).mean()
</code></pre>
<p>and my pred_acc function is -</p>
<pre><code class="lang-auto">def pred_acc(original, predicted):
    return torch.round(predicted).eq(original).sum().numpy()/len(original)
</code></pre>
<p>Here the accuracy I am getting is pretty less, can anyone help to figure out what I am doing wrong (ig some issue in the pred function or the way I am passing the pds in them)</p>
Does nvidia driver version affect performance?

happy93: <p>I am havinvg identical set up (i.e. torch2.0.0+cu117) on google cloud and aws cloud.</p>
<p>The only difference is the nvidia driver version (google is 51x version, aws is still using 4xx version)</p>
<p>Google is running much faster than aws. Is it possible nvidia driver version affects performance?</p>

ptrblck: <p>It could bit the more likely root cause could be caused by another part of your setup, e.g. the data loading speed etc. You should profile the code to check where exactly the slowdown comes from.</p>
Why DPSGD use Gaussian noise instead of Laplace

lwx0420: <p>What’s the difference of Gaussian noise and Laplace noise under the context of DPSGD?</p>

Kai_Yao: <p>Probably take a look at <a href="https://desfontain.es/privacy/gaussian-noise.html" class="inline-onebox" rel="noopener nofollow ugc">The magic of Gaussian noise - Ted is writing things</a>. This explains why in real cases people use Gaussian for DP.</p>
What is the most efficient way to automatically support custom data type?

Koukyosyumei: <p>I’m currently trying to support Paillier Encryption as the custom type of PyTorch. I already have supported basic addition, multiplication, and nn.Linear module. The examples can be found at <a href="https://koukyosyumei.github.io/AIJack/notebooks/aijack_paillier.html" class="inline-onebox" rel="noopener nofollow ugc">10.1. Paillier Encryption — AIJack documentation</a>. The current implementation (<a href="https://github.com/Koukyosyumei/AIJack/blob/main/src/aijack/defense/paillier/torch_wrapper.py" rel="noopener nofollow ugc">https://github.com/Koukyosyumei/AIJack/blob/main/src/aijack/defense/paillier/torch_wrapper.py</a><br>
) wraps each operation one by one manually, but I want to automatically support as many operations as possible like conv2d without manual implementation. Any idea would be helpful. I really appreciate any help you can provide.</p>
CNN for sentences

sa2706: <p>This is my first time implementing a complete neural network myself and I am finding some difficulties.<br>
I am implementing a paper’s neural network that predicts a score of an essay.<br>
I finished the first part which is getting the essays embeddings.<br>
Now I have an essays input of shape (batch_size, max_sent, max_word, embedding_dim)<br>
max_sent: number of sentences in the longest essay<br>
max_word: number of words in the longest sentence</p>
<p>Now I need to pass each sentence to a 1d CNN layer<br>
then the output for each sentence is passed to its own attention pooling layer<br>
I am unsure how to reshape my input in order to perform this operation.</p>

vdw: <p>If made a <a href="https://discuss.pytorch.org/t/conv1d-layer-for-text-classification/184856/6">reply</a> earlier today that might help.</p>

sa2706: <p>Thanks but the architecture of the input I mentioned is different.</p>
Pytorch-Nightly Install Issue

Varun_Gumma: <p>I am trying to install the latest nightly version using the command <code>conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch-nightly -c nvidia</code>. I have an RTX 3060 GPU on my device, but conda only installs the cpu version instead of the cuda version. If I try to install any other version, say nightly cuda=11.8 version, it install the correct cuda-compiled pytorch version. Any leads on this?</p>
<p>Thanks.</p>
<p>PS. I am using Ubuntu 23.04 and Python 3.9</p>

InnovArul: <p>Try installing via <code>pip</code>:</p>
<pre><code class="lang-auto">pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118
</code></pre>
With Adam optimizer, is it necessary to use a learning scheduler?

Hong_Cheng: <p>Adam can adapt its learning rate by the gradient updating. I think we may not need the learning rate scheduler.<br>
However, I worry that if with that kind of learning rate scheduler in Adam can jump out of the local minimal or get away from the local minimal</p>
<p>In <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">transfer_learning_tutorial</a>, it use momentum SGD with a learning scheduler.</p>

Eta_C: <p>Yes I have had such experience.<br>
Now in my project, I split <code>num_epochs</code> into three parts.</p>
<ul>
<li><code>num_epochs_1</code> warm up.</li>
<li><code>num_epochs_2</code> Adam for speeding up covergence.</li>
<li><code>num_epochs_3</code> momentum SGD+CosScheduler for training.</li>
</ul>
<p>My friend used Adam without learning rate scheduler in his project, and he found that the loss started to rise after some epochs.</p>
<p>You can find some discuss <a href="https://stackoverflow.com/questions/39517431/should-we-do-learning-rate-decay-for-adam-optimizer" rel="noopener nofollow ugc">here</a>. Although Adam can adaptively adjust the learning rate, but such ability is limited.</p>
<p>At least, for me, I think momentum SGD is the most stable optimizer and Adam/AdamW is a good tick to speed up covergence.</p>
<p>All these are my personal experiences. Is it necessary to use a learning scheduler? Maybe as the answer in the link says,</p>
<blockquote>
<p>It depends.</p>
</blockquote>

Aldebaran: <p>Pytorch Adam algorithm implementation follows changes proposed in <a href="https://arxiv.org/abs/1711.05101" rel="noopener nofollow ugc">Decoupled Weight Decay Regularization</a> which states:</p>
<blockquote>
<p>Adam can substantially benefit from a scheduled learning rate multiplier. The fact that Adam<br>
is an adaptive gradient algorithm and as such adapts the learning rate for each parameter<br>
does not rule out the possibility to substantially improve its performance by using a global<br>
learning rate multiplier, scheduled, e.g., by cosine annealing.</p>
</blockquote>

TMK04: <p><a href="https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html" rel="noopener nofollow ugc">Pytorch AdamW* algorithm</a></p>
Two-dimensional MSE loss in MLP training

mariekiii: <p>Hello,</p>
<p>I want to train a multilayer perceptron that maps a multidimensional input to a two-dimensional target. The target consists of two different variables (gross primary productivity and evapotranspiration) that are also of very different magnitude: mean of the first is ~ 0.4, mean of the second ~ -22 . Doing so, I construct a fully-connected model with two output dimensions and linear activation, which I evaluate in the MSE-loss criterion against my batch sample with the same two output dimensions.</p>
<p>One variable is predicted pretty good - but the other is predicted completely wrong, following the shape of the first variable. I scaled the target in order to reduce the effect of the different units, but the results remain.<br>
The batch loss is small at first, and then increasing to a fixed loss - just the opposite as I expect it.</p>
<p>Is this approach with a two-dimensional target reasonable at all, will I have to use a different loss function or a completely different approach overall? Maybe just use two different MLPs, one for each target?</p>
<p>I am very happy about any comments or shared experiences on this! Thank you.</p>

ays: <p>Hi,<br>
please where you able to find a solution to this?</p>
Errors with libtorch (#include <torch/script.h>)

hedeya1980: <p>I’m trying to use ‘libtorch’ for the first time.<br>
I set the C/C++&gt;General&gt;‘Additional Include Directories’, as well as the linker’s ‘Additional Library Directories’ and ‘Additional Dependencies’.<br>
I just added <code>#include &lt;torch/script.h&gt;</code> to my .cpp file, and tried to build to make sure everything is ok, but I got the following errors:</p>
<pre><code class="lang-auto">Severity	Code	Description	Project	File	Line	Suppression State
Error	C2872	'nullopt': ambiguous symbol	Road_Defects_Detector	C:\libtorch-win-shared-with-deps-1.12.0+cu116\libtorch\include\ATen\DeviceGuard.h	26	

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2446	':': no conversion from 'const std::nullopt_t' to 'c10::optional&lt;c10::Device&gt;'	Road_Defects_Detector	C:\libtorch-win-shared-with-deps-1.12.0+cu116\libtorch\include\ATen\DeviceGuard.h	26	

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2872	'optional': ambiguous symbol	Road_Defects_Detector	C:\libtorch-win-shared-with-deps-1.12.0+cu116\libtorch\include\ATen\ops\from_blob.h	35	

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2872	'optional': ambiguous symbol	Road_Defects_Detector	C:\libtorch-win-shared-with-deps-1.12.0+cu116\libtorch\include\ATen\ops\from_blob.h	54	

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2872	'optional': ambiguous symbol	Road_Defects_Detector	C:\libtorch-win-shared-with-deps-1.12.0+cu116\libtorch\include\ATen\ops\from_blob.h	83	

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2872	'optional': ambiguous symbol	Road_Defects_Detector	C:\libtorch-win-shared-with-deps-1.12.0+cu116\libtorch\include\ATen\ops\from_blob.h	86	

Severity	Code	Description	Project	File	Line	Suppression State
Error	C2664	'at::TensorMaker &amp;at::TensorMaker::target_device(std::optional&lt;c10::Device&gt;) noexcept': cannot convert argument 1 from 'const c10::optional&lt;c10::Device&gt;' to 'std::optional&lt;c10::Device&gt;'	Road_Defects_Detector	C:\libtorch-win-shared-with-deps-1.12.0+cu116\libtorch\include\ATen\ops\from_blob.h	105	

</code></pre>

poirot0805: <p>Hi there, I solved this problem by adding “std::” in front of each “optional” and “nullopt”</p>
Architecture suggestions for learning image to image?

milan_kalkenings: <p>hi, i try to train a neural network to map inputs of shape [2, 257, 161] to outputs of shape [2, 73, 221]. (stft representation of audio data)</p>
<p>idea 1: encoder maps input to a flat tensor, reshape flat tensor to desired shape<br>
idea 2: encoder maps input to flat tensor, decoder (some trans convs) create output of shape [2, 128, 256], use output[:, :73, :221] as final prediction</p>
<p>both ideas result in roughly the same bad valdiation loss, results show that the network has -some- idea of the data but the results are far from good.</p>
<p>any ideas on better architectures for this usecase?</p>
How to know nproc_per_node in python code

guochengqian: <p>hi community,<br>
how could we know the nproc_per_node parameter in the code?</p>
<p>for example, running the following:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE
           --nnodes=2 --node_rank=0 --master_addr="192.168.1.1"
           --master_port=1234 YOUR_TRAINING_SCRIPT.py (--arg1 --arg2 --arg3
           and all other arguments of your training script)
</code></pre>
<p>how to know the nproc_per_node, master_addr in the YOUR_TRAINING_SCRIPT.py code?</p>
<p>Thank you!</p>

H-Huang: <p>Hello! You would need to duplicate them and pass those in as your own arguments. Example:</p>
<p>command (single node, 2 GPUs):</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=2 
train_script.py --master_addr=localhost --nproc_per_node=2
</code></pre>
<p>train_script.py:</p>
<pre><code class="lang-auto">import argparse
parser = argparse.ArgumentParser()
# This is always passed in by default
parser.add_argument("--local_rank", type=int)
# These are your own arguments
parser.add_argument("--master_addr", type=str)
parser.add_argument("--nproc_per_node", type=int)
args = parser.parse_args()
print(args)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">Namespace(local_rank=0, master_addr='localhost', nproc_per_node=2)
Namespace(local_rank=1, master_addr='localhost', nproc_per_node=2)
</code></pre>
<p>We would be interested in hearing why you need the nproc_per_node and master_addr in your training script, generally just the rank is sufficient?</p>

guochengqian: <p>Hi Howard,<br>
Thank you very much!<br>
I want to implement some new asynchronous averaging algorithms for Federated learning using PyTorch. But, I am also new to the area. Maybe my implementation is not optimal.</p>

H-Huang: <p>I see! That makes sense, thank you. The launcher is not absolutely necessary but could be useful, here is the source code (it’s short) to glean some insight into what it is doing <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/launch.py at master · pytorch/pytorch · GitHub</a></p>
<p>For your use case, I would recommend looking into the RPC framework <a href="https://pytorch.org/docs/stable/rpc.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed RPC Framework — PyTorch 1.8.1 documentation</a> if you haven’t already.</p>

guochengqian: <p>Yes. I have noticed this document. thank you very much?<br>
BTW, is Pytorch team working on a general federated learning framework that supports flexible control on each client (processors, GPUs) and the way of aggregating their gradients or model parameters.</p>

H-Huang: <p>To my knowledge, there isn’t a project for a general federated learning framework. Feel free to start a new thread regarding this as others may have insight, it will also be useful for feature tracking purposes.</p>

ndvbd: <p>Is there a way to specify the:<br>
-m torch.distributed.launch<br>
programmatically? otherwise how can we do a debug session inside pycharm?</p>
Scale Difference in Multi-Task Loss

Dhruv_Jain: <p>Hello,</p>
<p>I have a 3D-U-net network for tumor segmentation, and I would also like to output the volume of the tumor from the bottleneck of the U-net.<br>
My question:<br>
I have two losses - Dice Loss + MSE Loss</p>
<p>How can I have them in the same scale?</p>
<p>Thank You</p>

ays: <p>Please did you find the solution to this problem?</p>
Loading in Checkpoints for a model with some new additions

Zohaib_Khan: <p>I am trying to implement a paper where the authors have made some additions to the Encoder of a pretrained ViT: performing a convolution with a learnable filter on the attention maps.</p>
<p>According to them, they have used the official implementation (and made changes in this class: <a href="https://github.com/jeonsworld/ViT-pytorch/blob/main/models/modeling.py#L56C3-L56C3" rel="noopener nofollow ugc">https://github.com/jeonsworld/ViT-pytorch/blob/main/models/modeling.py#L56C3-L56C3</a>) - loading in the pretrained checkpoints for image classification from HuggingFace which means the untouched layers retain their weights, and the new additions (i.e. the Convolution here) have their weights randomly initialized.</p>
<p>How can I do this in PyTorch? What are the steps to accomplish this? If I use the original ViT repo and make the additions there, then download the checkpoint from HuggingFace, is it as simple as calling <code>load_state_dict()</code> or something similar (it’ll match the layers left alone automatically while ignoring the new additions)? If this is the case, is it as simple to do this for other problems: perhaps making some changes to BERT then loading in the pretrained ckpt from HF to test whether the finetuning process for a downstream task performs better?</p>
<p>I found another topic (<a href="https://discuss.pytorch.org/t/how-to-load-my-pre-trained-model-into-a-modified-new-one-model/22519" class="inline-onebox">How to load my pre-trained model into a modified new one model?</a>) that seems to ask a similar question, but I’m unsure if this is the same situation (changes vs. additions, and loading in from an official checkpoint).</p>
<p>Thank you!</p>

Zohaib_Khan: <p>(New users can’t share &gt;2 links in their first post so I’m adding this here)<br>
For more details on what I’m trying to add, here’s the rough idea in TensorFlow for another model by the same authors (its an augmented version of the Attention mechanism that I’m trying to bring in the Encoder block): <a href="https://github.com/pkuyym/EvolvingAttention/blob/276447b62223f643f9e54c7c2225a9a2e707ee6e/EAResnet50/official/resnet/resnet_model.py#L209-L220" rel="noopener nofollow ugc">https://github.com/pkuyym/EvolvingAttention/blob/276447b62223f643f9e54c7c2225a9a2e707ee6e/EAResnet50/official/resnet/resnet_model.py#L209-L220</a></p>
Usage guidelines for torch.cuda.utilization()

surya00060: <p>I came across this <a href="https://pytorch.org/docs/stable/generated/torch.cuda.utilization.html" rel="noopener nofollow ugc">torch.cuda.utilization()</a> function to measure the GPU utilization. What’s the sample period between each measurement? If i want to measure the utilization of specific two operation do i manually add time.sleep() to ensure that utilization doesn’t get affected?</p>
<pre><code class="lang-auto">torch.Op1()
torch.cuda.utilization(0) # To get GPU utilization of Op1
time.sleep(1)
torch.Op2()
torch.cuda.utilization(0) #To get GPU utilization of Op2
</code></pre>
<p>Does this work roughly?</p>

ptrblck: <p>Internally a <code>nvmlUtilization_t Struct</code> will be queried which will return the utilization rate by using sampling periods. From the docs:</p>
<blockquote>
<p>Each sample period may be between 1 second and 1/6 second, depending on the product being queried.</p>
</blockquote>
Hook RoIHeads layer

shayan55579: <p>Hi, I’ve been trying to hook in the Faster R-CNN ResNet50 pre-trained model in the RoIHeads layer.</p>
<p>First, I load the model and feed the image to the model. After that, I hook in the layer to access its tensors. My objective is to obtain the boxes with a confidence threshold of more than 50%.</p>
<p>My question is, did I write the correct code? If not, what should I do instead?</p>
<pre><code class="lang-auto">import torch
import torchvision
import pickle
from PIL import Image

# Define the model
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()

!wget -nv https://www.goodfreephotos.com/cache/other-photos/car-and-traffic-on-the-road-coming-towards-me.jpg -O traffic.jpg
image = Image.open('traffic.jpg')
image_tensor = torchvision.transforms.ToTensor()(image)
image_tensor = image_tensor.unsqueeze(0)

outputs = []
hook = model.backbone.register_forward_hook(lambda self, image_tensor, output: outputs.append(output))
res = model(image_tensor)
hook.remove()
selected_rois = model.roi_heads.box_roi_pool(outputs[0], [r['boxes'] for r in res], [i.shape[-2:] for i in image_tensor])
print(selected_rois.shape)

# Define the threshold value
threshold = 0.5

# Get the box features and scores
box_features = res[0]
scores = box_features.get('scores') # Assuming the last column contains the scores

# find indices of grather score
above_threshold_indices = torch.where(scores &gt; threshold)[0] #tensor([0, 1, 2, 3])

#get value of scores
above_threshold_scores = scores[above_threshold_indices] #tensor([0.9840, 0.9784, 0.9386, 0.8663]

#finde the features
above_threshold_box_features = selected_rois[above_threshold_indices]
above_threshold_box_features.shape

</code></pre>
Vectorized inference of multiple models

hadarshavit: <p>I have multiple MLPs with the same architecture but different weights.<br>
I want to get the result from all of those MLPs on the same data (tensor of size [B, N], B - number of samples, N - number of input features).<br>
How can I vectorized the inference on those models?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/hadarshavit">@hadarshavit</a>,</p>
<p>Have a look at the <code>torch.func</code> library, you can functionalize your model to take the params as an input. You can then vmap over them.</p>
<p>An example can be found on the forums here: <a href="https://discuss.pytorch.org/t/another-way-to-implement-maml-model-agnostic-meta-learning/164439" class="inline-onebox">Another way to implement MAML(Model-Agnostic Meta-Learning)?</a></p>
<p>Although, you’ll need to replace <code>functorch</code> with <code>torch.func</code> for pytorch2.0 and above.</p>
Share qparams in pt2e

plankp: <p>Hi, I’m following the <a href="https://pytorch.org/tutorials/prototype/quantization_in_pytorch_2_0_export_tutorial.html" rel="noopener nofollow ugc">QUANTIZATION IN PYTORCH 2.0 EXPORT TUTORIAL</a> guide and the xnnpack-quantizer in the repo to create a new Quantizer. I am trying to have both inputs of <code>add</code> share the same qparams, just like the sample code provided in the tutorial.</p>
<p>It works when I just have add nodes, but as soon as a convolution node appears, the two inputs no longer have the same qparams. Here’s a simple model that illustrates this issue:</p>
<pre><code class="lang-python">class Net(nn.Module):
  def __init__(self):
    super().__init__()
    self.conv = nn.Conv2d(1, 1, 3, padding=(1, 1))

  def forward(self, x):
    return x + self.conv(x)
</code></pre>
<p>Is there something that I am missing in my quantizer?<br>
link to the quantizer: <a href="https://gist.github.com/plankp/c606815948af43d5f5ecaae814fc11ca#file-sample_quantizer-py-L55" class="inline-onebox" rel="noopener nofollow ugc">Quantizer for PT2E quantization · GitHub</a></p>
<p>(In this example, it’s sufficient to swap the operands of the add node, but once both operands come from different convolutions, the problem reappears)</p>

supriyar: <p>thanks for reporting! FYI the PT2 export quantization is still in early prototype so there may be some cases that we may have missed or might need to update the tutorial to make it clearer.</p>
<p>cc <a class="mention" href="/u/jerryzh168">@jerryzh168</a> to take a look</p>

jerryzh168: <p>thanks for trying out our new APIs! this can be supported by the current API I think, yeah swapping operands seems reasonable in this case, talking about the case when there are two conv_ops</p>
<p>basically let’s say we have:</p>
<pre><code class="lang-auto"> [conv1_op] -&gt; conv1_output -&gt; add_input1 -&gt; [add_op]
 [conv2_op] -&gt; conv2_output -&gt; add_input2    /
</code></pre>
<p>so in total we have <code>conv1_output</code>, <code>add_input1</code>, <code>conv2_output</code> and <code>add_input2</code> that we can annotate, the reason when you have annotate both [conv1_op], [conv2_op] and [add_op] the inputs don’t share quantization parameters is because the output of [conv1_op] and [conv2_op]  (<code>conv1_output</code> and <code>conv2_output</code>) do not have shared quantization specs, so they will create different quantization parameters, what you can do to support this is something like this:</p>
<pre><code class="lang-auto">arg1, arg2 = add_node.args
# this means sharing with the output qspec of arg1 node
_share_with_arg1_output = SharedQuantizationSpec(arg1)
# set output of arg2 to share with arg1
if _is_annotated(arg1) and _is_annotated(arg2):
    arg2.meta["quantization_annotation"].output_qspec = _share_with_arg1_output

# all input of add should share with the same thing as well
add_node.meta["quantization_annotation"] = QuantizationAnnotation(
    input_qspec_map = {arg1: _share_with_arg1_output, arg2: _share_with_arg1_output}
   ...
}
</code></pre>
What kind of machine learning tasks do you primarily work on with your Core i7 laptop, and how well does it handle them?

jessicashepherd: <p>Hey ML enthusiasts,</p>
<p>If you’re working with machine learning on a Core i7 laptop, let’s have a discussion about the kind of tasks you primarily undertake and how well your Core i7 laptop [ <a href="https://www.lenovo.com/gb/en/d/i7-laptops/" class="inline-onebox" rel="noopener nofollow ugc">we-ch-i7-laptop-dlp | Lenovo UK</a> ] handles them. Share your experiences, tips, and challenges to help others get the most out of their Core i7 laptops for machine learning projects!</p>
<p>Here are some questions to kickstart the conversation:</p>
<p>Machine Learning Tasks: What are the main machine learning tasks you work on with your Core i7 laptop? (e.g., image classification, natural language processing, object detection, regression, etc.)</p>
<p>Performance Evaluation: How does your Core i7 laptop perform during these machine learning tasks? Is it sufficient for your needs, or do you find any limitations?</p>
<p>Training Time: How long does it take for your Core i7 laptop to complete training on typical datasets for your specific machine learning tasks?</p>
<p>Model Sizes: Are you working with large models, and how well does your laptop handle them in terms of memory and computation?</p>
<p>Optimizations: Have you implemented any optimizations to improve machine learning performance on your Core i7 laptop, such as using specific libraries, parallel processing, or hardware acceleration?</p>
<p>Thermal Management: Do you encounter any thermal throttling issues during extended machine learning sessions? If so, how do you handle them?</p>
<p>External Resources: Do you rely on cloud-based resources or external hardware (e.g., eGPUs) to augment your Core i7 laptop’s capabilities for more demanding tasks?</p>
<p>Recommendations: Based on your experiences, what would you suggest to those considering using a Core i7 laptop for machine learning tasks? Any specific models or configurations that have worked well for you?</p>
<p>Challenges and Solutions: Have you faced any challenges specific to using a Core i7 laptop for machine learning, and how did you overcome them?</p>
<p>Remember, this thread aims to build a supportive community where we can learn from each other’s experiences and help fellow machine learning practitioners make informed decisions regarding their Core i7 laptops.</p>
<p>Feel free to share details about your machine learning projects, code snippets, or any other insights that could be valuable to others in the community. Let’s collaborate and make the most of our Core i7 laptops for machine learning tasks!</p>
<p>Looking forward to your contributions! Happy learning and experimenting!</p>
Reverse of concatenation in a specific dimension

Tm4: <p>I am working with <a href="https://github.com/lucidrains/vit-pytorch/blob/6e2393de9595114d1eac0ac154f527fd01772805/vit_pytorch/vit.py#L120" rel="noopener nofollow ugc">vision transformer (ViT)</a>. In one of its steps, the class token is concatenated with the tensor in a specific dimension (dim=1) and I want to remove the class token that had been concatenated. How I can do this?</p>
<pre><code>batch_class_token = self.class_token.expand(n, -1, -1)
x = torch.cat([batch_class_token, x], dim=1)
</code></pre>
Deploying custom model to ONXX

Martim_Afonso: <p>I have the following model defined in Pytorch. I want to export it to onxx (and eventually to onxxjs) so that I can deploy it.<br>
However, due to the way I have defined it, I’m not sure if it can be directly exported without any issues, can anyone give me their thoughts on the matter</p>
<pre><code class="lang-auto">class AttentionModelMIL(nn.Module):
    def __init__(self):
        super(AttentionModelMIL, self).__init__()
        self.L = 1024
        self.D = 128
        self.K = 1

        self.attention = nn.Sequential(
            nn.Linear(self.L, self.D),
            nn.Tanh(),
            nn.Linear(self.D, self.K)
        )

        self.classifier = nn.Sequential(
            nn.Linear(self.L*self.K, 1),
            nn.Sigmoid()
        )

    def forward(self, x):
        A = self.attention(x)  # NxK
        A = A.to(device)
        A = torch.transpose(A, 1, 0)  # KxN
        A = F.softmax(A, dim=1)  # softmax over N
        M = torch.mm(A, x)  # KxL
        M = M.to(device)
        Y_prob = self.classifier(M)
        Y_prob = Y_prob.to(device)
        Y_hat = torch.ge(Y_prob, 0.5).float()

        return Y_prob, Y_hat, A
        
    def calculate_classification_error(self, X, Y):
        Y = Y.float()
        Y = Y.to(device)
        Y_prob, Y_hat, _ = self.forward(X)
        Y_prob, Y_hat = Y_prob.to(device), Y_hat.to(device)
        error = 1. - Y_hat.eq(Y).cpu().float().mean().data.item()

        return error, Y_hat, Y_prob

    def calculate_objective(self, X, Y):
        Y = Y.to(device)
        Y = Y.float()
        Y_prob, _, A = self.forward(X)
        Y_prob, A = Y_prob.to(device), A.to(device)
        Y_prob = torch.clamp(Y_prob, min=1e-5, max=1. - 1e-5)
        neg_log_likelihood = -1. * (Y * torch.log(Y_prob) + (1. - Y) * torch.log(1. - Y_prob)) + (1-Y)# negative log bernoulli
        neg_log_likelihood = neg_log_likelihood.to(device)
</code></pre>
Libth_transformer.so: cannot open shared object file: No such file or directory

arnab_manna: <p>I am using Faster Transfomers, I have built it successfully and I can see the <code>libth_transformer.so</code> file in <code>build/lib</code> folder, but I’m not sure why it’s not detecting it. I am running bart <code>translate_example.py</code>. I have tried multiple times and have provide both relative and absolute path, still nothing. The <code>torch.classes.load_library</code> command is not able to to detect the shared file at all. The file has all rwx permissions too.</p>
<p>Env: pytorch - 1.13.0, cuda version - 11.8, RTX 3080 Ti<br>
Same issue on the g5 instances of AWS too</p>
<p><code>OSError: /workspace/FasterTransformer/build/lib/libth_transfomer.so: cannot open shared object file: No such file or directory</code></p>
<p>Can anyone please help, Been stuck at it for a long time</p>
Super light weight architecture for ImageNet

serafej: <p>Hello, I’m looking for a super light weight baseline architecture in PyTorch that achieves some level of performance on ImageNet (i.e. 30-40% top-1). I am aware of SqueezeNet, MobileNet etc. but I am looking for something even simpler with a lower requirement for accuracy. The idea is to simply provide a reference for how introductory textbook architectures (not too far from a simple CNN) can perform on this task.</p>
<p>Does something like this already exist (ideally with the architecture already written in PyTorch)?</p>
Expected weight

Uktamk: <p>Expected weight to be of same shape as normalized_shape, but got weight of shape [1024] and normalized_shape = [768]</p>
<p>help</p>

ptrblck: <p>Could you post a minimal and executable code snippet to reproduce the issue, please?</p>

Uktamk: <p>pipe = StableDiffusionImg2ImgPipeline.from_pretrained(<br>
r"C:\Users\FLEXCG.cache\huggingface\hub\models–runwayml–stable-diffusion-v1-5\snapshots\c9ab35ff5f2c362e9e22fbafe278077e196057f0",<br>
torch_dtype=torch.float32, safety_checker=None).to(<br>
device<br>
)</p>
<p>images = pipe(prompt=prompt, image=init_image, strength=strength, guidance_scale=guidance_scale,<br>
num_inference_steps=num_inference_steps, negative_prompt=negative_prompt,<br>
num_images_per_prompt=num_images_per_prompt, generator=generator).images</p>
Torchvision Transformation

Abhishek_Gupta: <p>Hello everyone,</p>
<p>I am trying data augmentation in Torchvision CIFAR10 dataset. But there is one catch, I can not find probability option, i.e. for example if I apply <code>Pad</code> then it will apply padding to every image, but since it is augmentation this augmentation need to be applied randomly with some <code>Probablity</code> and I cannot find any probability parameter unlike Albumentation.</p>
<p>Please guide me</p>
<p>Also I also want to know can we use Albumentation Transform in transform parameter of <code>torchvision.datasets.CIFAR10</code></p>
<p>Thank You</p>

ptrblck: <p>Would <a href="https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.RandomApply"><code>transforms.RandomApply</code></a> work for your use case?</p>
ERROR: No matching distribution found for torch==1.3.1

Shivansh-yadav13: <p>Hello I’m a newbie learning ml and I’m facing this issue while installing dependencies of a project. I’m using conda env with python 3.6 <code>conda create -n mlenv python=3.6</code> <code>pip install -r requirements.txt</code> <strong>requirements.txt</strong></p>
<pre><code class="lang-auto">librosa==0.8.1
tgt==1.4.4
pyloudnorm==0.1.0
praatio==3.8.0
tensorboardX==1.9
torch==1.3.1
</code></pre>
<p>output</p>
<pre><code class="lang-auto">Collecting librosa==0.8.1
  Using cached librosa-0.8.1-py3-none-any.whl (203 kB)
Collecting tgt==1.4.4
  Using cached tgt-1.4.4.tar.gz (21 kB)
Collecting pyloudnorm==0.1.0
  Using cached pyloudnorm-0.1.0-py3-none-any.whl (9.3 kB)
Collecting praatio==3.8.0
  Using cached praatio-3.8.0-py3-none-any.whl (67 kB)
Collecting tensorboardX==1.9
  Using cached tensorboardX-1.9-py2.py3-none-any.whl (190 kB)
ERROR: Could not find a version that satisfies the requirement torch==1.3.1 (from versions: 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2)
ERROR: No matching distribution found for torch==1.3.1
</code></pre>
<p>why is this happening?</p>

ptrblck: <p>You can find the old <code>1.3.1</code> release for Python 3.6 <a href="https://pypi.org/project/torch/1.3.1/#files">here</a>, but note that it was released in Nov. 2019 and also note that Python 3.6 is at its EOL, so I would recommend sticking to a newer release.</p>
Proper Sampling Strategy

smmislam: <p>Hello good people! I am a bit stuck with the following problem, if anyone can kindly provide some feedback, I’d really appreciate it!</p>
<p><strong>Introduction:</strong><br>
Let’s assume that we are implementing a probabilistic model (i.e. the encoder of a VAE) that takes an image and predicts stochastic latent vector. To be concrete, let us say that the batch size is <strong>16</strong> and latent variable dimension is <strong>32</strong>. Hence, the final layer of this network would have 64 units (32 for the mu, 32 for the log_var). And for one batch input, the output shape will be 16x64. Chunking this output into two parts, we get mu (16x32) and log_var (16x32).</p>
<p><strong>Problem:</strong><br>
We want to draw samples from the stochastic variables that must allow gradient flow.</p>
<p><strong>Current Solution I:</strong></p>
<pre><code class="lang-auto">sample = mu + torch.exp(log_var)*torch.randn(16, 32)
</code></pre>
<p><strong>Current Solution II:</strong></p>
<pre><code class="lang-auto">from torch.distributions import Normal

dist = Normal(loc=torch.zeros(32), scale=torch.eye(32))
sample = mu + torch.exp(log_var)*dist.sample(sample_shape=torch.Size([16, 32]))
</code></pre>
<p><strong>Alternative Solution … ?</strong></p>
<pre><code class="lang-auto">...
dist = torch.distributions.Normal(loc=mu, scale=torch.exp(log_var))
...
...
sample = dist.rsample()
</code></pre>
<p><strong>Questions:</strong></p>
<ol>
<li>Are these <strong>current solutions</strong> correct? Are they identical?</li>
<li>Can I get an <strong>alternative solution</strong> with the usage of <code>rsample()</code> or may be <code>torch.distribution.Independent</code>? That is can we simply plug-in the predicted <code>mu</code> and <code>sigma</code> into one of the built-in pytorch distributions and sample from that?</li>
</ol>
How to use the `meta` device to efficiently load large models onto the GPU?

vgoklani: <p>Hey there,</p>
<p>I’d like to leverage the <code>meta</code> device to efficiently load large models (i.e. MPT-30B, etc) onto my GPU, without first creating the model on the CPU and then doing a copy. I would also like to do quantization on the fly <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> so this seems like a necessary first step.</p>
<p>My understanding is that the <code>meta</code> device should be used for this purpose, but I’m not sure how to proceed.</p>
<p>I did trace through the <code>accelerate</code> implementation:</p>
<pre><code>https://github.com/huggingface/accelerate/blob/main/src/accelerate/big_modeling.py#L79
</code></pre>
<p>but it’s unclear to me why the <code>init_on_device</code> function is doing all this work.</p>
<p>Is there a simple clear implementation that I could trace through to understand all the steps properly.</p>
<p>Thanks!</p>

vgoklani: <p>Hey - just wanted to bump this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I’m trying to understand how to properly use the <code>meta</code> device to load models directly on the GPU… Could you please point me to a clear example. thanks!</p>

vgoklani: <p>Just wanted to  this up, as i’m still stuck on it. thank you</p>