akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Default process group has not been initialized

Odin_NI: <p>Problems with init_process_group.</p>
<pre><code class="lang-auto">mp.spawn(init_distributed, args=(hparams, n_gpus, group_name,), nprocs=n_gpus, join=True)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">def init_distributed(rank, hparams, n_gpus, group_name):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '54321'
    print(f"{rank=} init complete")

    dist.init_process_group(
        backend=hparams.dist_backend,
        init_method=hparams.dist_url,
        world_size=n_gpus,
        rank=rank,
        group_name=group_name,
    )

    print("Done initializing distributed")
</code></pre>
<p>I get:<br>
FP16 Run: True<br>
Dynamic Loss Scaling: True<br>
Distributed Run: True<br>
cuDNN Enabled: True<br>
cuDNN Benchmark: False<br>
Initializing Distributed<br>
Initializing Distributed<br>
rank=1 init complete<br>
rank=0 init complete<br>
Done initializing distributed<br>
Done initializing distributed</p>
<p>and next error:<br>
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.</p>
<p>I use 4090 and 3080ti gpus.</p>
<p>Can someone please help me?</p>

Odin_NI: <p>Change trainer by <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#basic-use-case" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.0.0+cu117 documentation</a>.<br>
Now train work.</p>
Why is the OneCycleLR scheduler's "max_lr" is smaller than the SGD optimizer's "lr" in the documentation example?

jagandecapri: <p>Hello,</p>
<p>Referring to the documentation of OneCycleLR at <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.OneCycleLR.html#torch.optim.lr_scheduler.OneCycleLR" rel="noopener nofollow ugc">here</a>, there is a provided sample code as follows:</p>
<pre><code class="lang-auto">data_loader = torch.utils.data.DataLoader(...)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10)
for epoch in range(10):
    for batch in data_loader:
        train_batch(...)
        scheduler.step()
</code></pre>
<p>From my understanding, the <code>max_lr</code> of the OneCycleLR is the maximum learning rate for the optimizer. However, I am not sure why in the example code the <code>max_lr</code> is smaller than the SGD optimizer’s <code>lr</code>?</p>

Giang_Nguyen: <p>Hey, this is also my concern for a long time! Is this a mistake as I also see ChatGPT does the same thing.</p>
nn.LSTM, nn.Embedding, nn.Tanh C++ Implementations

umutsuluhan: <p>Hi,</p>
<p>I need C++ backend implementations of nn.LSTM, nn.Embedding and nn.Tanh for a project that I must match exact or close behaviour of PyTorch. I was able to find code but couldn’t be sure. Could you please pin these implementations?</p>
<p>Thanks</p>

ptrblck: <p>Could you post the code locations you have already found so that we could double check them, please?</p>

umutsuluhan: <p>Yeah, sure my bad. I had to include it. This is LSTM implementation I found:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/6faa6c68e8b76fb68f3a2b2783685102d0e87c00/aten/src/ATen/native/vulkan/ops/Lstm.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/6faa6c68e8b76fb68f3a2b2783685102d0e87c00/aten/src/ATen/native/vulkan/ops/Lstm.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/6faa6c68e8b76fb68f3a2b2783685102d0e87c00/aten/src/ATen/native/vulkan/ops/Lstm.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/6faa6c68e8b76fb68f3a2b2783685102d0e87c00/aten/src/ATen/native/vulkan/ops/Lstm.cpp</a></h4>


      <pre><code class="lang-cpp">#include &lt;ATen/TensorOperators.h&gt;
#include &lt;ATen/native/vulkan/ops/Lstm.h&gt;
#include &lt;ATen/native/vulkan/ops/Mm.h&gt;
#include &lt;torch/library.h&gt;

#ifndef AT_PER_OPERATOR_HEADERS
#include &lt;ATen/Functions.h&gt;
#else
#include &lt;ATen/ops/addmm.h&gt;
#include &lt;ATen/ops/cat.h&gt;
#include &lt;ATen/ops/sigmoid.h&gt;
#include &lt;ATen/ops/slice.h&gt;
#include &lt;ATen/ops/tanh.h&gt;
#endif

namespace at {
namespace native {
namespace vulkan {
namespace ops {
namespace {
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/6faa6c68e8b76fb68f3a2b2783685102d0e87c00/aten/src/ATen/native/vulkan/ops/Lstm.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Please let me know if this is correct. I haven’t had a chance to look for Embedding and Tanh but those should be pretty straightforward.</p>
<p>Thanks</p>
Difference between wall duration and self time in profiler

manurare: <p>What is the difference between wall duration and self time reported in the profiler? Which one describes the actual time that took to execute <code>aten::take</code>?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7e2be67ccfe08d5c839674a57ca605b3035523b.png" data-download-href="https://discuss.pytorch.org/uploads/default/f7e2be67ccfe08d5c839674a57ca605b3035523b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7e2be67ccfe08d5c839674a57ca605b3035523b_2_690x177.png" alt="image" data-base62-sha1="zmTJlQH7xcnr19NRHL2B1bp27Cb" width="690" height="177" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7e2be67ccfe08d5c839674a57ca605b3035523b_2_690x177.png, https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7e2be67ccfe08d5c839674a57ca605b3035523b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7e2be67ccfe08d5c839674a57ca605b3035523b.png 2x" data-dominant-color="E8E8EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">927×238 9.75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Select Data to Be Processed On Forward Call

Martim_Afonso: <p>Hello!</p>
<p>I’m developing a model for classification of Images with Gigapixel size and multiple levels of amplification. Due to this fact, this images need to be divided in multiple patches across the different amplifications.</p>
<p>I want my model to process patches at an amplification level m, give them a confidence score and, according to that score, select the corresponding patches at a higher level of amplification n to then be processed as well.</p>
<p>I’m using http requests to fetch the patches at different magnifications. However, I do not want to make several http requests to fetch all the patches at level n, since I might not be using a good part of them.</p>
<p>The only approach I can think of is, in the model’s forward function, after getting the scores for patches at level m, make the http calls to the necessary patches at level n and process them. However, this doesn’t look like a good practice.</p>
<p>Is there any other good way to do this?</p>
Tensor data pointer is inconsistent between dtypes

BasileBasile: <p>Hi, I’m new with torch cpp extensions and I’m having trouble when trying to access part of the bits stored on a tensor.</p>
<pre><code class="lang-auto">void foo(torch::Tensor A) {
	void* ptr = A.data_ptr();
	((uint16_t*)ptr)[0] = 1;
	printf("byte 0 = %u\n", ((uint8_t*)ptr)[0]);
	printf("byte 1 = %u\n", ((uint8_t*)ptr)[1]);
}
</code></pre>
<p>returns :</p>
<blockquote>
<p>byte 0 = 1<br>
byte 1 = 0</p>
</blockquote>
<p>It seems that the 8 bits array and the 16 bits array are not aligned, the 16 bits one starting one byte before the other. Why is that so ?</p>
Attribution understanding

KonstB: <p>Hello. Can you help me with understanding of features attributions?<br>
I suppose, that they are connected to the feature importance, yes?<br>
I looked through California Housing example (<a href="https://captum.ai/tutorials/House_Prices_Regression_Interpret" class="inline-onebox" rel="noopener nofollow ugc">Captum · Model Interpretability for PyTorch</a>) and Titanic (<a href="https://captum.ai/tutorials/Titanic_Basic_Interpret" class="inline-onebox" rel="noopener nofollow ugc">Captum · Model Interpretability for PyTorch</a>). In both attributions (IntegratedGradients, for example) are calculated for each data sample, and then averaged for the whole dataset. If final value is close to zero - this means that feature is not very important, yes?<br>
But in Titanic there is a note, where larger magnitude attributions correspond to the examples with larger Sibsp feature values, suggesting that the feature has a larger impact on prediction for these examples. (See section <strong>In [19]</strong>, sorry I can put only 1 image in this message)</p>
<p>Now I’m looking at my data and see, that attribution have a strong dependence from the parameter value, but <em>in average</em> - it is close to zero. How to interpret that? Can you give me a clue?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/3/4365d195f4ba0f6010632ff09abdd44fe1987922.png" alt="image" data-base62-sha1="9Ce8r2w53YUez0LXIJs13835ero" width="194" height="104"><br>
(x-axis is a parameter value, y-axis is a DeepLift value here, but IG performs in the same manner)</p>
Training yolov5 model with 4K images (best GPU)

PedroC: <p>Hi,</p>
<p>I need to train a yolov5 model with 4K images. I have a RTX 3060 12 GB VRAM but I get a cuda out of memory even with batch size equal to 1. I’m wondering what is a good GPU that can work just fine to handle yolov5x6.pt with batch sizes of 16+. Also I need to convert it to onnx. What is also the best GPU for inferencing a onnx model with 4K images?</p>
<p>Thanks</p>

PedroC: <p>Does anyone have an idea? This is urgent.</p>
<p>Thanks</p>
Torch.jit.trace example_inputs how to set specified parameters？

moluchase: <p>code like this:<br>
<code>self.bert = torch.jit.trace(bert_model,(input_ids,attention_mask))</code><br>
I want to set <code>output_hidden_states=True,return_dict=True</code>, but forward function of BertModel has other parameter, so I must feed all parameters ?<br>
torch version is 1.10<br>
2.0 version seems to have <code>example_kwarg_inputs</code> to achieve</p>
My classification model is giving me different predictions for the same word when it's alone and when its in a dataframe

Bayrem_Abdellaoui: <p>i trained a classification model on skills using BERT.<br>
the model gives different results when it predicts on a single skill vs a data frame of skills.<br>
this is the code i use for inference on single words:</p>
<pre><code class="lang-auto">def predict_skill(skill: str):
    encoded_dict = tokenizer.encode_plus(
                        skill,                     
                        add_special_tokens = True,
                        max_length = 32,          
                        pad_to_max_length = True,
                        return_attention_mask = True,  
                        return_tensors = 'pt')
    
    skill_input_ids = encoded_dict['input_ids']
    skill_attention_masks = encoded_dict['attention_mask']
    skill_pred = model(skill_input_ids, skill_attention_masks)['logits'].cpu().detach().numpy()
    skill_pred_label = np.argmax(skill_pred, axis=1)
    
    return CategoriesD[skill_pred_label[0]]
</code></pre>
<p>Ps: CategoriesD is the skills categories dictionnay.</p>
<p>and this is the code i use when predicting on a dataframe of skills:</p>
<pre><code class="lang-auto">test_preds = []

for input_ids, attention_masks in test_dataloader:
    input_ids = input_ids.to(device)
    attention_masks = attention_masks.to(device)  

    # get the predicted labels for the test data
    test_logits = model(input_ids, attention_masks)['logits'].cpu().detach().numpy()
        
    test_preds.append(np.argmax(test_logits, axis=1).flatten())
    
test_preds = np.concatenate(test_preds, axis = 0)
</code></pre>
<p>I am using the same parameters and the same tokenizer also, it gives the same words different categories based on wether they are fed to the model in a dataframe or as single words, and i can’t figure out why!<br>
PS: it gives better classification when it’s single skills!</p>

ptrblck: <p>Is your model using layers which use some batch stats such as batchnorm layers and if so, did you call <code>model.eval()</code> before comparing the outputs?<br>
If that’s the case, could you compare the output of a single sample vs. a batch using the same repeated sample?</p>

Bayrem_Abdellaoui: <p>thatks for the reply <a class="mention" href="/u/ptrblck">@ptrblck</a>, initially I was using model.eval() and they were different predictions so i did a comparison without using model.eval() and i used the same code, the only difference is the batch in a dataloader and the samples are just text!</p>
Segmentation fault when using custom tensor

zhouxian: <p>A minimal example to reproduce:</p>
<pre><code class="lang-auto">import torch


class MyTensor(torch.Tensor):
    pass

actions = torch.rand(size=(10, 3), device='cuda')
actions = MyTensor(actions).requires_grad_()+1



actions[:, 0] = 0.01
actions.retain_grad()
</code></pre>
<p>The last line of <code>retain_grad()</code> throws segmentation fault. If I comment out <code>actions[:, 0] = 0.01</code>, it works fine. Why is this happening?<br>
Thanks in advance!</p>

ptrblck: <p>I cannot reproduce this issue in <code>2.0.0+cu118</code> and get:</p>
<pre><code class="lang-python">class MyTensor(torch.Tensor):
    pass

actions = torch.rand(size=(10, 3), device='cuda')
actions = MyTensor(actions).requires_grad_()+1

actions[:, 0] = 0.01
actions.retain_grad()
print(actions)
# MyTensor([[0.0100, 1.0819, 1.2367],
#           [0.0100, 1.1117, 1.2771],
#           [0.0100, 1.3408, 1.4321],
#           [0.0100, 1.8467, 1.1795],
#           [0.0100, 1.9159, 1.1804],
#           [0.0100, 1.0648, 1.5493],
#           [0.0100, 1.3131, 1.5877],
#           [0.0100, 1.5165, 1.1700],
#           [0.0100, 1.6162, 1.8943],
#           [0.0100, 1.1664, 1.7574]], device='cuda:0',
#          grad_fn=&lt;AsStridedBackward0&gt;)
</code></pre>
<p>Could you update your PyTorch version in case you are using an older one and check if you are still seeing this issue?</p>
Pytorch DistributedDataParallel training

Shia_Edwards: <p>code: <a href="https://github.com/basiclab/GNGAN-PyTorch/blob/master/train_ddp.py" class="inline-onebox" rel="noopener nofollow ugc">GNGAN-PyTorch/train_ddp.py at master · basiclab/GNGAN-PyTorch · GitHub</a></p>
<p><code>train_ddp.py</code> is optimized for multi-gpu training, e.g.,<br>
CUDA_VISIBLE_DEVICES=0,1,2,3 python train_ddp.py <br>
–flagfile ./config/GN-GAN_CELEBAHQ256_RES.txt</p>
<p>colab seems to have only one gpu but it doesn’t matter. Whether you write 4 in this way or just one cuda=0 will be a problem.</p>
<p>error list：<br>
The server socket has failed to bind to [::]:55556 (errno: 98 - Address already in use).<br>
Process Process-2:<br>
Process Process-3:<br>
Process Process-4:<br>
Traceback (most recent call last):<br>
Traceback (most recent call last):<br>
Traceback (most recent call last):<br>
RuntimeError: Timed out initializing process group in store based barrier on rank: 1, for key: store_based_barrier_key:1 (world_size=4, worker_count=10, timeout=0:00:30)</p>
<p>or like this</p>
<p>RuntimeError: CUDA error: invalid device ordinal<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
<p>After searching a lot in google, it seems that none of them can solve the problem, apparently it is a problem of parallel training (ddp). And google colab compatibility should be the best, s so I have been debugging on google colab.</p>
<p>The following code seems to have worked before, but it was used on the cifar10 and stl10 datasets, maybe the celeba dataset is larger? And using parallel computing? So I do not know how to do.<br>
!export CUDA_VISIBLE_DEVICES=0,1,2,3<br>
!python train_ddp.py --flagfile .[/config/GN-GAN_CELEBAHQ256_RES.txt]</p>
Pytorch 2.0 supported Python version

kushal140: <p>Which python version is Pytorch 2.0 supported?<br>
I have read the documentation, which says</p>
<p><em>Currently, PyTorch on Windows only supports Python 3.7-3.9; Python 2. x is not supported.</em></p>
<p>Python 3.11 is faster compared to Python 3.9.<br>
So, When will Pytorch be supported with updated releases of python (3.10,3.11)?</p>
TypeError: unsupported operand type(s) for /: 'str' and 'str'

shrbrh: <p>Hi,<br>
I am training a neural network and want to save my model state after every 100 iterations. My code snippet is:</p>
<pre><code class="lang-auto">state_dict  = model.state_dict()
for key in state_dict.keys():
			state_dict[key] = state_dict[key].to(torch.device('cpu'))
torch.save(state_dict, args.save_dir /
			'model_iter_{:d}.pth.tar'.format(i + 1))
</code></pre>
<p>But I am getting the following error:</p>
<pre><code class="lang-auto"> 'model_iter_{:d}.pth.tar'.format(i + 1))
TypeError: unsupported operand type(s) for /: 'str' and 'str'
</code></pre>
<p>What can I do to resolve this? Please help.</p>
My RetinaNet mAP goes crazily high

Katono88: <p>Im doing a object detection job, and the dataset contains many ducks</p>
<p>the dataset has 500 samples and I used yolov5 and retinaNet to train the model.</p>
<p>Now the weird thing happens, when using yolov5, my 1-shot and 5-shot mAP are like 25-30%, 10-shot is around 65%, all-dataset is around 95%.</p>
<p>But when using RetinaNet, my 1-shot, 5-shot, 10-shot are all aroung 80%, and the all-dataset is aroung 95%, which is pretty weird for me since I think 1-shot and 5-shot should not get such a high mAP.</p>
<p>I know that not showing the whole code won’t give much information about my problem, I’m just curious if anyone has similar problems? And what would probably go wrong?</p>
Unrealistic loss using BCEWithLogitsLoss()

SimCan: <p>I am trying to use <code>nn.BCEWithLogitsLoss()</code> with <code>ResNet50</code> architecture for a binary classification task.<br> In order to do that, I have set the net’s fully connected layer in this way:  <code>fc = nn.Linear(2048, 1)</code>. Furthermore, I have also changed the targets’ shape: <code>labels = labels.unsqueeze(1).float()</code>.<br>
It seems don’t work properly, because using this kind of loss I get an accuracy score greater than 1, and a loss very near 0.</p>
<p>Afterward, I tried to use <code>nn.CrossEntropyLoss()</code> and  <code>fc = nn.Linear(2048, 2) </code>without unsqueeze the labels; and it works correctly.</p>
<p>I would like to know why <code>nn.BCEWithLogitsLoss()</code> returns these strange values, and <code>nn.CrossEntropyLoss()</code> works properly instead.</p>

ptrblck: <p>Check the min./max. values in your target and make sure they are in the range <code>[0, 1]</code>.<br>
Also, how do you calculate the accuracy? Since your model outputs logits you should use a threshold e.g. via:</p>
<pre><code class="lang-python">preds = output &gt; 0.0
</code></pre>

brighteningeyes: <p>when you want to calculate the accuracy after you obtained the prediction with model, you need to apply sigmoid as the outputs of your models are logitss (it can be from -inf to inf). then, apply thresholding after it.<br>
basically, what BCEWithLogits does, is to apply sigmoid then calculates BCELoss. if you are calculating sigmoid by your own, you dont need BCEWithLogits. you instead want to use BCELoss<br>
if you are using BCEWithLogits, after you obtain your predictions, you need to apply sigmoid to calculate accuracy and sigmoid should not be applied when passing model’s output to loss</p>
How to split dataset into subset with non-uniform distribution of labels

Zowie_Tay: <p>I’m experimenting with federated learning and i need non-iid data but using torch.utils.data.random_split() splits my mnist dataset into subsets with uniformed label distribution</p>
<blockquote>
<p>Server Label Distribution<br>
0 : 2930 | 1 : 3371 | 2 : 3004 | 3 : 3079 | 4 : 2866 | 5 : 2695 | 6 : 2962 | 7 : 3150 | 8 : 2966 | 9 : 2977 |<br>
Clients Label Distribution<br>
0 :  263 | 1 :  290 | 2 :  269 | 3 :  268 | 4 :  277 | 5 :  247 | 6 :  271 | 7 :  276 | 8 :  272 | 9 :  267 |<br>
0 :  218 | 1 :  263 | 2 :  192 | 3 :  214 | 4 :  208 | 5 :  181 | 6 :  216 | 7 :  198 | 8 :  222 | 9 :  188 |<br>
0 :   76 | 1 :   70 | 2 :   63 | 3 :   65 | 4 :   68 | 5 :   34 | 6 :   44 | 7 :   57 | 8 :   68 | 9 :   55 |<br>
0 :  448 | 1 :  490 | 2 :  430 | 3 :  469 | 4 :  443 | 5 :  414 | 6 :  448 | 7 :  470 | 8 :  437 | 9 :  451 |<br>
0 :  746 | 1 :  829 | 2 :  784 | 3 :  768 | 4 :  737 | 5 :  693 | 6 :  758 | 7 :  778 | 8 :  666 | 9 :  741 |<br>
0 :   95 | 1 :  100 | 2 :   83 | 3 :   94 | 4 :   95 | 5 :   80 | 6 :   85 | 7 :   94 | 8 :   80 | 9 :   94 |<br>
0 :   77 | 1 :   97 | 2 :  110 | 3 :   93 | 4 :   74 | 5 :   97 | 6 :   85 | 7 :   91 | 8 :   86 | 9 :   90 |<br>
0 :  244 | 1 :  302 | 2 :  269 | 3 :  266 | 4 :  268 | 5 :  248 | 6 :  264 | 7 :  292 | 8 :  284 | 9 :  263 |<br>
0 :  770 | 1 :  859 | 2 :  703 | 3 :  738 | 4 :  739 | 5 :  678 | 6 :  728 | 7 :  803 | 8 :  712 | 9 :  770 |<br>
0 :   56 | 1 :   71 | 2 :   51 | 3 :   77 | 4 :   67 | 5 :   54 | 6 :   57 | 7 :   56 | 8 :   58 | 9 :   53 |</p>
</blockquote>
<p>any idea how to make the data skewed?</p>

ptrblck: <p>You could use e.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"><code>sklearn.model_selection.train_test_split</code></a> and use the <code>stratify</code> argument to create the imbalanced split. Once you have the indices you could then use these to create <code>torch.utils.data.Subset</code>s.</p>

Zowie_Tay: <p>do you know how to do that with the torchvision datasets?</p>

ptrblck: <p>You would need to access or load the internal targets from the desired dataset and pass it to <code>train_test_split</code> and <code>stratify</code>. If your dataset is preloading the data, check its internal attribute for e.g. <code>.targets</code>, else iterate the dataset once to load the targets and use them afterwards.</p>
Time series classification with variable series length

mirksonius: <p>Hi there!<br>
I’m solving a problem with RNNs where the time series have varying lengths, to be more concrete the examples in the training set are mostly of the same size (which I solve with paddding) but when it comes to the validation dataset the examples are 3-5 times longer (with respect to time).<br>
The idea was to chunk the validation set examples into portions that have similar size to the examples in the train set. Is there an efficient way to implement this with <code>collate_fn</code>, or with the Dataset module?</p>
<p>Thank you!</p>
IMDB returns only sentiment of 1

IljaAvadiev: <p>Hallo community,</p>
<p>I was trying to create a simple sentiment analysis model in Pytorch and I wanted to use torchtext.<br>
<code><br>
train_dataset = torchtext.datasets.IMDB(root=‘./datasets’, split=‘train’)<br>
test_dataset = torchtext.datasets.IMDB(root=‘./datasets’, split=‘test’)<br>
dl = DataLoader(train_dataset, batch_size=1000, shuffle=True)<br>
next(iter(dl))[0]<br>
</code></p>
<p>When I use the above code I expected to get a mixture of 0’s and 1’s as the sentiment value, but for some reasons I get only 1’s.</p>
<p>Is there something wrong on my end?</p>

raj-rishav: <p>I dont know what’s going on for sure. But yeah if you iterate over the dataloader using a <code>for</code> loop, you will find that there are both sentiments available in the batch.<br>
Also  dataset provided by torchtext, has labels <code>1</code> and <code>2</code> . I dont think theres anything wrong at your end, as when I tried to do it, I got the same results</p>
<pre><code class="lang-auto">for i, j in dl:
    print(i)
</code></pre>
<p>you will see that there are instances of label <code>2</code> too.</p>

IljaAvadiev: <p>Thanks for the answer <a class="mention" href="/u/raj-rishav">@raj-rishav</a>.</p>
<p>I think this is an issue with torchtext.<br>
<a href="https://github.com/pytorch/text/issues/2041" rel="noopener nofollow ugc">GitHub Issue</a></p>
Reduce Memcpy DtoD time in backward pass

manurare: <p>Hi,</p>
<p>I have a small model with some custom convolutional layers which. Before the actual conv happen I do some slicing and copying as following:</p>
<pre><code class="lang-auto">outputs = inputs.new_zeros(inputs.size(0), self.in_channels, self.padded_height_in, self.padded_width_in)

if self.in_feature_depth == 6:
    outputs[:, :, self.padding_target_y, self.padding_target_x] = inputs[:, :, self.padding_src_y, self.padding_src_x]
    outputs[:, :, self.padding_ch_up_target_y, self.padding_ch_up_target_x] = inputs[:, self.ch_shift_up, self.padding_ch_up_src_y, self.padding_ch_up_src_x]
    outputs[:, :, self.padding_ch_down_target_y, self.padding_ch_down_target_x] = inputs[:, self.ch_shift_down, self.padding_ch_down_src_y, self.padding_ch_down_src_x]
else:
    outputs[:, :, self.padding_target_y, self.padding_target_x] = inputs[:, :, self.padding_src_y, self.padding_src_x]

outputs.narrow(3, 1, self.padded_width_in-2).index_copy_(2, self.row_target, inputs)
</code></pre>
<p>Where does the Memcpy DtoD come from and how to reduce its time? I know 12ms is not much but this is a small model with just 3 conv layers for a unit test. In the real application I am using a big model and would like to make the training faster.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbab2ce623b85a52de24f251462857d0dbac4730.png" data-download-href="https://discuss.pytorch.org/uploads/default/fbab2ce623b85a52de24f251462857d0dbac4730" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbab2ce623b85a52de24f251462857d0dbac4730_2_690x63.png" alt="image" data-base62-sha1="zUmAn6Q9JHchwZiCSpg9rcPaW2I" width="690" height="63" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbab2ce623b85a52de24f251462857d0dbac4730_2_690x63.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbab2ce623b85a52de24f251462857d0dbac4730_2_1035x94.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbab2ce623b85a52de24f251462857d0dbac4730_2_1380x126.png 2x" data-dominant-color="2F2F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2745×254 30.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Combining Tensors in a Simple Multiplicative Way

arcanjomjr: <p>Hello, my first post here. I did search for the question but wasn’t able to find it elsewhere in the forum.</p>
<p>I have a tensor X of dimension A, and a tensor Y of dimensions MxNxP and want to create a new tensor Z of dimensions MxAxNxP where Z[m,a,n,p] = X[a]*Y[m,n,p].</p>
<p>I did find a solution but was rather inelegant and possibly inefficient.</p>
<p>There is a simple way of doing this?</p>

ptrblck: <p>Broadcasting should work as seen here:</p>
<pre><code class="lang-python"># setup
A, M, N, P = 2, 3, 4, 5
X = torch.randn(A)
Y = torch.randn(M, N, P)
Z_ref = torch.zeros(M, A, N, P)

# slow approach to create reference
for m in range(M):
    for a in range(A):
        for n in range(N):
            for p in range(P):
                Z_ref[m, a, n, p] = X[a] * Y[m, n, p]

# non-loop approach
Z = X[None, :, None, None] * Y[:, None, ...]

# compare
print((Z_ref - Z).abs().max())
# tensor(0.)
</code></pre>

arcanjomjr: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, that is a very nice solution!</p>
How to aovid for-loop in torch

jiahuil: <p>Hi all, I meet a simple problem in torch.<br>
I have a vector <code>num_nodes</code> which indicate the number of nodes on each graph, then I want to get vector <code>batch</code> to present which graph each node belongs to, it can be written in for-loop easily but has high latency, is there any operation I can use to vectorize it?</p>
<pre><code class="lang-python"># input
# num_nodes: [2,3]
for i, num in enumerate(num_nodes):
        batch[cum : cum + num] = i
        cum += num
# output
# batch: [0, 0, 1, 1, 1]
</code></pre>
<p>Any advice from you would be appreciated! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

J_Johnson: <p>This should be a bit faster:</p>
<pre><code class="lang-auto">def build_nodes_list(num_nodes:list):
    return torch.cat([torch.full(num, i) for i, num in enumerate(num_nodes)])
</code></pre>

KFrank: <p>Hi Jiahuil!</p>
<aside class="quote no-group" data-username="jiahuil" data-post="1" data-topic="174913" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jiahuil/48/52216_2.png" class="avatar"> jiahuil:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python"># batch: [0, 0, 1, 1, 1]
</code></pre>
</blockquote>
</aside>
<p>Here’s a loop-free version:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.1'
&gt;&gt;&gt; num_nodes = torch.tensor ([2, 3, 2])
&gt;&gt;&gt; batch = torch.zeros (num_nodes.sum(), dtype = torch.long)
&gt;&gt;&gt; batch[num_nodes[:-1].cumsum (0)] = 1
&gt;&gt;&gt; batch = batch.cumsum (0)
&gt;&gt;&gt; batch
tensor([0, 0, 1, 1, 1, 2, 2])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Is there any issue if i create a new loss function and optimizer each time i run the training?

Zowie_Tay: <p>is there any issue if i create a new loss function and optimizer each time i run the training?</p>
<blockquote>
<p>for t in range(epoch):<br>
loss_fn = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)<br>
print(f"Epoch {t+1}\n-------------------------------")<br>
train(train_dataloader, model, loss_fn, optimizer)</p>
</blockquote>

Omroth: <p>No, nothing wrong with it.  Tiny bit slower but probably not significant.</p>

ptrblck: <p>You shouldn’t see any issues for <code>SGD</code> as <a class="mention" href="/u/omroth">@Omroth</a> mentioned besides the additional overhead of creating the classes, but note that other optimizers with internal states, such as <code>Adam</code>, will reset these states after each reinitialization which might create issues during the model training.</p>

Zowie_Tay: <p>yeah, my worry was that the crossentropyloss might have an internal state, if that is the case, then i would probably just create both the lossfn and optimizer at the beginning and just keep reusing them</p>
Loss.backward() error when using M1 GPU (mps device)

johnnyhuw: <p>I want to train a Seq2Seq model with M1 GPU. The <a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/1%20-%20Sequence%20to%20Sequence%20Learning%20with%20Neural%20Networks.ipynb" rel="noopener nofollow ugc">source code</a> runs well with cpu. However, if I change the device from cpu to mps:</p>
<p><code>device = torch.device("mps")</code></p>
<p>RuntimeError occurs when training model:</p>
<p><code>RuntimeError: Expected a proper Tensor but got None (or an undefined Tensor in C++) for argument #0 'grad_y'</code></p>

liuquartz: <p>Hi, have you got the sulotion?</p>

aogara: <p>I am having this problem too, and cannot find any information on how to solve it.</p>

SamedAli: <p>same error, just can’t find a way to fix it.<br>
Tried</p>
<ul>
<li>loss on gpu and cpu</li>
<li>retain_grad</li>
<li>tried to push output and label to cpu before loss</li>
</ul>

mikesol: <p>Same issue here as well, it looks like it is due to the LSTM in my network. I tried a GRU too but to no avail. CPU is fine, cuda is fine on Linux.</p>
Parallel async inference of multiple models gives CUDA error: initialization error

zshn25: <p>I am trying to run inference of multiple models parallelly on CUDA. My use case is to run them on the same input image. I get the following error.</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>Even with <code>CUDA_LAUNCH_BLOCKING=1</code> I  get <code>RuntimeError: CUDA error: initialization error</code></p>
<p>A MWE would be</p>
<pre><code class="lang-python">import torch
from torch import nn
import torch.multiprocessing as mp
from torchvision import models


if __name__ == '__main__':

    num_processes = 2

    input_image_pytorch = torch.randn((1,3, 256, 256), requires_grad=False, device="cuda")

    processes = []   
    for rank in range(num_processes):
        model = models.resnet18().eval().to("cuda")
        p = mp.Process(target=model, args=(input_image_pytorch,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()

</code></pre>

zshn25: <p>Solved this by adding <code>mp.set_start_method('spawn')</code></p>
Is it possible to get the highest CUDA version supported by PyTorch via python?

songyuc: <p>Hi, guys!<br>
Is it possible to get the highest CUDA version supported by PyTorch via python?<br>
Such as, in the Start Locally | PyTorch,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7dcdbb973556bc687d13273f243089c27c35a60.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7dcdbb973556bc687d13273f243089c27c35a60" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7dcdbb973556bc687d13273f243089c27c35a60_2_690x274.png" alt="image" data-base62-sha1="qewyXiL11HOc9tWAeOq5GTqTxII" width="690" height="274" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7dcdbb973556bc687d13273f243089c27c35a60_2_690x274.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7dcdbb973556bc687d13273f243089c27c35a60_2_1035x411.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7dcdbb973556bc687d13273f243089c27c35a60_2_1380x548.png 2x" data-dominant-color="F4D8D4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1591×634 56.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The highest supported CUDA is 11.7.</p>
<p>So, can we get this version number (11.7) via python?</p>
<p>Your answer and guide will be appreciated!</p>
Followup Email for First Time Posters with Posting Tips

J_Johnson: <p>It might prove helpful to have an automated email go out congratulating them on their first question/post and also providing some bullet point tips that they can use to improve the post quality. For example:</p>
<blockquote>
<p>Thank you [name] for your first post on PyTorch Forums! Your post is being reviewed by other Torchers and will help grow our community.<br>
Here are some tips to help your post be easier to find and view:</p>
<ul>
<li><strong>Does your post title accurately describe the problem or error code?</strong>
<ul>
<li>I.e. Vague titles like <strong>“how to deal with problem”</strong> tend to not get viewed as often. Instead, it’s better to describe the problem or copy the error code —&gt; <strong>“AttributeError: module torchtext.data has no attribute ‘Field’”</strong> This will also help other users to more easily find the same answer when they run into a similar problem.</li>
</ul>
</li>
<li><strong>Is the code you copied wrapped in backticks?</strong>
<ul>
<li>For example placing “`” around your code will make it  inline and placing triple “```” around your code will make it into a code snippet. This will be much easier to read by other Torchers.</li>
</ul>
</li>
<li><strong>Is your code simplified and reproducible?</strong>
<ul>
<li>Copying your actual production code might be difficult for others to read or find the issue. Additionally, it may depend on other libraries others in the community do not use or have.<br>
- First, you may want to try to simplify your code so it can be as short as possible that still reproduces the error. and can be copied and run.<br>
- Second, create a <code>torch.rand()</code> tensor or other appropriate sample data of the appropriate size. This way other users can copy the code and see the same error right away.</li>
</ul>
</li>
</ul>
</blockquote>
<p>Anyways, I just think something like this might help new users starting out to get the help they need quicker.</p>
What does δ < 1.25, δ < 1.25², δ 1.25³ (delta 1.25) stand for?

MikeTensor: <p>I read a paper where δ &lt; 1.25, δ &lt; 1.25², δ 1.25³ are given.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea4760f4f93a7b21fc10011f1d53cde93ec1f670.png" alt="grafik" data-base62-sha1="xqwEGUQGWv5u4DBrNERsTnxS2ha" width="354" height="266"></p>
<p>Since I do not really know for what they stand for …</p>
<p>My question would be:</p>
<ol>
<li>For what exactly do the values stand for?</li>
<li>Why do we want to get/compair those values?</li>
<li>How to get those values?</li>
</ol>
<p>Thanks a lot for your help!</p>

ptrblck: <p>I believe this metric is used in depth estimation use cases comparing the amount of output pixels with a small, medium, and large difference from their ground truth value.<br>
However, I don’t know how these values were defined.</p>

MikeTensor: <p>I additionally found this one:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d72230dc232191aa9d2b7b3d8e47397345f308dd.png" alt="grafik" data-base62-sha1="uH9UoYK1aTvb68G2DH8TP6wmpvD" width="664" height="52"></p>
<p>But I still can’t figure it out, what it means.<br>
Any idea?</p>