akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to connect vision with tensorboard

Iskandar_Zulkarnaen: <p>Hello i’m newbie in pytorch and vision.<br>
I have a code like this.</p>
<pre><code class="lang-auto">from engine import train_one_epoch
# Training
writer = SummaryWriter()
start = datetime.now()
for epoch in range(epochs):
    # train for one epoch, printing every 50 iterations
    train_one_epoch(model, optimizer, loader_train, device, epoch, print_freq=20)
    # update the learning rate
    lr_scheduler.step()
    # evaluate on the test dataset
    evaluate(model, loader_val, device=device)
end = datetime.now()
</code></pre>
<p>Anyone can tell me how to add a tensorboard to monitor my training. Thanks</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/recipes/recipes/tensorboard_with_pytorch.html">This tutorial</a> might be helpful.</p>
Given a tensorflow model how do i convert it into pytorch

Sumesh_Sankar: <p>given a tensorflow model how do i convert it into pytorch</p>

ptrblck: <p>You could rewrite the model by porting each layer and verifying its functionality e.g. by making sure the random initialization of each parameter and buffer is equal.<br>
The layer names should be quite similar for the majority of layers. I don’t know if there are any useful tools to automate this process.</p>
How to limit the number of cpus each process

nikiguo93: <p>Hi, the remote server has 32 cpus. and now I am running the code on the server with 4 gpus. But I want to limit the usage of cpus. For examples, can I use only 16 cpus for my code? we are doing a benchmark and are interested in that. how to realize that?</p>

ptrblck: <p>For inter-op parallelism you should be able to use <code>torch.set_num_interop_threads()</code>, for intra-op <code>torch.set_num_threads()</code>, <code>OMP_NUM_THREADS</code>, and <code>MKL_NUM_THREADS</code> should work.</p>
<blockquote>
<p>For the intra-op parallelism settings, <code>at::set_num_threads</code>, <code>torch.set_num_threads</code> always take precedence over environment variables, <code>MKL_NUM_THREADS</code> variable takes precedence over <code>OMP_NUM_THREADS</code>.</p>
</blockquote>
In place sorting and don't keep the indices

anyfin22: <p>Just wondering if there’s such thing as in place sorting in pytorch?</p>
<p>This comes handy when one is constraint by GPU memory size. I think torch.sort() allocate a new tensor for the sorted result. Doing something in place can double my batch size.</p>
<p>I was able to get only the sorted values and not indices by doing torch.sort()[0].</p>
<p>Thanks</p>
Convert onnx to pytorch state dict?

Andrei_Cristea: <p>Does anyone know whether it’s possible to (and how to) convert a model from onnx back to pytorch? Going from pytorch to onnx is detailed <a href="https://pytorch.org/docs/stable/onnx.html" rel="noopener nofollow ugc">here</a>, but I can’t find any resource about going back.</p>
<p>EDIT: Came across <a href="https://github.com/ENOT-AutoDL/onnx2torch" rel="noopener nofollow ugc">this</a> shortly after posting, which seems promising.</p>
<p>Thanks!<br>
Andrei</p>
Passing PackedSequence as input to nn.Linear

x.projekt: <p>I am implementing neural machine translation using attention. The pre-attention LSTM takes a <code>PackedSequence</code> as input, and produces a <code>PackedSequence</code> as output. But to compute the attention-energies, I need to pass this output (from the pre-attention RNN) through a <code>nn.Linear</code>. So, how to extract the variable length output-sequences from this <code>PackedSequence</code>?</p>
<ul>
<li>Also, Is there a way to extract the individual output sequences from the <code>PackedSequence.data</code> tensor?</li>
</ul>

x.projekt: <p>I think the <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pad_packed_sequence.html" rel="noopener nofollow ugc">torch.nn.utils.rnn.pad_packed_sequence()</a> should do the job.</p>
TimeSformer Pruning

hamza_karim: <p>Hello everyone, I am new to Pytorch, but I am loving the experience. Recently I have been trying to prune the TimeSformer model to get better inference times. I prune the model and save the new model as follows:</p>
<pre><code class="lang-auto">ARG = [12, 1,'model.pyth']
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = TimeSformer(img_size=224, num_classes=400, num_frames=8, attention_type='divided_space_time',ARGS=ARG).to(device=device)
    #model.head = torch.nn.Linear(in_features=768, out_features=50, bias=True)
    num_zeros, num_elements, sparsity = measure_global_sparsity(model)
    print(num_zeros,num_elements,sparsity)
    amount=0.9
    for module_name, module in model.named_modules():
        if isinstance(module, torch.nn.Linear):
            prune.l1_unstructured(module,name="weight",amount=amount)
    pruned_model = remove_parameters(model)
    num_zeros, num_elements, sparsity = measure_global_sparsity(pruned_model)
    print(num_zeros, num_elements, sparsity)
    torch.save(model.state_dict(),"pruned_model_"+str(amount)+".pyth")
</code></pre>
<p>measure_global_sparsity measures the number of zeros, elements and sparsity before and after pruning. While remove_parameters removes the weight and bias parameters after computing masks:</p>
<pre><code class="lang-auto">def remove_parameters(model):

    for module_name, module in model.named_modules():
        if isinstance(module, torch.nn.Conv2d):
            try:
                prune.remove(module, "weight")
            except:
                pass
            try:
                prune.remove(module, "bias")
            except:
                pass
        elif isinstance(module, torch.nn.Linear):
            try:
                prune.remove(module, "weight")
            except:
                pass
            try:
                prune.remove(module, "bias")
            except:
                pass

    return model
</code></pre>
<p>When I prune the model, the size of the model does not change and the inference speed is also unaffected. Although the measured sparsity does increase as I increase the amount of pruning. Why are the model size and inference speed unaffected? Does anyone know what am I doing wrong here?<br>
Thank you.</p>
Find the biggest of two pytorch tensor on size

shijithp: <p>How to find the biggest of two pytorch tensors on size</p>
<pre><code class="lang-auto">&gt;&gt;&gt; tensor1 = torch.empty(0)
&gt;&gt;&gt; tensor2 = torch.empty(1)
&gt;&gt;&gt; tensor1
tensor([])
&gt;&gt;&gt; tensor2
tensor([5.9555e-34])
</code></pre>
<p>torch.maximum is returrning the empty tensor as the biggest tensor</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.maximum(tensor1,tensor2)
tensor([])
</code></pre>
<p>Is there a way to find the biggest tensor among two tensors (mostly 1d), base on the number of elements in the tensor.<br>
desires output is</p>
<pre><code class="lang-auto">&gt;&gt;&gt; maxfunc(tensor1,tensor2)
tensor([5.9555e-34])
</code></pre>

ptrblck: <p>Something like this should work:</p>
<pre><code class="lang-python">def maxfunc(*tensors):
    idx = torch.argmax(torch.tensor([t.nelement() for t in tensors]))
    out = tensors[idx]
    return out


tensor1 = torch.empty(0)
tensor2 = torch.empty(1)
tensor3 = torch.empty(10)

maxfunc(tensor1, tensor2)
maxfunc(tensor1, tensor2, tensor3)
maxfunc(tensor3, tensor1, tensor2)
maxfunc(tensor2)
</code></pre>

shijithp: <p>Thanks @ <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a>,  is there any performance difference if i use len(tensor) or tensor.shape[0] (1d tensor)?</p>

ptrblck: <p>I don’t know if there is an expected performance difference (but you could certainly profile it).<br>
However, these operations are also not used in my code.</p>

shijithp: <p>Thanks for the help.</p>
Conversational chatbot

bello621: <p>so, imagine i have a dataset containing user conversations(user1, user2), and say i want a transformer model to be able to generate a user2 response given user1 as input, but plus the previous history of the conversations so that the model can generate a plausible response.<br>
example, i have a dataset of one example,</p>
<p>conversations = [’’,<br>
‘hello there how are you ?’,<br>
‘i am doing well . how are you ?’,<br>
‘i am great thanks , do like boating ?’,<br>
‘i like it when i can get away from my job at the grocery store’,<br>
‘its my favorite activity outside of being a doctor . do you like beaches ?’,<br>
‘i give deep sea fishing tours at the beach sometimes’,<br>
‘i volunteer at a farm , do you like animals ?’,<br>
‘i am vegan so i love animals . same here with the farm volunteering’,<br>
‘i too am a vegan , how long for you ?’]<br>
one approach is to extract speaker2 response as the target and speaker1 response with its previous information(none if no previous info) as the input to the encoder network<br>
now, prev_history_with_user1 of each target output will be of dimension (n_conversation,n_history_with_speaker1)</p>
<p>assume for first response,</p>
<p>sample_question_with_history = [’  hi , what is your name ?’]<br>
sample_output = [‘hello . i am jake . nice to meet you .’]</p>
<p>for second response,<br>
sample_example = [’ hi , what is your name ? hello . i am jake . nice to meet you . nice to meet you too ! i am vladimir .’]<br>
sample_output = [‘so do you have a profession ? in an orchestra i am the violinist .’]</p>
<p>and so on up untill the current conversation finished.<br>
now imagine i want to train the transformer with lots of examples, how will you arrange the training data?, since the training data will be of shape<br>
(n_examples,n_conversation,n_history_with_speaker1)  will concatenating them so that it will be of shape (n_examples * n_conversations,n_history_with_speaker1) helps?. thank you for your considerate answer</p>
ROC curve not smooth as a curve

tamersaleh: <p>Hi,<br>
When I plotted a ROC curve using a Unet model with a binary segmentation image, I found that the curve was not as smooth as a curve but appeared as a straight line with only three points, I can’t solve this problem. Please, anyone encountered such a situation, how to get a smooth curve. This is the code.</p>
<p>def roc(label, cd_preds):<br>
fpr, tpr, thresholds = roc_curve(label, cd_preds)<br>
roc_auc = auc(fpr, tpr)<br>
plt.figure(figsize=(8,8))<br>
plt.plot(fpr,<br>
tpr,<br>
color=‘darkorange’,<br>
linestyle=‘-’,<br>
lw=2,<br>
label= ‘Unet (AUC = {0:0.2f})’<br>
‘’.format(roc_auc))<br>
plt.plot([0, 1], [0, 1], color=“navy”, lw=3, linestyle=‘–’)<br>
plt.xlim([0.0, 1.0])<br>
plt.ylim([0.0, 1.0])<br>
plt.xlabel(‘False Positive Rate’)<br>
plt.ylabel(‘True Positive Rate’)<br>
plt.title(‘ROC Curve’)<br>
plt.legend(loc=“lower right”)<br>
plt.show()</p>
<p>dev = torch.device(‘cuda:0’ if torch.cuda.is_available() else ‘cpu’)<br>
test_loader = get_test_loaders()</p>
<p>model = torch.load(opt.path, map_location=‘cpu’)<br>
model.to(dev)<br>
model.eval()</p>
<p>with torch.no_grad():<br>
tbar = tqdm(test_loader)<br>
for batch_img1, batch_img2, labels, fname in tbar:</p>
<pre><code>    # Set variables for testing
    batch_img1 = batch_img1.float().to(dev)
    batch_img2 = batch_img2.float().to(dev)
    labels = labels.long().to(dev)
    
    # Get predictions
    cd_preds = model(batch_img1, batch_img2)
    cd_preds = cd_preds[-1] 
    _, cd_preds = torch.max(cd_preds, 1)
  
    ROC_Plot = roc(labels.data.cpu().numpy().flatten(), 
                     cd_preds.data.cpu().numpy().flatten())
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d40bda1080715c571cfe90d2123e58068703d0d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/d40bda1080715c571cfe90d2123e58068703d0d9" title="roc"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d40bda1080715c571cfe90d2123e58068703d0d9_2_507x500.png" alt="roc" data-base62-sha1="ufQCcp5a41MDgJzxPGD411w6kvn" width="507" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d40bda1080715c571cfe90d2123e58068703d0d9_2_507x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d40bda1080715c571cfe90d2123e58068703d0d9_2_760x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d40bda1080715c571cfe90d2123e58068703d0d9_2_1014x1000.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">roc</span><span class="informations">3536×3486 267 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
ONNX opset version 16

alinutzal: <p>Do you have an estimate on when the ONNX opset version 16 will be released?</p>
<p>I am especially interested in updates about the scatter_add operator<br>
<a href="https://github.com/pytorch/pytorch/issues/32960#issuecomment-1003945949" rel="noopener nofollow ugc">[ONNX] scatter_add may result in wrong ONNX · Issue #32960 · pytorch/pytorch (github.com)</a></p>
<p>Thank you,<br>
Alina</p>

justinchuby: <p>Support for opset 16 is currently being added</p>
Distribute application with detectron2 maskrcnn model

Sylvain_Ard: <p>Hi,<br>
I have an application wich use instance segmentation with detectron2 maskrcnn. I want to distribute it as a Windows program. I tried to convert it in C++ but it doesn’t work. I tried to use conda-pack but it say to me that it cannot pack the environment because there are editable packages :<br>
C:\detectron2-main<br>
I pack it without editable packages but I have an error that there is no module detectron2.<br>
How to distribute applications with detectron2 please ?<br>
Thank you<br>
Best regards</p>
Does .cpu() use the 'CurrentStream'?

Nanlalalalalal: <p>Hi,<br>
A)  Does the gpu2cpu copy(with pinned memory)   run in   Stream s2 or the default stream(stream 0)?</p>
<pre><code class="lang-python">line1 with torch.cuda.stream(s2):       
line2         s2.wait_stream(torch.cuda.current_stream())
line3         Tensor.to(device=cpu)    # bocked here
        .......
</code></pre>
<p>B)</p>
<pre><code class="lang-python">line1 with torch.cuda.stream(s2):       
line2         s2.wait_stream(torch.cuda.current_stream())
line3         Tensor = ...    
line4         Tensor.to(device="cpu",non_blocking=True)      &lt;-
line5         TensorB=Tensor...      
</code></pre>
<p>Does Tensor.to use Stream s2 in this case?  Shall we insert stream synchronization before or after line4?</p>
Dataloader bottleneck

Qmin-Lee: <p>def train(model, train_dataset):</p>
<pre><code>print("Starting training ........")

dataloader = DataLoader(train_dataset, batch_size=8)

train_loss = 0.0

batch_count = 0

model.train()

loading = time.time()

for batch in dataloader:

    print("data loader time :", time.time() - loading) 

    measure = time.time()

    optimizer.zero_grad()
</code></pre>
<p>this is the part of my train function.</p>
<p>The problem is : ‘for batch in dataloader’  this line takes much more time than I expected<br>
(It took 15 seconds)</p>
<p>Batch size is only 8.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e528437f486c551446c1fa271a12486815b3a770.png" data-download-href="https://discuss.pytorch.org/uploads/default/e528437f486c551446c1fa271a12486815b3a770" title="SmartSelectImage_2022-06-08-14-36-08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e528437f486c551446c1fa271a12486815b3a770_2_545x499.png" alt="SmartSelectImage_2022-06-08-14-36-08" data-base62-sha1="wHdBwuUbenyj8GCLreTQBte14AM" width="545" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e528437f486c551446c1fa271a12486815b3a770_2_545x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e528437f486c551446c1fa271a12486815b3a770_2_817x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/5/e528437f486c551446c1fa271a12486815b3a770.png 2x" data-dominant-color="262727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">SmartSelectImage_2022-06-08-14-36-08</span><span class="informations">980×899 81.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
this image is my custom dataset class.</p>
<p>I don’t think <strong>getitem</strong> function in the custom dataset does many things.<br>
self.encoded_context[“input_ids”][idx], self.encoded_question[“input_ids”][idx],               self.encoded_context[“start_positions”][idx] and  self.encoded_context[“end_positions”][idx]<br>
have (batch=8, 200), (batch=8, 120), (batch=8, 1), (batch=8, 1) shapes respectively.</p>
<p>In addition, tensors that getitem function returns don’t look that big…</p>
<p>I thought that the <strong>init</strong> function in the custom dataset would do like preprocessing, so <strong>getitem</strong> would not take that long.</p>
<p>Due to the spending 15 seconds making batch from dataloader,  I don’t think I use GPU efficiently, and it seems to cause a bottleneck</p>
<p>How can I solve this??</p>
Pytorch Blocks for no reason

CDoutzen: <p>I’m training a unet network using pytorch.<br>
At first it works well. Later I found the GPU util was not completely used so i follow the instrument to load data in cpu and moving it to gpu so as to enable pin_memory option, and i also change num_workers from 0 to 4.<br>
In this case the training blocks after 60~70 epochs.  It raises no exception and does not panic. It is just like “block” for unknown reason. this doesn’t happen before i alter the configuration above.<br>
this is the output of nvidia-smi when the training is blocked.<br>
±--------------------------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 470.129.06   Driver Version: 470.129.06   CUDA Version: 11.4        |<br>
|-------------------------------±---------------------±---------------------±-------------------------+<br>
| GPU  Name        Persistence-M   | Bus-Id        Disp.A     | Volatile Uncorr. ECC     |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M.  |<br>
|                               |                       |                                                             MIG M. |<br>
|================+======================+==================== =|<br>
|   0  NVIDIA GeForce …  Off       |         00000000:01:00.0  On  |                      N/A |<br>
| 41%   41C    P8    10W / 225W  |   2098MiB /  7981MiB           |     26%      Default|<br>
|                                                    |                                               |                       N/A |<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|             ID   ID                                                                           Usage      |<br>
|======================================================|</p>
<p>|    0   N/A  N/A      5151      C   …/envs/pytorch/bin/python3     1429MiB |</p>
<p>as seen, the GPU memory still occupied by the model, but GPU-util decreases to 30% and lower, which is 90%+ when it runs normally.  And the training epoch stops to output. It seems training is blocked in this epoch, but i can’t figure out what’s the problem<br>
what kind of state is the program in this case? how can i fix it ?<br>
thanks if anyone can help</p>
CUDA_VISIBLE_DEVICES not effective on windows?

seongmin: <p>I believe this problem is Windows specific, as I have run the same test on Ubuntu 18.04 without problem.</p>
<h2>
<a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h2>
<p>Even after <code>os.environ['CUDA_VISIBLE_DEVICES'] = ''</code>, torch still uses CUDA GPU.</p>
<p>I expect <code>RuntimeError: No CUDA GPUs are available</code> with the following command:</p>
<pre data-code-wrap="sh"><code class="lang-nohighlight">python -c "import os; os.environ['CUDA_VISIBLE_DEVICES']=''; import torch; torch.randn(5).cuda()"
</code></pre>
<p>However, the command exists without error.</p>
<p>On the other hand, setting CUDA_VISIBLE_DEVICES in <code>cmd</code> works properly:</p>
<pre data-code-wrap="sh"><code class="lang-nohighlight">set CUDA_VISIBLE_DEVICES="" &amp; python -c "import torch; torch.randn(5).cuda()"
-&gt; RuntimeError: No CUDA GPUs are available
</code></pre>
<h2>
<a name="running-environment-2" class="anchor" href="#running-environment-2"></a>Running environment</h2>
<p>Windows 10<br>
Python 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32<br>
PyTorch 1.10.2<br>
CUDA 11.3</p>
<h2>
<a name="what-i-have-tried-3" class="anchor" href="#what-i-have-tried-3"></a>What I have tried:</h2>
<p>The problem occurred to me when I tried pytest’s <code>monkeypatch.setenv</code>.<br>
I tried <code>os.putenv</code> directly which seems to be used internally by <code>os.environ</code>, still unsuccessful.<br>
Then I followed through <a href="https://docs.python.org/3/using/windows.html" rel="noopener nofollow ugc">Python on Windows documentation</a> without solution.<br>
I found no help on <a href="https://pytorch.org/docs/stable/notes/windows.html" rel="noopener nofollow ugc">PyTorch Windows FAQ</a> either.</p>
Torch inplace Wrong ? No Relu and No +=,*=

sev777: <pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1]] is at version 278; expected version 277 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
python-BaseException
</code></pre>

AlphaBetaGamma96: <p>Can you share a minimal reproducible example? An error message by itself means nothing, and having an example code will help identify the problem.</p>
<p>Also, have you tried running your code within the <code>torch.autograd.set_detect_anomaly(True)</code> context manager like the <code>RuntimeError</code> suggests?</p>

sev777: <p>Thanks，torch.autograd.set_detect_anomaly(True) and  with torch.autograd.set_detect_anomaly(True) help me located the error~</p>
How to deallocate the DDP gradient buckets?

Di_Wu1: <p>How can I deallocate the DDP gradient buckets to reduce memory footprint?</p>
<p>In my model, I want to use customized buckets for gradient bucketing instead of DDP gradient buckets. Can I deallocate DDP gradient buckets that are roughly the same size as the model parameters?</p>
<p>Thanks.</p>

wanchaol: <p>Curious how you do the custom bucketing, maybe you are using ddp comm hook to implement a custom bucketing? One suggestion maybe using views of gradient bucket, instead of doing clones, this might help saving some memory.</p>
Cuda out of memory when setting "stride of 1"

tldrafael: <p>I’m handling an architecture that uses ResNet50 as the backbone on Colab, when I try to train with stride of 1 on the first conv7x7 layer (There’s 3 input channels and 64 output channels) it raises CUDA out of memory, although the code runs pretty well for stride of 2 (the default behavior) for a batch size of 8. I tested until the batch size of 2 for a stride of 1, and I couldn’t run it.</p>
<p>I’m curious why stride would affect the CUDA usage, I’m totally unaware of what happens under the hood on GPU computation. But isn’t the memory usage independent of stride? Is there anything that I could do to run this layer with stride of 1?</p>
<p>EDIT:<br>
Is there a count relation (batch_size, kernel_size, in_channels, out_channels, stride, input_resolution) with the loaded memory?</p>

eqy: <p>If the stride is decreased, then the spatial dimensions of the output (as well as those of stored intermediate activations during training) will increase and incur additional memory usage.</p>

tldrafael: <p>I see, although I already computed higher input resolution there with a stride of 2, such the output feature map was a higher dimension than my current input dimension and didn’t get any problem, that’s why It seems strange to me.</p>

eqy: <p>If you can post a standalone example of these two cases (e.g., stride 2 with higher resolution that should use a comparable amount of memory and the failing stride 1 case), that could be useful in understanding if there is a bug causing unexpected memory usage.</p>
Using pack_padded_sequence and torch.jit for GPU model

ackjay: <p>I have an RNN layer in a model that I wish to use <code>torch.jit.trace</code> on. When I have the model on CPU, it’s fine. However, when I load the model onto GPU, I face an error as the <code> RuntimeError: 0INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1646756402876/work/torch/csrc/jit/ir/alias_analysis.cpp":607, please report a bug to PyTorch. We don't have an op for aten::to but it isn't a special case.  Argument types: Tensor?, int, bool, bool, NoneType</code>.</p>
<p>I have located the source of the error as this line:</p>
<pre><code class="lang-auto">x = nn.utils.rnn.pack_padded_sequence(x, valid_frames.to(torch.device('cpu')), 
                                      batch_first=True, enforce_sorted=True)  
</code></pre>
<p><code>valid_frames</code> is a cuda tensor when the rest of the model is. I have tried a couple of different methods, trying to avoid the usage of <code>to</code> but none of them seem to work.</p>
<p>I am using torch 1.11.</p>

davidberard98: <p>Is valid_frames annotated as a <code>Optional[Tensor]</code>? If so then you would need to either change it to be a <code>Tensor</code>, or wrap the <code>.to()</code> casting in <code>if valid_frames is not None</code>.</p>
Multiple equals

shon711: <p>Hello,</p>
<p>I have tensor with values in <code>[-1, 0, 1, 2, 3, 4, 5]</code><br>
I want to create mask for each value.</p>
<p>So far I did:</p>
<pre><code class="lang-auto">cats =  [-1, 0, 1, 2, 3, 4, 5]]
masks = [labels == cat_id for cat_id in cats]
masks = torch.stack(masks, dim=1).int()
</code></pre>
<p>Do you know about faster and cleaner way to implement it?</p>
<p>Thanks in advance,<br>
Shon</p>
How to calculate class-wise pixel accuracy of Semantic Segmentation Model?

Atif_Butt: <p>I want to calculate class-wise pixel accuracy of my semantic segmentation model. Currently, I am computing MIoU using sklearn metrics (as shown in code snippet below). I shall be grateful I anyone help me out.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/175f838cd2ae85762a1c60b10f4d97b484adc44c.png" data-download-href="https://discuss.pytorch.org/uploads/default/175f838cd2ae85762a1c60b10f4d97b484adc44c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/175f838cd2ae85762a1c60b10f4d97b484adc44c.png" alt="image" data-base62-sha1="3kLCoJwHhp0hfCc9V0KroVz8GOU" width="472" height="499" data-dominant-color="323131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">556×588 17.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Exporting mt5 model to onnx, changes the output size of model

ori: <p>I am trying to export mt5 model to onnx. Since onnx model is large than 2GB, its required to export model via <code>use_external_data_format=True</code> in <code>torch.onnx.export</code>. Exporting model works, it saves weights into small binary files but i also see the output shape is changed. Can ideas to get output shape as pytorch model??</p>
Map the value in a tensor using dictionary

qiminchen: <p>Hi, I have a <code>4x4</code> tensor, for example <code>[[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]]</code> and I also have a mapping dict, like <code>{4: 5, 7: 10, 13: 10, 15: 1, 16: 1}</code>. Is there any way to map the value in the tensor using this dictionary efficiently?</p>

celviofos: <p>Hi, did you manage to find a way to do this?<br>
Thank you</p>

tj45794: <p>I do not know how efficient it is but it is readable by using the apply function. If you have the tensor <code>t</code> and the dictionary <code>d</code>, then simply write <code>t.apply_(d.get)</code>.</p>

Felix_Lessange: <p>I believe this is useful <a href="https://discuss.pytorch.org/t/cv2-remap-in-pytorch/99354/10" class="inline-onebox">Cv2 remap in pytorch? - #10 by Felix_Lessange</a></p>
Mapping values in a tensor

tdigt: <p>Hello,</p>
<p>I’m trying to map the values of a 1D Long CUDA tensor to the values given by a dictionary. For instance:</p>
<pre><code class="lang-auto">x = torch.tensor([3, 4, 2, 1, 3, 4]).to('cuda')
d = {'0':25, '1':4, '2':65, '3':33, '4':17}
y = dictionary_mapping_fn(x, d)
</code></pre>
<p>Then <code>y</code> should be:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; y
tensor([33, 17, 65, 4, 33, 17])
</code></pre>
<p>If it helps, the dictionary contains keys for all elements in between 0 and the maximum value in <code>x</code>. Also, the mapping does not need to be necessarily expressed as a dictionary but it can be given as a pair of 1D tensors (or even a single tensors with the values of the dictionary as we know they keys are just a range).</p>
<p>A similar question was asked in <a href="https://discuss.pytorch.org/t/map-the-value-in-a-tensor-using-dictionary/100933">here</a> with no solution available.</p>

mnicnc404: <p>It seems that you cannot do that on a cuda tensor; should move it to cpu before <code>apply_</code>:</p>
<pre><code class="lang-auto">val_if_not_shown = 0  # 0 can be any other number within dtype range
y.cpu().apply_(lambda val: d.get(val, val_if_not_shown)).to(y.device)
# in case that some of the values in `y` don't exist in `d.keys()`
</code></pre>
<p>Beware that it is an inplace operation.</p>

Felix_Lessange: <p>I think this is what you’re looking for <a href="https://discuss.pytorch.org/t/cv2-remap-in-pytorch/99354/10" class="inline-onebox">Cv2 remap in pytorch? - #10 by Felix_Lessange</a></p>
ONNX export - do not trace tensor size reading

tal123: <p>I am using Torch 1.3.1 and trying to export a model to ONNX.</p>
<p>Implementing an FPN-like module, I use an <code>Upsample</code> operation whose target size is determined by the previous feature map size.</p>
<p>The ONNX exporter traces the operation of reading the previous tensor size, so instead of taking the upsample target size as a constant, it translates the size reading into a messy ONNX graph. This is really unneeded, since given the fixed input size, the upsampling size is fixed too.</p>
<p>I even get this warning, when trying to get rid of the tracing using <code>detach</code> and converting into tuple. It still traces it:</p>
<pre><code>TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
</code></pre>
<p>How can I make the ONNX exporter not trace that operation and use the upsample size as a constant?</p>

tal123: <p>Just found an ugly workaround: after converting the tensor’s HW dimensions into a tuple, which still doesn’t get rid of the tracing, I now create another tuple that consists of the same values plus 0.2. Then I floor those values. That gets rid of the tracing, so the upsampling resize dimensions are treated as constants rather than ONNX operations, while preserving their values. <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20"></p>
What is the AUPRO metric?

Miguel_Campos: <p>Hello guys.</p>
<p>Suddenly I’ve been reading some papers which mention the AUPRO metric. I did a quick googling but did’t find anything that answered directly.</p>
<p>So the question is simple, what is the AUPRO? or where I could read the official paper or wherever where AUPRO is described?</p>
<p>Thanks</p>

Miguel_Campos: <p>Hello. No one? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
I am very interesting in knowing more</p>
Load ONNX model via torch.fx

Bone: <p>Assuming a setting:</p>
<ol>
<li>I have a model.onnx</li>
<li>I want to quant this model via a new quant algrithom(i.e. 4bit)</li>
</ol>
<p>The question: I want to use the torch.fx as IR to represent the model.onnx, has any API? or any methods?</p>
Fx tracing pytorch lightning models

gilfree: <p>I am working with pytorch lightning, and want to do some fx transformations on the code of the model.</p>
<p>The issue I face is that when tracing a model, you loose the original model instance and get a<br>
GraphModule instance. This has two effects:</p>
<ol>
<li>The resulting class loses all methods from of the pytorch lightning model (e.g. no <code>train_step</code>).</li>
<li>Any state in that class that is not part of the graph is lost.</li>
</ol>
<p>The above means that the traced model cannot be used in lightning.<br>
It seemed to me that the way to solve this is by Sub-classing GraphModule, which is a<br>
thing mentioned in a few places, but I couldn’t make it work.</p>
<ol>
<li>All the copy/serialization do not work properly.</li>
<li>The <code>__call__</code> wrapping and <code>__new__</code> implementation interact badly with inheritance.<br>
<a href="https://github.com/pytorch/pytorch/issues/63883" rel="noopener nofollow ugc">#63883</a> comes up very fast as you get<br>
a class that is not <code>GraphModuleImpl</code>.</li>
</ol>
<p>Of course it is possible to rewrite all the copy methods, <code>__new__</code> and <code>__call__</code><br>
but this will duplicate most of <code>GraphModule</code> code, and moreover - be very brittle to<br>
implementation changes inside <code>GraphModule</code></p>
<ul>
<li>Did anyone manage to properly subclass <code>GraphModule</code>?</li>
<li>Is there any other way to trace pytorch lightning model, or any<br>
other model with non graph state for that matter?</li>
</ul>
Can't save jit model with forward hook and custom function using torch.jit.script

Sally: <p>Hi all, I’m recently learning to convert a model into jit model since I want to load a model without source class. And I use forward hook to extract middle layer outputs as features and then try to return the modified features, the code looks like follows:</p>
<pre><code class="lang-auto">from typing import Tuple, List, Callable

class MyClass(nn.Module):
    features: List[torch.Tensor]

    def __init__(self):
        super(MyClass, self).__init__()
        self.features = []
        self.model = torchvision.models.resnet50()
        self.model.layer3[-1].register_forward_hook(self.extract_feature())
        self.model.layer3[-1].register_forward_hook(self.extract_feature())

    def extract_feature(self) -&gt; Callable:
        def hook(module, input: Tuple[torch.Tensor], output):
            self.features.append(output)
        return hook_t

    def __some_operation(self):
        self.result = torch.cat((self.features[0], self.features[1]), 1)

    def forward(self, x):
        self.features = []
        _ = self.model(x)
        self.__some_operation()
        return self.result

my_model = MyClass()
my_script = jit.torch.scipt(my_model)
my_script.save("my_jit_model.pth")
</code></pre>
<p>but I got this error:</p>
<pre><code class="lang-auto">attribute lookup is not defined on python value of type 'MyClass':
            self.features.append(output)
            ~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>Sorry I’m not familiar with jit and just getting started, I would appreciate it if anyone could help me figure out the mistakes I make, thanks!</p>
CUDA enabled torch

nby: <p>Hello everyone,<br>
I’ve got an issue with CUDA enabled pytorch. I’ve tried so many times to install from here : <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>. But I always get this kind of error messages : " Could not find a version that satisfies the requirement torch==…"<br>
I know this is a common problem but  I need to find a solution.</p>

ptrblck: <p>Check if your current OS and Python versions (3.7-3.10) are supported.<br>
If so, try to update your <code>pip</code> and <code>conda</code> base environment and try to install the wheels again.</p>

nby: <p>Actually I’m not using conda. Python and pip versions are up to date. I faced up this kind of error so many times.</p>

ptrblck: <p>What was the solution the last times? Since you seem to hit the issue repeatedly, I would guess that your base environment might be running into the same issue?</p>

nby: <p>I solved with docker or other enviroment software tools. But this time I have to work with local enviroment.</p>
Initializing weights of each module within a Sequential module in Libtorch

JamesDickens: <p>Supposing in a struct inheriting from torch::nn::Module I have a registered sequential module like</p>
<pre><code class="lang-auto">branch1 = register_module("branch1", torch::nn::Sequential(torch::nn::Conv2d(torch::nn::Conv2dOptions(in_channels, branch_channels, kernel_size).padding(0)),
				torch::nn::BatchNorm2d(torch::nn::BatchNorm2dOptions(branch_channels)), 
				torch::nn::ReLU());
</code></pre>
<p>I am interested in applying a weight initialization function to each component separately, say a function that takes in a torch::nn::Module or a pointer to a torch::nn::Module, what is the simplest way to achieve this?</p>
Question about single node multi gpu set-up/ DDP questions

hexken: <p>I have a single model that I would like to train on a single node with N gpus. Currently I am using DDP, however, I suspect I may not be using it in an ideal way, or, perhaps a different method of distributing the computation would be better for my use case.</p>
<p>Each process gets a subset of a fixed task set. For each of these tasks, the process uses the model (inside a no_grad context manager) to generate a variable number of training examples which are then used to compute a loss and run a backprop. Each process accumulates gradients for B tasks (recall each task generates a variable number of training examples) before calling optimizer.step().</p>
<p>To reduce synching overhead I am currently computing the losses with model.requires_background_grad_sync=False for all examples except the last example generated from every Bth task.</p>
<p>First, I am wondering if this is even a good use case for DDP, given that I am explicitly turning off the synching and only require it, on average, once every B*(average # of examples generated for each task) calls to backward()?</p>
<p>Second, do I actually need to call forward() AND backward() (with required_background_grad_sync=True) on each submodule to have that particular submodule synch’d.</p>
<p>Third, Is it okay to have requires_background_grad_sync=True, call forward() on a submodule, set requires_background_grad_sync=False immediately after, then call forward() and backward() on some tensors depending on the outputs of this submodule, then set requires_background_sync=True and call forward() and backward() once more on Tensors depending on the outsput of the submodule. Will this submodule be properly synch’d? with the gradients accumulated over all of these backward() calls?</p>

wanchaol: <p>Thanks for posting the question <a class="mention" href="/u/hexken">@hexken</a> After reading your question, I got lost on what you are trying to do here. What are <code>B</code> and <code>requires_background_grad_sync</code> mean?</p>
<p>In general, I think if your program does not match DDP pattern, i.e. there’s no data paralellism as it seems you are generating variable length of data instead of batched data, and don’t do gradient sync after every backward, you probably don’t want to go with DDP, unless there’s a specific reason. In your case you can just write a multiprocess training script, where you manages the data and the gradient sync/update by calling allreduce whenever you need to.</p>
What is the difference between dist.all_reduce_multigpu and dist.all_reduce

Rakshith_V: <p>In single-node multi-gpu setup I have used dist.all_reduce. Will it work for multi-node multi-gpu setup ? or should I have to use dist.all_reduce_multigpu ?<br>
In general, what;s the difference between both ?</p>

wanchaol: <p>Thanks for posting the question <a class="mention" href="/u/rakshith_v">@Rakshith_V</a>. multigpu version of collectives are being used when you have one rank manages multigpus, say if you have 2 nodes, each with 8 gpus, but you construct a world_size=2 processes, each process manages 8 gpus, you might need these collectives to do operations.</p>
<p>But if you construct a world_size = 16 processes, you can use <code>all_reduce</code> instead of <code>all_reduce_multigpu</code> for sure.</p>
<p>see the doc here <a href="https://pytorch.org/docs/stable/distributed.html#multi-gpu-collective-functions" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.11.0 documentation</a></p>
Sparse=True Error in distributed training

rajamohan_reddy: <p>RuntimeError: Cannot access data pointer of Tensor that doesn’t have storage</p>
<p>I am training model in using distributeddataparallel when I set sparse=true in nn.embedding. I am getting above error.</p>
<p>please help me in resolving above error<br>
I am training model via Sagemaker pytorch estimator and smddp as backend</p>

wanchaol: <p>Thanks for posting the question <a class="mention" href="/u/rajamohan_reddy">@rajamohan_reddy</a> This seems not a DDP problem, but an optimizer problem. As you can see from the note of the doc, sparse gradients only supported by certain optimizers (not all optimizer). Did you specify some optimizers that does not support sparse gradients? <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#torch.nn.Embedding" class="inline-onebox" rel="noopener nofollow ugc">Embedding — PyTorch 1.11.0 documentation</a></p>
How can I improve? Is this correct?

aaditkapoor1201: <p>I have a dataset of 1400 rows. My model takes in two inputs (10 dimension numeric data and 40 dimension embeddings of an image or sound). Following is my architecture:</p>
<pre><code class="lang-auto">from torch.nn import functional as F
from sklearn.metrics import accuracy_score
import torchmetrics
class HeartModel(pl.LightningModule):
  def __init__(self):
    super().__init__()

    self.accuracy = torchmetrics.Accuracy()

    self.model1 = nn.Sequential(
            nn.Linear(10, 8),
            nn.ReLU(),
            nn.Linear(8, 4),   
    )
    self.model2 = nn.Sequential(
            nn.Linear(40, 64),
            nn.ReLU(),
            nn.Linear(64, 32),   
            nn.ReLU(),
            nn.Linear(32, 4),   
      )
    self.last = nn.Sequential(
        nn.Linear(8, 10),
        nn.ReLU(),
        nn.Linear(10, 1),
    )


  def forward(self, input1, input2):
    f1 = self.model1(input1)
    f2 = self.model2(input2)
    combined = torch.cat((f1.view(f1.size(0), -1),
                          f2.view(f2.size(0), -1)), dim=1)
    out = self.last(combined)
    return out

  def test_step(self, batch, batch_idx):
    x1,x2, y = batch
    y_hat = self(x1, x2).float()
    loss = F.binary_cross_entropy_with_logits(self(x1, x2).float(), y.view(y.size(0), -1).float())
    self.log('test_loss', loss.item()) # Automatic aggregation in the background
    return loss
  

  def training_step(self, batch, batch_nb):
    x1,x2, y = batch
    preds = self(x1, x2).float()
    loss = F.binary_cross_entropy_with_logits(self(x1, x2).float(), y.view(y.size(0), -1).float())
    return loss
  
  def predict_step(self, batch, batch_idx):
    x1,x2, y = batch
    preds = self(x1, x2).float()
    c = []
    for i in preds:
      if i &lt; 0.5:
        c.append(0)
      else:
        c.append(1)
    
    acc = accuracy_score(y, c)
    return acc


  def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)


m = HeartModel()
from pytorch_lightning import LightningModule, Trainer
from pytorch_lightning.callbacks.progress import TQDMProgressBar


trainer = Trainer(
    auto_lr_find=True,
    accelerator="auto",
    devices=1 if torch.cuda.is_available() else None,  # limiting got iPython runs
    max_epochs=100,
    callbacks=[TQDMProgressBar(refresh_rate=20)],
)
trainer.fit(m, train_dataloader)
</code></pre>
<p>The training loss is 0.68 and testing is 0.69.</p>
<p>I am getting an accuracy of 47%. How can I improve? Is the loss function correct?</p>
<p>Thanks</p>
Combining the output of 3 Bayesian Convolutional NNs

Maryam_Osama: <p>I tried to implement 3 Bayesian NN simultaneously each outputs a tuple, however when I combined their results using torch cat I got the error: ValueError: only one element tensors can be converted to Python scalars.<br>
my code is as following:</p>
<blockquote>
<p>self.bcnn1=nn.Sequential(bcnn1d(in_channels=self.in_channels, out_channels=8,<br>
kernel_size=5, padding=2, dilation=1),<br>
BN(num_features=8))<br>
self.bcnn2=nn.Sequential(bcnn1d(in_channels=self.in_channels, out_channels=8,<br>
kernel_size=5, padding=6, dilation=3),<br>
BN(num_features=8))<br>
self.bcnn3=nn.Sequential(bcnn1d(in_channels=self.in_channels, out_channels=8,<br>
kernel_size=5,padding=12, dilation=6),<br>
BN(num_features=8))<br>
bcnn_out = self.bcnn(torch.cat(bcnn1, bcnn2, bcnn3),dim=1)</p>
</blockquote>
<p>however, I tried co convert each one onf the bcnn to a tensor but got <code>ValueError: only one element tensors can be converted to Python scalars</code>:</p>
<blockquote>
<p>bcnn_out = self.bcnn(torch.cat(torch.Tensor(bcnn_out1), torch.Tensor(bcnn_out2), torch.Tensor(bcnn_out3))</p>
</blockquote>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/typeerror-expected-tensor-as-element-0-in-argument-0-but-got-int/107802/9">here</a> with follow-up.<br>
Let’s please keep the discussion in one place (here) to avoid re-debugging the same issue.</p>
Transfer Learning using a PreTrained Model

Tayeba_Qazi: <p>Hi,<br>
I am implementing transfer learning using a pretrained weights. I am doing only feature extracting and not fine tuning the entire model. I have initialized the linear layer on a ResNet50 pretrained model.<br>
I have video inputs .<br>
I get the following runtime error:<br>
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 5-dimensional input of size [30, 60, 3, 224, 224] instead.</p>
<p>(batch size is 30, sample duration(temporal duration of inputs) is 60)</p>
<p>Can anyone pls help me resolve the issue?</p>

ptrblck: <p>Since you are also changing the input shape (from the original 4-dimensional inputs to 5-dimensional ones), you would need to change at least the first conv layer (e.g. by replacing it with <code>nn.Conv3d</code>, which accepts 5-dimensional inputs). However, I’ve mentioned “at least” since the output of this new conv layer would also be 5-dimensional, so if you want to keep this activation shape you would need to replace the entire feature extractor.<br>
I’m not familiar with your use case, but maybe iterating the input could also work, i.e. indexing the input in the temporal dimension, passing each slice to the model, and concatenating it afterwards before feeding it to the classifier.</p>

Tayeba_Qazi: <p>Thanks ptrblck. It helped.</p>