akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the best practice to save/load state_dict when enable `use_orig_params=True` in FSDP?

HAOCHENYE: <p>I can save/load the <code>state_dict</code> of model and optimizer successfully in <code>StateDictType.FULL_STATE_DICT</code> mode. However, I prefer the way to save checkpoint in <code>LOCAL_STATE_DICT</code> mode, which can free me from unsharing the parameters when saving the <code>state_dict</code> and sharding the parameters when loading the <code>state_dict</code>. Is it possible to do this when I enable <code>use_orig_params=True</code>?</p>
How to allocate more memory to pytorch

Kazuma_Sato: <p>Hi, i keep getting Cuda out of memory errors, i have a 3090 with 24gb of vram, cuda only allocates 7gb, 15gb is always free.</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 92.00 MiB (GPU 0; 24.00 GiB total capacity; 6.90 GiB already allocated; 14.90 GiB free; 6.98 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
<p>i also checked nvidia smi if any other process was taking up memory, found none<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b65327112318fb01152063cc20df270183ee6a51.png" data-download-href="https://discuss.pytorch.org/uploads/default/b65327112318fb01152063cc20df270183ee6a51" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b65327112318fb01152063cc20df270183ee6a51.png" alt="image" data-base62-sha1="q0V3iIAGKUjq4okXtLAOGBXnfpL" width="690" height="488" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">704×498 13.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
this was peak mem usage before it gave me the error</p>
<p>any help?</p>
Deadlock when using multiprocessing

he_xiangdong: <p>Hello!</p>
<p>I try to run the code below:</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
import torch
from torch import nn
import numpy as np

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.vars = nn.ParameterList()
        
        print("Net init 0")
        weight = nn.Parameter(torch.nn.init.orthogonal_(torch.zeros([64, 64]), 1.4))
        print("Net init 1")
        bias = nn.Parameter(torch.nn.init.constant_(torch.zeros(64), 1.4))
        print("Net init 2")
        self.vars.extend([weight, bias])
        print("Net init 3")
        
def f():
    net = Net()   
    
if __name__ == "__main__":
    agent = Net()
    processes = [mp.Process(target=f) for _ in range(2)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()
</code></pre>
<p>It seems to have encountered a deadlock. The init of Net in subprocesses doesn’t continue to print “Net init 1” and the program is stuck:</p>
<pre><code class="lang-auto"># python3 test.py 
Net init 0
Net init 1
Net init 2
Net init 3
Net init 0
Net init 0
</code></pre>
<p>However, if I comment the 1st line in <strong>main</strong>:</p>
<pre><code class="lang-auto"># agent = Net()
</code></pre>
<p>It works fine.</p>
<pre><code class="lang-auto"># python3 test.py 
Net init 0
Net init 0
Net init 1
Net init 2
Net init 3
Net init 1
Net init 2
Net init 3
</code></pre>
<p>This problem is too strange!</p>

eqy: <p>It seems that this might be a known issue: <a href="https://github.com/pytorch/pytorch/issues/21956" class="inline-onebox" rel="noopener nofollow ugc">nn.init.orthogonal_ doesn't work with multiprocessing · Issue #21956 · pytorch/pytorch · GitHub</a></p>
FSDP Hybrid Shard Mode - is it feature complete?

orchidmajumder: <p>Hi,<br>
I wanted to know if the HYBRID_SHARD mode of FSDP has feature parity at this point with respect to FSDP with FULL_SHARD or SHARD_GRAD_OP mode. For a given model (CLIP-like), right now, for me, saving optimizer state_dict in the following fashion does not work in HYBRID mode, but works in FULL_SHARD. It fails with an index-out-of-bound error.</p>
<p>FSDP.set_state_dict_type(<br>
module=self.model,<br>
state_dict_type=StateDictType.FULL_STATE_DICT,<br>
optim_state_dict_config=FullOptimStateDictConfig(rank0_only=True, offload_to_cpu=True),<br>
state_dict_config=FullStateDictConfig(rank0_only=True, offload_to_cpu=True),<br>
)<br>
state_dict = FSDP.optim_state_dict(self.model, optimizer)</p>
<p>I wanted to know if this is expected or if I need to something different for the HYBRID as opposed to FULL shard. Thanks in advance and really appreciate all the great work you have done on FSDP and in PyTorch in general.</p>

agu: <p>Could you file an issue on Github with a way to reproduce the issue?</p>

orchidmajumder: <p>Thanks, I’ll do it. The error only happens in a multi-node setup, it works fine on single node.</p>
Identifying subwindow of a PyTorch tensor with certain properties

WirryWoo: <p>Hello everyone,</p>
<p>I’m trying to identify a sub-window in <code>test</code> such that the sub-window yields the maximum sum of the averages of all other subwindows. (Averages defined with respect to each subwindow, sum representing along dimension 2 (the number of features), and max across dimension 1, the original sequence length).</p>
<p>Here’s my code:</p>
<pre><code class="lang-auto">import torch
test = torch.randn(3, 2307, 2)

window_len = 50

unfolded = test.unfold(len(test.shape) - 2, window_len, 1)
_, indices = torch.max(torch.sum(torch.mean(unfolded, dim = -1), dim = 2), dim = 1)

## Should match this result: Size: torch.Size([3, 50, 2])
torch.stack([
    unfolded[0, :, :, :][indices[0], :, :].T,
    unfolded[1, :, :, :][indices[1], :, :].T, 
    unfolded[2, :, :, :][indices[2], :, :].T
])
</code></pre>
<p>Is there a more efficient and straightforward way to get this solution using PyTorch’s functions?</p>
<p>Thanks,<br>
Wilson</p>
Loading trained model to make prediction of single image

uqhah: <p>I trained a UNET model using data loaders now I want to use the trained model to get predictions for a new random image of the size (256x256) single channel. How do I do that?<br>
I tried:</p>
<pre><code class="lang-auto">model.eval()
predb = model(torch.from_numpy(img).float().cuda())
</code></pre>
<p>But getting:</p>
<pre><code class="lang-auto">v.py:459, in Conv2d._conv_forward(self, input, weight, bias)
    455 if self.padding_mode != 'zeros':
    456     return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),
    457                     weight, bias, self.stride,
    458                     _pair(0), self.dilation, self.groups)
--&gt; 459 return F.conv2d(input, weight, bias, self.stride,
    460                 self.padding, self.dilation, self.groups)

RuntimeError: GET was unable to find an engine to execute this computation
</code></pre>

eqy: <p>That looks like a cuDNN error and would be unexpected. Could you provide a code snippet that reproduces the issue (e.g., by calling <code>F.conv2d</code> using random data with the same shape)?</p>

uqhah: <p>My model code is here incase that helps! <a href="https://discuss.pytorch.org/t/unet-model-on-gpu-runtime-error/175425" class="inline-onebox">UNET model on GPU runtime error</a></p>
Pytorch nn.dataparallel

Michael_Lempart: <p>Hi,</p>
<p>I have access to a GPU cluster and can use up to 4 GPU cards with 20GB each, or a single GPU with 80GB.</p>
<p>Iam wondering if training on multiple GPUs using nn.datapralllel is the same as increasing the batch-size?</p>
<p>As far as I understand, nn.DistributedDataParallel splits a mini-batch into several smaller mini-batches for multi GPU training. E.g. if my batch size is 16 and I would train on 4 GPUs, each GPU would get a mini-batch containing 4 samples.</p>
<p>Now Iam somewhat unsure how the the computation is done. Is e.g. the loss computed for each mini-batch of 4 samples and then averaged over the GPUs?<br>
Then this would not be the same as computing the  loss on a single GPU with batch size 16.</p>
<p>Or is the loss over the 4 GPUs computed in the same way as it would be on a single GPU?<br>
Then what is preferable? Using 4 GPUs with a batch size of 4 each, or a single GPU with a batch size of 16?</p>
<p>Thanks in advance,</p>
<p>kind regards,</p>
<p>Mi</p>

tiramisuNcustard: <p>It is recommended to use nn.DistributedDataParallel (DDP) over the nn.DataParallel.</p>
<p>In DDP, if you have K gpus. then DDP will create K processes, copy the model to each K gpus and train each batch/gpu (batch size will be determined based on your system RAM and individual gpu RAM) and then gather the gradients from each of the K gpu and update the model in each gpu with the averaged gradients and continue with the training.</p>
<p>You determine the batch_size per GPU. Ideally, all the gpus will be the same type i.e. all of them are RTX 3070 or all of them are RTX 3060.</p>
<p>Just for pure simplicity, my personal preference will be to select a single card with 80GB RAM. DDP is multiprocess not multithreaded. You need a CPU with enough cores so that lack of cores (remember, multiprocess) does not become a bottleneck. Context switching is costly.</p>
<p>Code written for a single process training (1 gpu) will work on most ML machines (*most because you will need to satisfy the required libraries and software dependencies).</p>
<p>If your future machines don’t have multiple GPUs then code written using DDP will need to be modified.</p>
TransformerEncoder in eval mode: Different results depending on mask

scientia: <p>Please consider the following <code>TransformerEncoderLayer</code> which is used in two ways, with a full mask and a causal mask:</p>
<pre><code class="lang-python">src_length = 6
embedding_size = 12
batch_size = 1

transformer_encoder_layer = TransformerEncoderLayer(embedding_size, 2, 8, activation='gelu', norm_first=True)
transformer_encoder_layer.eval()

x = torch.rand((src_length,batch_size,embedding_size))

full_mask = torch.zeros((src_length,src_length))
transformer_encoder_layer(x, src_mask=full_mask)[:,:,-1]

# tensor([[-0.0359],
#         [ 0.3602],
#         [ 0.5291],
#         [ 0.2098],
#         [ 0.4393],
#         [ 0.4099]], grad_fn=&lt;SelectBackward0&gt;)

causal_mask = torch.triu(torch.full((src_length, src_length), float('-inf')), diagonal=1)
transformer_encoder_layer(x, src_mask=causal_mask)[:,:,-1]

# tensor([[-0.1079],
#         [ 0.4843],
#         [ 0.6760],
#         [ 0.1818],
#         [ 0.5266],
#         [ 0.4099]], grad_fn=&lt;SelectBackward0&gt;)
</code></pre>
<p>As expected, it returns the same results for the last element in the sequence since both in the causal and full masked case, it is able to attend to the whole sequence while for previous elements, the results differ.</p>
<p>However, where it gets unexpected is when I do the same with a <code>TransformerEncoder</code>:</p>
<pre><code class="lang-python">src_length = 6
embedding_size = 12
batch_size = 1

transformer_encoder_layer = TransformerEncoderLayer(embedding_size, 2, 8, activation='gelu', norm_first=True)
transformer_net = TransformerEncoder(transformer_encoder_layer, 6)
transformer_net.eval()

x = torch.rand((src_length,batch_size,embedding_size))

full_mask = torch.zeros((src_length,src_length))
transformer_net(x, mask=full_mask)[:,:,-1]

# tensor([[1.1491],
#         [1.3810],
#         [1.4263],
#         [1.8420],
#         [1.8647],
#         [1.9420]], grad_fn=&lt;SelectBackward0&gt;)

causal_mask = torch.triu(torch.full((src_length, src_length), float('-inf')), diagonal=1)
transformer_net(x, mask=causal_mask)[:,:,-1]

# tensor([[-0.6918],
#         [-0.2881],
#         [ 0.0460],
#         [ 0.1590],
#         [-0.4355],
#         [-0.2541]], grad_fn=&lt;SelectBackward0&gt;)
</code></pre>
<p>where you can see that I get totally different results, even for the last dimension. What is the reason for that and is there a way to avoid that?</p>

arunppsg: <p>It maybe due to the number of encoder layers. When I set the number of encoder layers to <code>1</code> in this line <code>transformer_net = TransformerEncoder(transformer_encoder_layer, 1)</code>, it seems to return same output for last dimension.</p>
Custom gradient calculation for tensor-like type

KPJoshi: <p>I am attempting to create a tensor-like class. I’ve been following the instructions at <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-with-a-tensor-like-type" rel="noopener nofollow ugc">extending torch with a Tensor-like type</a>. I have questions especially pertaining to gradient storage and calculation:</p>
<ol>
<li>I want to initialize my class from a (float) tensor, and be able to convert it back. I know I can retrieve the data using the <code>numpy()</code> function, but how do I get gradient data if I wish to store that too? When I convert back to tensor, how can I give it back the stored gradient data?</li>
<li>I saw the instructions for making functions like <code>torch.add</code> work with my custom type, but I will also need to modify how the gradient is calculated by autograd. How do I define both the custom forward and backward versions of <code>torch.add</code>? I saw the instructions for <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" rel="noopener nofollow ugc">extending torch.autograd</a> with a custom function but I am not sure if that feature can be brought over to this case.</li>
</ol>
<p>Any help is appreciated. Thank you!</p>

Raul_Murillo: <aside class="quote no-group" data-username="KPJoshi" data-post="1" data-topic="153824">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kpjoshi/48/50436_2.png" class="avatar"> KPJoshi:</div>
<blockquote>
<p>gradient storage and calculation</p>
</blockquote>
</aside>
<p>Hi, I am facing the same problem. Did you find out how to solve this?<br>
I also tried <a href="https://pytorch.org/docs/stable/notes/extending.html#subclassing-torch-tensor" rel="noopener nofollow ugc">Subclassing <code>torch.Tensor</code></a>, but the problem persists when trying to get gradients, for example with <code>torch.nn.Parameter()</code>, since it inherits directly from <code>torch.Tensor</code>.</p>

KPJoshi: <p>Hi,</p>
<p>I could not get it to work through PyTorch. I actually ended up writing my own PyTorch-like framework that was sufficient for my needs. You can find it on <a href="https://github.com/KPJoshi/Fixed-Point-RNN-Training" rel="noopener nofollow ugc">https://github.com/KPJoshi/Fixed-Point-RNN-Training</a>. In particular, take a look at fxpTensor.py. You can try adapting and/or extending this as you wish, and feel free to ask me if you have any questions.</p>
Install torch2.0 from source

fiona: <p>Hi, when I want to compile torch2.0 from source, I have met some problems.<br>
The following is the bug information.<br>
pytorch/c10/cuda/CUDACachingAllocator.cpp:1152:15: error: ‘nvmlProcessInfo_v1_t’ was not declared in this scope; did you mean ‘nvmlProcessInfo_t’?<br>
1152 |   std::vector&lt;nvmlProcessInfo_v1_t&gt; procs(8);<br>
|               ^~~~~~~~~~~~~~~~~~~~<br>
|               nvmlProcessInfo_t<br>
/pytorch/c10/cuda/CUDACachingAllocator.cpp:1152:35: error: template argument 1 is invalid<br>
1152 |   std::vector&lt;nvmlProcessInfo_v1_t&gt; procs(8);<br>
|                                   ^<br>
/pytorch/c10/cuda/CUDACachingAllocator.cpp:1152:35: error: template argument 2 is invalid<br>
/pytorch/c10/cuda/CUDACachingAllocator.cpp:1153:29: error: request for member ‘size’ in ‘procs’, which is of non-class type ‘int’<br>
1153 |   unsigned int size = procs.size();<br>
|                             ^~~~<br>
/pytorch/c10/cuda/CUDACachingAllocator.cpp:1156:41: error: request for member ‘data’ in ‘procs’, which is of non-class type ‘int’<br>
1156 |               nvml_device, &amp;size, procs.data())) ==<br>
|                                         ^~~~<br>
/pytorch/c10/cuda/CUDACachingAllocator.cpp:1158:11: error: request for member ‘resize’ in ‘procs’, which is of non-class type ‘int’<br>
1158 |     procs.resize(size);<br>
|           ^~~~~~<br>
/pytorch/c10/cuda/CUDACachingAllocator.cpp:1165:23: error: invalid types ‘int[unsigned int]’ for array subscript<br>
1165 |     auto&amp; proc = procs[i];<br>
|                       ^</p>
<p>Who can help me for this?</p>

ptrblck: <p>Double post from <a href="https://github.com/pytorch/pytorch/issues/100618">here</a>.</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (984x1 and 4x512)

KUSHAL_SHAH: <pre><code class="lang-auto">def forward(self,src:Tensor,tgt:Tensor,src_mask:Tensor=None,tgt_mask:Tensor=None) -&gt; Tensor:
        
        src = self.encoder_input_layer(src)
        src = self.positional_encoding_layer(src)
        src = self.encoder(src)
        
        decoder_output = self.decoder_input_layer(tgt)
        
        decoder_output = self.decoder(
            tgt = decoder_output,
            memory = src,
            tgt_mask = tgt_mask,
            memory_mask = src_mask
        )
        
        decoder_output = self.linear_mapping_layer(decoder_output)
        
        return decoder_output
    
</code></pre>
<p>I have been getting dimensionality error from quite a long time and this is my forward method for the model.</p>

ptrblck: <p>It’s unclear which layer raises the shape mismatch error but based on the error message it should come from a linear layer. You would have to check the stacktrace and isolate which layer fails. Once this is done make sure the input activation has the expected number of features or adapt the linear layer and change its <code>in_features</code> attribute to the feature dimension of the input activation.</p>
Program stuck trying to print torchvision.ops.roi_align

sbht_sbht: <p>I’m use print to debug my faster r-cnn model and make some control points. First epoch doesn’t make any problem. All work. But when start 2-nd epoch model stuck trying to print roi alight output. It doesn’t raise any error or masseges, doesn’t stop code execution. And after very long time it’s continue without any trouble and next epochs there is no problem again. What could be the reason for this?</p>
<p>I forgot to mention that when using torchvision.ops.roi_pool there are no learning problems</p>
<p>Edit 2: I noticed, that sometimes it doesn’t take much time, but most of the time learning just doesn’t go on</p>
Optimizing biases for custom feature generation function

Jannis1: <p>Hello together,</p>
<p>I created a custom function that generates features for univariate time-series datasets. I use these features in a Ridge Classification model afterwards. My custom function “generate_features” receives multiple inputs but i only want to train the biases to improve the performance of my model. Below you can see my code so far. The code is running but print(model.biases.grad) gives “None” and the biases are not updated. What did i do wrong?<br>
For clarification: In each epoch the ridge classifier is trained with the new features that were calculated with the new biases. Dont be confused by why i dont use X_train. In this version i use predefined initialization variables for the feature creation instead of using X_train for it.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

from sklearn.utils.extmath import softmax
class RidgeClassifierCVwithProba(RidgeClassifierCV):
    def predict_proba(self, X):
        p = self.decision_function(X)
        d_2d = np.c_[-p, p]
        return softmax(d_2d)

class features_ridge(torch.nn.Module):
    def __init__(self, weight_matrix, dilation_matrix, biases_matrix, top_x_indices, padding_matrix , C_alpha_matrix, C_gamma_matrix ):
        super(features_ridge, self).__init__()
        self.weights = torch.nn.Parameter(torch.tensor(weight_matrix), requires_grad=False)
        self.dilation = torch.nn.Parameter(torch.tensor(dilation_matrix), requires_grad=False)
        self.top_x_indices = torch.nn.Parameter(torch.tensor(top_x_indices), requires_grad=False)
        self.padding = torch.nn.Parameter(torch.tensor(padding_matrix), requires_grad=False)
        self.C_alpha_matrix = torch.nn.Parameter(torch.tensor(C_alpha_matrix), requires_grad=False)
        self.C_gamma_matrix = torch.nn.Parameter(torch.tensor(C_gamma_matrix), requires_grad=False)
        self.biases = torch.nn.Parameter(torch.tensor(biases_matrix), requires_grad=True)
        self.classifier_ridge = RidgeClassifierCVwithProba()
        
    def forward(self,x):
        features= generate_features(weights=self.weights, dilations =self.dilation, biases = self.biases, n_jobs = -1,top_x_indices = top_x_indices, padding = self.padding, C_alpha_matrix = self.C_alpha_matrix, C_gamma_matrix = self.C_gamma_matrix)
        x = self.classifier_ridge.predict_proba(features)
        x = torch.tensor(x, requires_grad=True)

        return x, features


loss_fn = torch.nn.CrossEntropyLoss()
target = torch.from_numpy(train_y).long()
model = features_ridge(weight_matrix = weight_matrix, dilation_matrix = dilation_matrix, biases_matrix = biases_matrix,top_x_indices = top_x_indices, padding_matrix = padding_matrix, C_alpha_matrix = C_alpha_matrix, C_gamma_matrix = C_gamma_matrix)
optimizer = torch.optim.SGD([model.biases], lr =0.01)
num_epochs=2

for epoch in range(num_epochs):
    model.classifier_ridge.fit(features, train_y)
    output, features = model(X_train)
    loss = loss_fn(output, target)
    optimizer.zero_grad()
    loss.backward()
    print(model.biases.grad)
    optimizer.step()
</code></pre>
<p>Many thanks before all,<br>
Jannis</p>
Issue importing functorch

Thomas_Pumir: <p>I tried importing functorch and torch in python 3.9.7</p>
<pre><code class="lang-auto">import torch
from functorch import vmap, grad
</code></pre>
<p>and ran into a circular import  issue.</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/var/folders/4p/5v2n8wpn58v6xw674tjlss680000gn/T/ipykernel_64234/1530828949.py in &lt;module&gt;
      1 import torch
----&gt; 2 from functorch import vmap, grad

/usr/local/lib/python3.11/site-packages/functorch/__init__.py in &lt;module&gt;
      5 # LICENSE file in the root directory of this source tree.
      6 import torch
----&gt; 7 from . import _C
      8 
      9 # Top-level APIs. Please think carefully before adding something to the

ImportError: cannot import name '_C' from partially initialized module 'functorch' (most likely due to a circular import) (/usr/local/lib/python3.11/site-packages/functorch/__init__.py)
</code></pre>
<p>Any way around it? Is this a bug in the packages?<br>
Submitting this here since I could not find an easy fix</p>

marksaroufim: <p>The functorch package has moved to core so you don’t need to have a seperate import instead you can use <code>torch.func.vmap</code> <a href="https://pytorch.org/docs/stable/generated/torch.func.vmap.html#torch.func.vmap" class="inline-onebox" rel="noopener nofollow ugc">torch.func.vmap — PyTorch 2.0 documentation</a></p>
Prediction integer-valued function

All_VODs: <p>I would like to create a model with pytorch_geometric, which outputs on each node, a function from the segment [0,1] to the integers. This function is supposed to count a number of neighboring edges at a given time, so it has integer values, and it is locally constant.</p>
<p>I wonder how to efficiently implement such an output, and which loss function to use. If I merely output a vector that gives the outputs over a sample of [0,1], I am not using the fact that my function will be locally constant, so I am working in too much generality.<br>
Any ideas?</p>
After one epoch of successful training, always get this error message. what could be the reason?

purvang_lapsiwala: <p>2023-05-01 15:16:45 | <strong>INFO</strong> | yolox.core.trainer:259 - <strong>epoch: 2/300, iter: 250/313, mem: 11174Mb, iter_time: 3.676s, data_time: 3.100s, total_loss: 8.0, iou_loss: 3.3, l1_loss: 0.0, conf_loss: 3.4, cls_loss: 0.9, seg_loss: 0.3, lr: 1.294e-03, size: 384, ETA: 4 days, 0:29:34</strong></p>
<p>Traceback (most recent call last):<br>
File “/opt/conda/lib/python3.9/runpy.py”, line 197, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/opt/conda/lib/python3.9/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/workspace/tools/train.py”, line 191, in <br>
launch(<br>
File “/workspace/yolox/core/launch.py”, line 82, in launch<br>
mp.start_processes(<br>
File “/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py”, line 198, in start_processes<br>
while not context.join():<br>
File “/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py”, line 140, in join<br>
raise ProcessExitedException(<br>
torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGKILL<br>
root@e7968c684346:/workspace# /opt/conda/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 8 leaked semaphore objects to clean up at shutdown<br>
warnings.warn('resource_tracker: There appear to be %d ’</p>

eqy: <p>I would monitor if the memory usage (on the host side) is growing during training and nearing the system limit as this could trigger the <code>SIGKILL</code>.</p>
<p>If you are seeing memory usage growing, I would check e.g., that you detaching losses before computing statistics/otherwise retaining a computation graph unnecessarily.</p>

purvang_lapsiwala: <p>Thanks <a class="mention" href="/u/eqy">@eqy</a> . Furthur understanding and verifying dataset, I found that I only get error when number of workers in dataloader is greater than 0. workers=0 works fine. I am using cv2 to resize images inside dataloader getitem method. I also tried restricting cv2.numthreads=0 and mp.set_start_method(‘spawn’, force=True) but not helping.<br>
Getting following error when increase workers &gt; 0.</p>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/opt/conda/lib/python3.9/multiprocessing/spawn.py”, line 116, in spawn_main<br>
exitcode = _main(fd, parent_sentinel)<br>
File “/opt/conda/lib/python3.9/multiprocessing/spawn.py”, line 126, in _main<br>
self = reduction.pickle.load(from_parent)<br>
_pickle.UnpicklingError: pickle data was truncated</p>
<p>/opt/conda/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 96 leaked semap<br>
hore objects to clean up at shutdown<br>
warnings.warn('resource_tracker: There appear to be %d ’</p>

eqy: <p>Just to be sure, I would also set environment variables for other backends like <code>OMP_NUM_THREADS</code> and <code>MKL_NUM_THREADS</code> and check if that makes a difference.</p>
Straight Through Estimator on Model Parameters

afterless: <p>I have been working on a PyTorch implementation of this <a href="https://arxiv.org/pdf/2209.04836.pdf" rel="noopener nofollow ugc">paper</a> and am implementing the Straight-through estimator algorithm on page 26.</p>
<p>Currently, I have the following code to calculate the midpoint parameters and store them in <code>pi_model_state_dict</code> and load this into a blank model called <code>pi_model</code> via <code>load_state_dict</code>:</p>
<pre><code class="lang-auto">for (
    p_m,
    (n, p_a),
    p_p,
) in zip(params_model.values(), params_a.items(), proj_params.values()):
    print(
        n, type(p_m), type(p_a), type(p_p)
    )  # parameter, parameter, tensor
    pi_model_state_dict[n] = 0.5 * (p_a + ((p_p - p_m).detach() + p_m))
    print(n, type(pi_model_state_dict[n]))  # tensor!

pi_model.load_state_dict(pi_model_state_dict)
optimizer.zero_grad() # optimizer = t.optim.Adam(model.parameters(), lr=args.lr)
output = pi_model(data)
loss = loss_fn(output, target)
loss.backward()
for p in model.parameters():
    print(f"grad time: {p.grad}") # All None!
</code></pre>
<p><code>params_model = {k: v for k, v in model.named_parameters()}</code> where <code>model</code> is a different module with same architecture as <code>pi_model</code> but the weights are loaded from a checkpoint. <code>proj_params</code> is the result of a non differentiable operation performed on params in <code>model.parameters()</code>.</p>
<p>Ideally, I want the computational graph to be that I can calculate the gradient from the loss on <code>pi_model</code> and see the gradients on <code>model.parameters()</code> but this doesn’t seem to be working since the <code>.grad</code> attribute on each parameter in <code>model.parameters()</code> is None. Is there any other attempt at a straight through estimator I can try to get this to work? Thank you so much!</p>

afterless: <p>Solved the issue, using <code>torch.func.functional_call</code> to change the functionality as follows worked:</p>
<pre><code class="lang-auto">for (
    p_m,
    (n, p_a),
    p_p,
) in zip(
    param_model.values(), param_a.items(), proj_params.values()
):
    pi_model_state_dict[n] = 0.5 * (p_a + ((p_p - p_m).detach() + p_m))

optimizer.zero_grad()
output = t.func.functional_call(pi_model, pi_model_state_dict, data)
...
</code></pre>
Error about tensorboard version

morteza_mahvelati: <p>I run my model without any problem, but newly i got an error :‘’ TensorBoard logging requires TensorBoard version 1.15 or above."<br>
i did some possible solutions such as upgrading tf, td, torch, … but the error has not gone yet!</p>

ptrblck: <p>What is <code>tensorboard.__version__</code> returning?</p>
Combining torchrun and torch.multiprocessing

Triptych: <p>Hi all,</p>
<p>I’ve worked myself into a bit of a pickle that I was hoping someone here might be able to help me out with. I have some torch code that for training requires using DDP through torchrun. At inference time, I have a process that I was able to build up through dataparallel, however the process itself became incredibly brittle, with repeated and consistent cuda blocking errors across two different systems. I was able to rebuild my inference time code through torch.multiprocessing, however running calling a function that establishes a torch.multiprocessing group within torchrun (even when calling it from local_rank = 0 only) hangs after attempting to establish the first process - I presume because you end up with two different processes trying to set and access local_rank on the backend.</p>
<p>I can call both functions independently, however I’d like to be able to wrap it all together into the one piece of code, and was wondering if anybody had any suggestions.</p>
<p>Hopefully an interesting enough question for those with knowledge of torchrun and torch.multiprocessing. Thanks in advance</p>

kumpera: <p>This is a good question for <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
How to double compression

nima_pw: <p>Hi,</p>
<p>My CNN-based model is trained on <code>jpg</code> images. I want to test my model on <code>double compression</code> test images to compare how accuracy changes.<br>
How do I do this?<br>
I resized test images to <code>320*320</code>, is it a <code>double compression</code>?</p>

J_Johnson: <p>JPEG compression is an algorithm that makes use of a discrete cosine transform and removing a set cutoff of frequencies, depending on the quality of compression chosen.</p>
<p>Here is one way you could make use of PIL and StringIO libraries to compress an image using the JPEG algorithm:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/30771652/how-to-perform-jpeg-compression-in-python-without-writing-reading">
  <header class="source">

      <a href="https://stackoverflow.com/questions/30771652/how-to-perform-jpeg-compression-in-python-without-writing-reading" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1245262/user1245262" target="_blank" rel="noopener nofollow ugc">
    <img alt="user1245262" src="https://www.gravatar.com/avatar/83503cef6b0550864d3a1c9ca5d87701?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/30771652/how-to-perform-jpeg-compression-in-python-without-writing-reading" target="_blank" rel="noopener nofollow ugc">How to perform JPEG compression in Python without writing/reading</a>
</h4>

<div class="tags">
  <strong>python, image, jpeg, python-imaging-library</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1245262/user1245262" target="_blank" rel="noopener nofollow ugc">
    user1245262
  </a>
  on <a href="https://stackoverflow.com/questions/30771652/how-to-perform-jpeg-compression-in-python-without-writing-reading" target="_blank" rel="noopener nofollow ugc">04:35AM - 11 Jun 15 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


nima_pw: <p>Dear <a class="mention" href="/u/j_johnson">@J_Johnson</a><br>
Thanks a lot.</p>
<p>Although I got a lot of errors because of the difference between Python 2 and 3, it was the right way.</p>
Ways to improve Pytorch's performance during inference using small model and small batches?

Igor_Vodopyanov: <p>Hi. I’m training a simple model (seq2seq RNN) and using it in inference mode to generate new batches for training from a stream of data using beam search. Beam search code uses a lot of slicing, indexing, conditional statements - everything there is being processed via Pytorch’s tensors and functions. Essentially, to generate new training batch it needs to run one RNN step multiple times for quite small batches. I’ve already wrapped it in torch.no_grad(), but what else can I do to improve performance?</p>
Using `torch.distributed.all_gather_object` returns error when using 1 GPU but works fine for multiple GPUs

seankala: <p>I’m currently using HuggingFace Accelerate to run some distributed experiments and have the following code inside of my evaluation loop:</p>
<pre><code class="lang-auto">model.eval()
device = accelerator.device
intermediate_value = {}
output = [None] * accelerator.num_processes

# Some evaluation code.

dist.all_gather_object(output, intermediate_value)
</code></pre>
<p>When I’m using multiple GPUs it’s fine, but when I’m using only one I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
</code></pre>
<p>What I’m wondering is, I thought that if you wrap your model, optimizer, etc. using the HuggingFace Accelerate module then you didn’t have to do <code>torch.distributed.init_process_group</code>? And if this is the case, then how come it’s not working when I only have 1 GPU?</p>
<p>Thanks in advance.</p>

ptrblck: <p>This question seems to be HuggingFace-specific so you might want to post it in their discussion board.</p>
PyTorch vision on Android with AMD GPU

KevinCain: <p>What is the most mature method of running inference for PyTorch models on a mobile AMD GPU? Specifically, I’m using an AMD Navi 2 having 4 WGP. Below are the few options I’m aware of, however I’d love to hear from those with direct experience who may suggest other options:</p>
<ul>
<li>Compiling PyTorch 2.0 for an AMD target: I gather this is possible, but early in development and intended more for desktop than mobile. Is that right?</li>
<li>Using PyTorch Mobile and TorchVision on Android, an established path but one currently without GPU support, let alone AMD GPU support. Is that correct?</li>
<li>Run inference on a PyTorch model on the Android GPU via <a href="https://github.com/pytorch/tutorials/tree/master/prototype_source" rel="noopener nofollow ugc">PyTorch on Vulkan Compute backend</a>, or <a href="https://medium.com/pytorch/pytorch-mobile-now-supports-android-nnapi-e2a2aeb74534" rel="noopener nofollow ugc">Android NNAPI</a>, or using libMACE via ONNX. Is it true that Vulkan Compute may arbitrarily send compute to CPU that we want ion the GPU instead?</li>
</ul>
<p>A few other notes follow in the hope they are helpful for background to my question.</p>
<p>PyTorch has mainly supported CUDA. While PyTorch AMD exists, converting from CUDA to Radeon Open Compute (ROCm) is notoriously difficult to use for real world deep learning. Similarly there are options for converting CUDA code to ROCm, which may not have been used yet in production.</p>
<p>PyTorch for AMD supports TorchVision and is ‘fully supported since ROCm 5.1 and upstreamed with PyTorch 1.12.’ but the project seems geared to servers, not mobile.</p>
<p>AMD support in PyTorch 2.0] may be more straightforward. PyTorch 2.0 includes torch.compile, via the Triton compiler, a language written in python that ‘provides much higher productivity than CUDA, but with the ability to beat the performance of highly optimized libraries like cuDNN with clean and simple code’:</p>
<p>“Triton can automatically optimize kernels generated by machine learning compilers such as TorchInductor for multiple AI accelerators including AMD Instinct GPU accelerator by leveraging hardware-specific features of the AMD CDNA™ GPU architecture.”</p>
<p>Before choosing to compile for AMD I ask:</p>
<ul>
<li>Does the AMD Navi 2 support code compiled by Triton?</li>
<li>Can we compile for AMD Navi 2 from a server with an nVIDIA GPU?</li>
</ul>
<p>For PyTorch Mobile, we need to adapt trained models by quantization and conversion to TorchScript, an intermediate representation of a PyTorch model. PyTorch optimization for mobile may help us run on AMD Navi 2, however GPU support is either nonexistent or very early.</p>
<p>Questions:</p>
<ul>
<li>Can PyTorch Mobile use the AMD Navi 2 GPU? Early prototype work has not been used widely.</li>
</ul>
CNN training for clustering

msr3384: <p>I am trying to compare different types of norms in terms of accuracy and using stopping criteria in terms of accuracy as well. For the same hyper-parameters setup one specific norm is working okay but for alternative norms my accuracy fluctuating as in the picture. Anyone can give me some idea why this types of behavior is seen for my model?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74916791c17b0db7cd40b8f66bf21fe70e27bff6.png" data-download-href="https://discuss.pytorch.org/uploads/default/74916791c17b0db7cd40b8f66bf21fe70e27bff6" title="Screenshot 2023-05-02 at 6.48.55 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74916791c17b0db7cd40b8f66bf21fe70e27bff6_2_403x500.png" alt="Screenshot 2023-05-02 at 6.48.55 PM" data-base62-sha1="gDcUH2kIUsns8SwWzCwMkLnnShE" width="403" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74916791c17b0db7cd40b8f66bf21fe70e27bff6_2_403x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74916791c17b0db7cd40b8f66bf21fe70e27bff6_2_604x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74916791c17b0db7cd40b8f66bf21fe70e27bff6_2_806x1000.png 2x" data-dominant-color="E2E2E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-05-02 at 6.48.55 PM</span><span class="informations">966×1198 137 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Any known workarounds for exporting nn.Transformer model to onnx?

Matt_Harrison: <p>Exporting a vanilla <code>nn.Transformer</code> via onnx, give me the error that <code>aten::unflatten</code> is not implemented. I see a few patches in the last few days related to this.</p>
<p>Without waiting for next release Is there a viable workaround?</p>

marksaroufim: <aside class="quote no-group" data-username="Matt_Harrison" data-post="1" data-topic="178337">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matt_harrison/48/59851_2.png" class="avatar"> Matt_Harrison:</div>
<blockquote>
<p><code>aten::unflatten</code></p>
</blockquote>
</aside>
<p>Which version of torch are you using? Looks like this was already fixed in nightlies <a href="https://github.com/pytorch/pytorch/issues/98190" class="inline-onebox">UnsupportedOperatorError: Exporting the operator 'aten::unflatten' to ONNX is not supported. · Issue #98190 · pytorch/pytorch · GitHub</a></p>

Matt_Harrison: <p>I’m using 2.0.0, how do I get nightlies?</p>

marksaroufim: <p>Go to <a href="http://pytorch.org">pytorch.org</a> and select nightlies, your os and desired cuda version <a href="https://pytorch.org/">https://pytorch.org/</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/56b4928b0c65e87b0dc94653fda7e1a8b7f67331.png" data-download-href="https://discuss.pytorch.org/uploads/default/56b4928b0c65e87b0dc94653fda7e1a8b7f67331" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56b4928b0c65e87b0dc94653fda7e1a8b7f67331_2_672x500.png" alt="image" data-base62-sha1="cn1W6gFcOK2t3FPr9dgnuzMKsU1" width="672" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56b4928b0c65e87b0dc94653fda7e1a8b7f67331_2_672x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56b4928b0c65e87b0dc94653fda7e1a8b7f67331_2_1008x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/6/56b4928b0c65e87b0dc94653fda7e1a8b7f67331.png 2x" data-dominant-color="F1E6E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1240×922 72.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Matt_Harrison: <p>Awesome, I’ll give it a try, thanks so much.</p>

Matt_Harrison: <p>Thanks, using the nightly solved the issue!</p>

DeepakSaini119: <p>Nightly build seems to be working for me without any error, but the export itself is extremely slow with opset version 16</p>
Problem Image classification or object detection task with yolo when model has good predictive ability on test set with transform.normalize but not working on video camera

LouisBao: <ul>
<li>
<p>After training my CNN or Resnet50 cutom, I have good predictive performance on test set when I split my dataset into training/val/test set. However, I also want to get the same exact prediction on camera video or on ligne but I have poor prediction ability, all objects are unclassifiable. When I test my transform for each frame of the video or small image of each frame, I get a black image or an image that I don’t want.</p>
</li>
<li>
<p>If I don’t use transforms.normalize to augment the data, I’ve got instability during training, which is very difficult to fix. If I test this trained model on camera, I get the same result, poor predictability.</p>
</li>
<li>
<p>I don’t know how to deal with this problem, but if the image dataset and the real camera image are very different, does my model have a bad prediction, it’s normal?  or How to get mean and std on my dataset is good for camera image?</p>
</li>
<li>
<p>My transforms for data augmentation:</p>
</li>
</ul>
<pre><code class="lang-auto">train_transform = transforms.Compose([
                                    transforms.Resize((img_size, img_size)),
                                    transforms.Pad(padding=30),
                                    transforms.GaussianBlur(kernel_size=(5,9),sigma=(0.1,0.5)),
                                    transforms.RandomHorizontalFlip(p=0.5),
                                    transforms.RandomVerticalFlip(p=0.5),
                                    transforms.RandomPerspective(p=0.5),
                                    transforms.RandomAffine(degrees=(30,70)),
                                    transforms.ToTensor(),
                                    transforms.Normalize(
                                        mean=[0.58067408, 0.5785063,  0.5922596 ],
                                        std=[0.33043968, 0.33169742, 0.33235405])])
</code></pre>
<ul>
<li>My transform for test model:</li>
</ul>
<pre><code class="lang-auto">test_transform = transforms.Compose([
                                    transforms.Resize((img_size, img_size)),
                                    transforms.ToTensor(),
                                    transforms.Normalize(
                                        mean=[0.58067408, 0.5785063,  0.5922596 ],
                                        std=[0.33043968, 0.33169742, 0.33235405])])
</code></pre>
Using max_pooling indices for tensor subsampling/pooling in PyTorch

blade: <p>Given two tensors of the same size, how can we use the indices obtained from max_pooling one tensor to subsample or pool the other tensor in PyTorch? When attempting this approach,</p>
<pre><code class="lang-auto">import torch

# Assuming y1 and e1 have the same dimensions
y1 = torch.randn(1, 32, 28, 28)
e1 = torch.randn(1, 32, 28, 28)

# Max-pooling
mxp = torch.nn.MaxPool2d(2, stride=2, return_indices=True)
y1_pooled, idx = mxp(y1)

# Subsample e1 using idx
e1 = e1[idx]
</code></pre>
<p>I get the error:</p>
<blockquote>
<pre><code class="lang-auto">Traceback (most recent call last):
    e1 = e1[idx]
IndexError: index 1 is out of bounds for dimension 0 with size 1
</code></pre>
</blockquote>

eqy: <p>Here’s a simple snippet illustrating how you could recover the same values using the original input and indices, which should also be applicable to sampling <code>e1</code> in your case:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; y1_pooled2 = torch.gather(torch.flatten(y1, -2), 2, torch.flatten(idx, -2)).reshape(1, 32, 14, 14)
&gt;&gt;&gt; torch.allclose(y1_pooled, y1_pooled2)
True
&gt;&gt;&gt;
</code></pre>
<p>You can think of the index as indexing the flattened spatial dimensions of the input for the max value in each window.</p>
C++ extension backward function

Iaroslav_Elistratov: <p>I’m trying to create a cpp extension following <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">this tutorial</a>.</p>
<p>It seems that if we’re using torch::Tensor as input to a C++ function, and we’re only doing PyTorch operations on it, no custom backward function is necessary.</p>
<aside class="quote" data-post="2" data-topic="33057">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/c-backward-function/33057/2">C++ backward function</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    if you are using torch::Tensor as input to your C++ function, and you are only doing PyTorch operations on it, no backward function is necessary (just like in Python you dont write one). 
However, if you are doing some custom calls, such as going into OpenCV and editing the contents, then custom backward call is necessary, as we dont know how to differentiate through those calls.
  </blockquote>
</aside>

<p>Yet the tutorial <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#backward-pass" rel="noopener nofollow ugc">implements a backward pass</a> even though, I believe, <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#forward-pass" rel="noopener nofollow ugc">in the forward pass</a>  it uses torch::Tensor as input and uses only native PyTorch operations on it. So it seems we don’t need to implement the backward pass in this case but instead in <code>lltm_backward</code> we can just call .backward() on the inputs – is it correct?</p>

Munich: <p><a class="mention" href="/u/iaroslav_elistratov">@Iaroslav_Elistratov</a> were you able to find the answer to this question? I have the exact same question after finding <a class="mention" href="/u/smth">@smth</a> response in the thread you posted above.</p>