akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Frequent discontinuities in loss function during training

ma-sadeghi: <p>I know it’s very difficult to debug without context, but I’m asking just in case this is a known phenomenon. I’m training a U-Net architecture for regression. During training, I frequently see discontinuities in the loss funciton, which look very strange to me. Here’s how the training history looks like:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/870058e3521ef1fe7184a457bb909db1fea52937.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/870058e3521ef1fe7184a457bb909db1fea52937" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/870058e3521ef1fe7184a457bb909db1fea52937_2_672x500.jpeg" alt="image" data-base62-sha1="jghcy768OVrG7xStzqcsB6nBQrl" width="672" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/870058e3521ef1fe7184a457bb909db1fea52937_2_672x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/870058e3521ef1fe7184a457bb909db1fea52937_2_1008x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/7/870058e3521ef1fe7184a457bb909db1fea52937.jpeg 2x" data-dominant-color="E0E4E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1247×927 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m using Adam optimizer with a learning rate of 1e-3.</p>

ptrblck: <p>Are you using any learning rate schedulers or are resetting any objects (e.g. the <code>optimizer</code>)?</p>

ma-sadeghi: <p>No, I use the default values:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(), lr=lr)
</code></pre>

ma-sadeghi: <p>My bad, I think the graph shows a different paramter that I wanted to observe during training, not the actual loss function.</p>

ma-sadeghi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Just wanted to post an update in case you were interested. Indeed, I was monitoring a different parameter, but that was not the issue.</p>
<p>The issue was the <code>eps</code> parameter of Adam optimizer, whose default value is <code>1e-8</code>. I changed it to a much larger value (ex. <code>1e-4</code>) and I no longer see any fluctuations.</p>

ptrblck: <p>That’s interesting to hear. Were you seeing the same behavior in the loss itself, too?</p>

ma-sadeghi: <p>Yes, the actual loss behaves very similar to the one shown in the screenshot.</p>
How to use multi-models on mobile (android) simultaneously?

pai: <p>Hello,</p>
<p>I want to deploy two models to an Android app. But when I modified the demo to load two different models, i noticed that the second loaded model doesn’t work as except. The result is different than only the second model is loaded.</p>
<p>My question here is, if it is supported by PyTorch to use two different models simultaneously?</p>
<p>Thank you in advice</p>
<p>Pai</p>
Multi-class focal loss for label with probabilities (mixup)?

mzimmerman: <p>I’ve been trying to use mixup with focal loss for my multi-class label training. I patch up some codes taken from various open-sources project for multi-class focal loss. However, I’m not able to modify it for label with probabilities (eg. mixup augmentation) rather than binary. Here is my code for one-hot type label:</p>
<pre><code class="lang-auto">def focal_loss(p, t, alpha=None, gamma=2.0):
    loss_val = F.cross_entropy(p, t, weight=alpha, reduction='none')
    p = F.softmax(p, dim=1)

    n_classes = p.size(1)
    # This is where I couldn't modify it for probabilities type label
    t = t.unsqueeze(1)
    shape = t.shape
    target_onehot = torch.zeros(shape[0], n_classes, *shape[2:], dtype=t.dtype, device=t.device)
    target_onehot.scatter_(1, t, 1.0)

    focal_weight = 1 - torch.where(torch.eq(t, 1.), p, 1 - p)
    focal_weight.pow_(gamma)
    focal_weight.to(p.device)

    if len(loss_val.shape) &lt; len(focal_weight.shape):
        loss_val = loss_val.unsqueeze(1)

    # compute loss
    alpha_weight  = 1.0
    focal_loss = focal_weight * alpha_weight * loss_val
    return torch.mean(focal_loss)
</code></pre>
<p>To use it,</p>
<pre><code class="lang-auto">num_classes = 15
pred = torch.rand((16, 50, num_classes), dtype=torch.float)
label = (torch.rand((16, 50, num_classes)) &gt; 0.5).long()
# Expected label type for mixup is:
# label = torch.rand((16, 50, num_classes), dtype=torch.float)

loss = focal_loss(pred.reshape(-1, num_classes), label.view(-1, num_classes), alpha=None)
</code></pre>
<p>Anyone with more knowledge on focal loss could help me with this? Many thanks in advance.</p>
From torch._C import * walkthrough

Larry_Li: <p>What’s happened when I call <code>from torch._C import  *</code> ? In the setup.py file, torch._C as a C extended module being added. In my opinion, when <code>from torch._C import *</code> being called, it should jump to torch/csrc/stub.c file and execute PyInit__C funciton immediately. But the fact is that TorchLibraryInit is reached before PyInit__C. So how does pytorch step into C++ world ?</p>
Replicate operation tensor using `torch.tensordot()` and `torch.stack()`

alvarogutyerrez: <p>I have a tensor operation that I would like to replicate using a combination of <code>torch.stack()</code> and <code>torch.tensordot()</code> to generalize it further on a larger program. In summary, I want to replicate the tensor <code>V_1</code> using said operations into another tensor called <code>V_2</code>.</p>
<pre><code class="lang-auto">N, t , J  = 4, 2 , 3
K_f , K_r = 1, 1 
R = 5
K = K_f + K_r
id = torch.arange(N).repeat(t).sort()
X = torch.randn(N*t, K , J)
Y = torch.randn(N*t, 1)
D = torch.randn(N, K_r , R)
Draw = D.repeat_interleave(t,0) 
beta = torch.randn(2*K_r + K_f, 1)
beta_R = (beta[0:K_r,0] + beta[K_r:2*K_r,0] * Draw ).repeat(1,J,1)
print("shape beta_R:", beta_R.shape)
beta_F = beta[2*K_r:2*K_r + K_f,0].repeat(N*t, J, R)
print("shape beta_F:", beta_F.shape)
XX_0 =X[:,0,:].unsqueeze(2).repeat(1,1,R) 
print("shape XX_0:", XX_0.shape)
XX_1 =X[:,1,:].unsqueeze(2).repeat(1,1,R)
print("shape XX_1:", XX_1.shape)
V_1 = XX_0 * beta_R  + XX_1 * beta_F
print("shape V_1:",V_1.shape)
#shape beta_R: torch.Size([8, 3, 5])
#shape beta_F: torch.Size([8, 3, 5])
#shape XX_0: torch.Size([8, 3, 5])
#shape XX_1: torch.Size([8, 3, 5])
#shape V_1: torch.Size([8, 3, 5])
</code></pre>
<p>Now I want to do the same but stacking my tensors (using <code>torch.stack()</code>) and applying a generalized version of the dot-product (using <code>torch.tensordot()</code>), but I am a bit confused with the <code>dims</code> argument which is not doing what I expected.</p>
<pre><code class="lang-auto">#%% Replicating using stacking and tensordot
stack_XX = torch.stack((XX_0, XX_1), 0)
print("shape stack_XX:",stack_XX.shape)
stack_beta = torch.stack((beta_R, beta_F), 0)
print("shape stack_beta:", stack_beta.shape)
# dot product bewteen stack_XX and stack_beta along the first dimension
V_2 = torch.tensordot(stack_XX, stack_beta, dims=([0], [0]))
print("shape V_2:",V_2.shape)
# check if the two are equal
torch.all(V_1.eq(V_2))

#shape stack_XX: torch.Size([2, 8, 3, 5])
#shape stack_beta: torch.Size([2, 8, 3, 5])
#shape V_2: torch.Size([8, 3, 5, 8, 3, 5])
#tensor(False)

</code></pre>
<p>So I am basically trying to get <code>tensor(True)</code> when running <code>torch.all(V_1.eq(V_2))</code>.</p>
Groupby aggregate product in PyTorch

alvarogutyerrez: <p>I have the same problem as in the question <a href="https://discuss.pytorch.org/t/groupby-aggregate-mean-in-pytorch/45335">Groupby aggregate mean in pytorch</a>. However, I want to create the <strong>product</strong> of my tensors inside each group (or labels). Unfortunately, I couldn’t find a native PyTorch function that could solve my problem, like a hypothetical <code>scatter_prod_</code> for products (equivalent to <code>scatter_add_</code> for sums), which was the function used in <a href="https://discuss.pytorch.org/t/groupby-aggregate-mean-in-pytorch/45335/2">@ptrblck’s answer</a>.</p>
<p>Recycling the example code from <a href="https://stackoverflow.com/a/56156724/10714156" rel="noopener nofollow ugc">@elyase</a>’s question, consider the 2D tensor:</p>
<pre><code class="lang-auto">samples = torch.Tensor([
    [0.1, 0.1],    #-&gt; group / class 1
    [0.2, 0.2],    #-&gt; group / class 2
    [0.4, 0.4],    #-&gt; group / class 2
    [0.0, 0.0]     #-&gt; group / class 0
])
</code></pre>
<p>with labels where it is true that <code>len(samples) == len(labels)</code></p>
<pre><code class="lang-auto">labels = torch.LongTensor([1, 2, 2, 0])
</code></pre>
<p>So my expected output is:</p>
<pre><code class="lang-auto">res == torch.Tensor([
    [0.0, 0.0],
    [0.1, 0.1], 
    [0.8, 0.8] # -&gt; PRODUCT of [0.2, 0.2] and [0.4, 0.4]
])
</code></pre>
<p>Here the question is, again, following <a href="https://stackoverflow.com/a/56156724/10714156" rel="noopener nofollow ugc">@elyase</a>’s question, how can this be done in <strong>pure PyTorch</strong> (i.e. no numpy so that I can autograd) and ideally without for loops?</p>
<p>Crossposted in: <a href="https://stackoverflow.com/questions/74657919/groupby-aggregate-product-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">python - groupby aggregate product in PyTorch - Stack Overflow</a></p>

ptrblck: <p><code>scatter_</code> introduced the <code>reduce</code> argument and <code>reduce='multiply'</code> should work.</p>

alvarogutyerrez: <p>You are right. Indeed, this was posted on StackOverflow. Reposting for future searches.</p>
<pre><code class="lang-auto">samples = torch.Tensor([
    [0.1, 0.1],    #-&gt; group / class 1
    [0.2, 0.2],    #-&gt; group / class 2
    [0.4, 0.4],    #-&gt; group / class 2
    [0.0, 0.0]     #-&gt; group / class 0
])

labels = torch.LongTensor([1,2,2,0])

label_size = 3
sample_dim = samples.size(1)

index = labels.unsqueeze(1).repeat((1, sample_dim))

res = torch.ones(label_size, sample_dim, dtype=samples.dtype)
res.scatter_(0, index, samples, reduce='multiply')
</code></pre>
Very strange issue with tensor asyncio and rpc

spyroot: <p>Hi Folks,</p>
<p>I have a bizarre issue and can’t narrow down a problem.</p>
<p>Agent &lt;— &gt; Observer.<br>
(Observer takes the tensors and sends via async RPC)</p>
<pre><code class="lang-auto">Observer:
   # agent will issue this call, and foobar_rpc put tensor from Cuda to CPU and send.
   foobar_rpc():
      tensor.clone().cpu()
</code></pre>
<p>The agent takes the future and puts asyncio queue.</p>
<p>Now in asyncio dequeue, does</p>
<pre><code class="lang-auto">Agent
   async def the_rpc_call() 
         fu = rpc_async().foobar() # args etc. removed to make it clear
         await THE_QUEUE.put(the_fu)
</code></pre>
<pre><code class="lang-auto">async def foo()
# skip code for a loop
        the_future = THE_QUEUE.get() &lt;- asyncio queue
	the_future.wait()       # wait for torch future
	Val = future.value()   # read value
	x, y = Val ( let assume and x and y two named tuple)
	i.e., 

        x.rewards &lt;-- is tensor
        x.observation.  &lt;-- is tensor
        x.someint &lt;-- native python type

	# If I do all good
	xsum=torch.sum(x.rewards, dim
        print(xsum) &lt;-- no issue
        some_dict , list etc = xsum &lt;--- error in RPC , or blocking&gt;
        left_hand = xsum.clone()  &lt;-- error
        left_hand = copy_ same error etc.

        # if I try to do anything else, it either 
        # block or crash, or I see an error in RPC 

       await some_foo(x, y)
       no errors. at all
</code></pre>
<p>It looks like somewhere in downstream code, it either has a lock, but on read-only, it works; on left-hand assignment, it fails for tensors.  But if that is the case, the last call some_foo (x, y) should produce the same behavior. i.e., if I pass x and y to downstream code, I have an issue now.</p>
<p>Does RPC logic do anything specific deep in a library with the tensors it receives?</p>
<p>Thank you,</p>
Torchscript with vulkan backend fails to interpret on Android Phone

vjayd: <p>I have generated a .pt file using the PyTorch Android documentation as given here <a href="https://pytorch.org/tutorials/prototype/vulkan_workflow.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Vulkan Backend User Workflow — PyTorch Tutorials 1.13.0+cu117 documentation</a></p>
<p>I am able to run the example mobilenetv2.pt  as given in the tutorial properly. However, when I try to run the model created by me it fails with the following error:</p>
<pre><code class="lang-auto">
E/libc++abi: terminating with uncaught exception of type std::runtime_error: The following operation failed in the TorchScript interpreter.
    Traceback of TorchScript, serialized code (most recent call last):
      File "code/__torch__/___torch_mangle_1036.py", line 232, in forward
          running_mean0 = _110.running_mean
          running_var0 = _110.running_var
          _59 = torch.instance_norm(input14, None, None, running_mean0, running_var0, False, 0.10000000000000001, 1.0000000000000001e-05, True)
                ~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
          input15 = _59
        map3 = torch.relu_(input15)
    
    Traceback of TorchScript, original code (most recent call last):
      File "/home/miniconda3/envs/pytorch-dev/lib/python3.8/site-packages/torch/nn/functional.py", line 2495, in forward
        if use_input_stats:
            _verify_spatial_size(input.size())
        return torch.instance_norm(
               ~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
            input, weight, bias, running_mean, running_var, use_input_stats, momentum, eps, torch.backends.cudnn.en
</code></pre>
<p>Does it mean that</p>
<ol>
<li>instance_norm is not yet supported for the vulkan backend or</li>
<li>The torchscript file is not properly generated or</li>
<li>It is an Android Issue?</li>
</ol>
<p>If instance_norm is not supported then why don’t we have a CPU FALLBACK mechanism like we have for Android NNAPI?</p>

ptrblck: <p>Please don’t tag specific users, as it could discourage others to post a valid answer, and you might tag someone who is not familiar with this particular question (in this case I don’t know enough about Android to help you out).</p>

vjayd: <p>Thanks for the heads up</p>

vjayd: <p>It’s fixed!, This issue is not present in the latest PyTorch master</p>
Adding attribute to a Pytorch Tensor

Prakhar_Pradhan1: <p>Hi  Everyone!<br>
I noticed at some places that we can add an attribute to a Pytorch Tensor, please see the below example:<br>
a = torch.rand(1,1,3,3)<br>
a.scale=2<br>
a.zeropoint = 128</p>
<p>Curious to understand this functionality and its implications on gradient calculation. Is any official doc available to understand this?<br>
Thanks in advance!</p>

ptrblck: <p>I guess internally the <code>tensor.__setattr__</code> operation is called to create a new attribute and assign the value to it. I don’t know if it would have any implications besides carrying the data.</p>
Dummy CPU device for debugging purposes?

colinrsmall: <p>I’m currently debugging an issue in a model I’m working on where some tensors aren’t properly being sent to the GPU. I run the model remotely on a machine with a GPU, but develop and debug the model locally on my laptop without a GPU. I can’t recreate the issue locally because everything it sent to the CPU since no GPU is available.</p>
<p>Is there a way to create an alias of the CPU as a dummy device that I can send my tensors to in order to debug my issue?</p>

ptrblck: <p>Maybe <a href="https://pytorch.org/torchdistx/latest/fake_tensor.html">Fake Tensors</a> would work for your use case?</p>
What is the best version of torchvision when using torch-2.0?

Yifan_Jiang: <p>Hi,</p>
<p>I’m trying to install torch-2.0. While I’ve successfully install it, I run it and met below issue</p>
<pre><code class="lang-auto">anaconda3/envs/torch2/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
  warn(f"Failed to load image Python extension: {e}")
</code></pre>
<p>I search through the internet and seems this issue is caused by mismatch version between torch and torchvision. Which version of tochvision should I install?</p>

ptrblck: <p>I would recommend to install the nightly releases of both libraries.<br>
Could you check if you’ve already did it or if you’ve used an older <code>torchvision</code> release?</p>
How to update just PyTorch

NewtoAI: <p>I am trying to test the AI solution SimSwap however when I used their guideline using<br>
" ```<br>
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=10.2 -c pytorch</p>
<pre><code class="lang-auto">I am getting "NVIDIA GeForce RTX 4080 with CUDA capability sm_89 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the NVIDIA GeForce RTX 4080 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/"
  when trying t fix it I updated the [PyTorch ](https://discuss.pytorch.org/t/geforce-rtx-3090-with-cuda-capability-sm-86-is-not-compatible-with-the-current-pytorch-installation/123499)
trying to fix it using "conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia" seem to miss up other related package any suggestion how to update just PyTorch</code></pre>

ptrblck: <p>You would have to install the latest PyTorch release with a CUDA 11.x runtime to be able to run code on your sm_89 GPU.</p>
Pre-quantized model with qint8 weights seems to be decimal

chenxj: <pre><code class="lang-auto">&gt;&gt;&gt; a["features.16.conv.1.0.weight"]
tensor([[[[ 1.6094,  2.3566,  1.6094],
          [ 1.0921,  0.2874,  1.0921],
          [ 0.5173, -1.2071,  0.1724]]],


        [[[ 0.4598,  0.2299,  0.4598],
          [ 0.3449,  2.2417,  0.5748],
          [ 0.4598,  0.7472,  0.4024]]],


        [[[ 1.3795, -0.4598, -0.6897],
          [ 2.1842, -0.8622, -1.1496],
          [ 1.0346, -0.8047, -0.7472]]],


        ...,


        [[[-0.8622, -1.6669, -0.9771],
          [ 0.5173, -0.6897,  0.7472],
          [ 0.8622,  2.1842,  0.9771]]],


        [[[ 0.4024, -0.3449,  0.4598],
          [-0.0575, -1.1496,  0.0000],
          [ 1.2071, -0.5748,  1.2071]]],


        [[[ 0.1724,  0.4598,  0.2299],
          [ 0.2299,  1.5519,  0.3449],
          [ 0.4598,  0.4598,  0.4598]]]], size=(960, 1, 3, 3),
       dtype=torch.qint8, quantization_scheme=torch.per_tensor_affine,
       scale=0.05747907608747482, zero_point=0)
</code></pre>
<p>but a[“features.16.conv.1.0.weight”][0, 0, 0, 1].item() is 2.<br>
Anyone can explain this observation? Thanks.</p>

Vasiliy_Kuznetsov: <p>Quantized tensors represent values in the floating point domain.  When you print a quantized tensor, the printed values by PyTorch are the fp32 values that your quantized tensor represents.</p>
<p>When you print <code>a[“features.16.conv.1.0.weight”][0, 0, 0, 1].item()</code>, you are looking at the raw value which needs to then be transformed with <code>fp = float(q - zp) * scale</code> to convert to the floating point domain.</p>
Quantization Aware Training (QAT) Choosing Hyperparameters

Vij_Flex: <p>Hi there!</p>
<p>I am trying to use Quantization Aware Training to get better performance on my quantized EfficientNet model, since PTQ performed very poorly. I tried using the original hyperparameters from the normal model to train it, and it did not perform well. After decreasing the learning rate, I was able to get much higher performance, but still not very close to the original model.</p>
<p>Can anyone provide me with some guidance for hyperparameter tuning with QAT or any other tips and tricks that could be useful? I was not able to find any good resources with this kind of information, so if you could point me in the direction of those, that would also be helpful.</p>
<p>Thank you.</p>

andrewor: <p>Hi <a class="mention" href="/u/vij_flex">@Vij_Flex</a>, can you share what QConfigs you are using to quantize the model? Hyperparameter tuning is mostly orthogonal to quantization, so unfortunately there aren’t really any recommended ways to do that from the PyTorch architecture optimization side.</p>

Vij_Flex: <p>Hi <a class="mention" href="/u/andrewor">@andrewor</a>, I am using fbgemm for the QConfigs.</p>

Vasiliy_Kuznetsov: <p>You can check <a href="https://pytorch.org/vision/stable/models.html" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision 0.14 documentation</a> for inspiration, there are a couple of recipes linked there for QAT (unfortunately not for EfficientNet but the hyperparameters from other vision models should be useful).</p>
Does pytorch use tensor cores by default?

indramal: <p>does PyTorch use tensor cores by default when using the train?</p>
<p>when the model inserts to Cuda cores using following code.</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 
model = model.to(device)
</code></pre>

eqy: <p>It would depend on the GPU, operations and data types being used. For Volta: fp16 should use tensor cores by default for common ops like matmul and conv. For Ampere and newer, fp16, bf16 should use tensor cores for common ops and fp32 for convs (via TF32). You can also enable tensor cores for fp32 matmuls on Ampere and newer via: <a href="https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision" rel="noopener nofollow ugc">torch.set_float32_matmul_precision — PyTorch 1.13 documentation</a> but it is not a default.</p>

indramal: <p>You are correct but I need to confirm it. If not use tensor cores in common ops, is there any method to do it. That is what I need to know. <code>.is_cuda</code> method tells whether it is CUDA or nor. I think pytorch indicates it both CUDA cores and Tensor cores. So there is no option to check about tensor cores.</p>
<p>When writing code for automat mixed precision, also we don’t know it use CUDA cores or Tensor cores.</p>

suraj.pt: <p>No it doesn’t use it by default in v1.12 or later. You have to set</p>
<pre><code class="lang-auto"># The flag below controls whether to allow TF32 on matmul. This flag defaults to False
# in PyTorch 1.12 and later.
torch.backends.cuda.matmul.allow_tf32 = True

# The flag below controls whether to allow TF32 on cuDNN. This flag defaults to True.
torch.backends.cudnn.allow_tf32 = True
</code></pre>
<p>See: <a href="https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices" class="inline-onebox">CUDA semantics — PyTorch 1.13 documentation</a></p>

indramal: <p>I enable both in A100 GPU but speed up <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

eqy: <p>Could you post a minimal version of the code that you are trying to speed up?</p>
Different sequences lenght in time series forecasting

vinicius_chacon: <p>Hello everyone!</p>
<p>I’m trying to run a model which predicts the river flow for next month (t+1) using a precipitation forecast model (also t+1) and observed data (t, t-1, …, t-n). So my dataset will be something like:</p>
<p>features: ‘precipitation forecast  data (t+1), observed precipitation (t, … t-n), observed inflow (t, … t-n)’</p>
<p>output : ‘inflow forecast (t+1)’</p>
<p>Does anyone have a suggestion on how I shall build my DataLoader for the NN input data? I’m currently building a manual data set with one column for each data and time step. Kinda looking like this:</p>
<p>P (t+1) | P (t) | P (t-1) | P (t-n) | Inf (t) | Inf (t-1) | Inf (t-n)| Inf (t+1)</p>
<p>Thanks!</p>
Memory leak if use torch.no_grad()?

chienpingtsung: <p>Here’s my code:</p>
<pre><code class="lang-auto">for epoch in range(10):
    net.train()
    # Good training.
    for data in trainloader:
        inputs, labels = data['images'], data['masks']

        for idx in range(0, len(inputs), 7):
            optimizer.zero_grad()

            outputs = net(inputs[idx:idx + 7])
            loss = criterion(outputs, labels[idx:idx + 7])
            loss.backward()
            optimizer.step()

    # Bad validation.
    net.eval()
    test_loss = 0.0
    test_times = 0
    for data in testloader:
        with torch.no_grad():
            inputs, labels = data['images'], data['masks']

            for idx in range(0, len(inputs), 7):
                outputs = net(inputs[idx:idx + 7])
                loss = criterion(outputs, labels[idx:idx + 7])
                test_loss += loss.item()
                test_times += 1
    test_loss /= test_times
</code></pre>
<p>If I use <code>torch.no_grad()</code> block, the cpu memory will continually increase untill OOM kill happens.<br>
But once I remove the <code>no_grad</code>, everything would be all right.<br>
I tried <code>del loss</code> or put the validation into a function, but the memory leak still happens.<br>
Is my code wrong?</p>

albanD: <p>Hi,</p>
<p>The code looks ok. And I’m really not sure how no_grad can cause this.</p>
<p>Do you have a small code sample (30/40 lines) that reproduces this?</p>

chienpingtsung: <p>Hello,<br>
I will make a sample code of it later.<br>
Thanks for your reply~</p>

chienpingtsung: <p>Hi,<br>
Thanks for your following up. It’s midnight here, and the GPUs are occupied. So I’m going to test it in a few days.</p>
<p>One interesting thing I found is that <strong>validation on CPU works fine</strong> via <code>no_grad</code> with same code.</p>
<p>My complex tensor slice might be potential reason, but I’m confused why the CPU RAM leak, not the GPU’s.</p>

David_Ireland: <p>I am having a similar issue to that which OP posted. I have commented out all my training methods, the only method that remains is similar to an RL setting where I use a network to get an action for an environment. However, despite wrapping the main call in an <code>torch.no_grad()</code> I am still getting memory leakage (happens on both CPU and GPU). The code is quite long, and I’m not sure I could make it more concise, but do you know what a common cause to look for would be? I’m losing my mind, as usually these memory issues are caused by computational graphs not being freed but everything is now without a grad so I’ve no idea what can cause it.</p>
AttributeError in DataLoader using Opacus

mmisiur: <p>Hello,<br>
I was excited to learn about Opacus so now I’m trying to apply it to my training. I’m using DataLoader with my custom dataset MyDataset(Dataset). During calling dataloader to get data I’m getting an error:<br>
File “***\lib\multiprocessing\reduction.py”, line 60, in dump<br>
ForkingPickler(file, protocol).dump(obj)<br>
AttributeError: Can’t pickle local object ‘wrap_collate_with_empty..collate’</p>
<p>Do anyone of you can help me with fixing it? I can provide more insight, but don’t know where to start.<br>
Win10<br>
Pytorch 1.13<br>
Opacus 1.1.3<br>
(training worked before applying opacus)</p>

alexandresablayrolle: <p>Thanks for raising this issue. Would you be able to share a minimal reproducing example? I also see that you use Opacus 1.1.3, can you update to the latest version (1.3)?</p>

mmisiur: <p>Hi Alexandre,<br>
thank you for your reply. I’ve updated opacus to 1.3, however somehow conda does not provide lastest opacus release - I had to do that via pip.</p>
<p>Nonetheless, the problem persisted. I thought that maybe it’s the multiprocessing/trainloader problem so I’ve set the number of workers to 0, and it helped to workaround the problem.</p>
<p>I will still try to post a minimal working example this week, so maybe we will figure out a solution rather than workaround.</p>
<p>(So now it is:<br>
Win10<br>
Pytorch 1.13<br>
Opacus 1.3)</p>

alexandresablayrolle: <p>Thanks! We are very interested in knowing if it is a general problem with multiprocessing.</p>
Bias-only layer

carbocation: <p>This question was asked <a href="https://discuss.pytorch.org/t/learnable-bias-layer/4221">about 5 years ago</a> but I feel like it might be appropriate to revisit it:</p>
<p>Can we build a learnable bias layer that will work for arbitrary-sized input? (E.g., a single parameter that is added to every element of a scalar or an image or a video, depending on input tensor shape.)</p>
<p>Is the answer just to build a trainable parameter and broadcast it?</p>

ptrblck: <p>Yes, your idea to create a custom layer with the trainable <code>bias</code> (containing a single value) and broadcast it to the input shape sounds valid.</p>

carbocation: <p>Thanks. It does seem to work as expected:</p>
<pre><code class="lang-python">class BiasLayer(torch.nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        bias_value = torch.randn((1))
        self.bias_layer = torch.nn.Parameter(bias_value)
    
    def forward(self, x):
        return x + self.bias_layer
</code></pre>
Combine predictions of torchvision and torchtext model

melampus123: <p>I am working on a multi-class image classification problem. The images being classified each have an associated text description. I began by first creating an image-only model (ResNet) that classifies the images based solely on the image (no text at all). I also trained a torchtext model (TextCNN) that classifies the images based solely on their text annotations.</p>
<p>Is there some way I can combine the outputs of the torchvison and torchtext model? Right now I’m just looking at the top 3 predictions from each model and selecting the class with the highest probability but I think there is probably a better way to do this.</p>
FSDP tutorial: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm

Brett_Clark: <p>Hi there,</p>
<p>I’m getting the following error when running the code (copied exactly) from the <a href="https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html" rel="noopener nofollow ugc">FSDP tutorial</a>:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
</code></pre>
<p>I read in other threads that this can be due to input size not matching layer size, but I have checked this. I have also tried reducing batch to 1. This error still persists, any ideas?</p>
<pre><code class="lang-auto">&gt; python -m torch.utils.collect_env          
Collecting environment information...
PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux Server release 7.9 (Maipo) (x86_64)
GCC version: (GCC) 10.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.8.6 (default, Mar 29 2021, 14:28:48)  [GCC 10.2.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.66.1.el7.x86_64-x86_64-with-glibc2.2.5
Is CUDA available: True
CUDA runtime version: 11.6.55
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe

Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.3
[pip3] pytorch-lightning==1.8.3
[pip3] torch==1.13.0
[pip3] torchio==0.18.84
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.14.0
[conda] Could not collect
</code></pre>

Brett_Clark: <p>Issue was mismatched version of CUDA (v11.6) to CUDA version used to build pytorch (v11.7). Upgrading to CUDA v11.7 fixed the problem.</p>
Sign Up Log In Is there a way to make the following code(signal filter) run faster?

csyyj: <p>I want to implement a filter like a(t) = 0.995*a(t-1) + (1-0.995)*in(t). I use TorchScript to speed up, but it is still very slow. Is there a better way to implement it?</p>
<pre><code class="lang-auto">class PostFilter(jit.ScriptModule):
    def __init__(self):
        super(PostFilter, self).__init__()

    @jit.script_method
    def forward(self, d_filter):
        b, c, t, f = d_filter.size()
        w = torch.zeros([b, c, 1, f], dtype=d_filter.dtype, device=d_filter.device)
        l = torch.jit.annotate(List[Tensor], [])
        for i in range(t):
            w = 0.995 * w + (1 - 0.995) * d_filter[:, :, i:i + 1]
            l += [w]
        d_filter_new = torch.cat(l, dim=2)
        return d_filter_new
</code></pre>

ptrblck: <p>In the current release (<code>1.12.1</code>) I’m seeing a 3x speedup when this method is scripted:</p>
<pre><code class="lang-python">class PostFilterEager(torch.nn.Module):
    def __init__(self):
        super(PostFilterEager, self).__init__()

    def forward(self, d_filter):
        b, c, t, f = d_filter.size()
        w = torch.zeros([b, c, 1, f], dtype=d_filter.dtype, device=d_filter.device)
        l = []
        for i in range(t):
            w = 0.995 * w + (1 - 0.995) * d_filter[:, :, i:i + 1]
            l += [w]
        d_filter_new = torch.cat(l, dim=2)
        return d_filter_new


class PostFilter(torch.jit.ScriptModule):
    def __init__(self):
        super(PostFilter, self).__init__()

    @torch.jit.script_method
    def forward(self, d_filter):
        b, c, t, f = d_filter.size()
        w = torch.zeros([b, c, 1, f], dtype=d_filter.dtype, device=d_filter.device)
        l = torch.jit.annotate(List[Tensor], [])
        l = []
        for i in range(t):
            w = 0.995 * w + (1 - 0.995) * d_filter[:, :, i:i + 1]
            l += [w]
        d_filter_new = torch.cat(l, dim=2)
        return d_filter_new

# eager
# setup
model = PostFilter()
x = torch.randn(64, 64, 64, 64, device='cuda')

# warmup
for _ in range(10):
    out = model(x)

# profile
torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(1000):
    out = model(x)
torch.cuda.synchronize()
t1 = time.perf_counter()
t_eager = (t1 - t0)/1000
print(t_eager)

# scripted
# setup
model = PostFilterEager()
x = torch.randn(64, 64, 64, 64, device='cuda')

# warmup
for _ in range(10):
    out = model(x)

# profile
torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(1000):
    out = model(x)
torch.cuda.synchronize()
t1 = time.perf_counter()
t_scripted = (t1 - t0)/1000
print(t_scripted)

# speedup
speedup = t_scripted/t_eager
print(speedup)
</code></pre>
<p>What is your expectation regarding the performance of this operation?</p>

csyyj: <p>thank you for your reply. This is the part of my network that is the bottleneck of speed during training. I’m wondering if there is a better implementation to speed up this part.</p>
Try to use pytorch through IDLE Shell

oo_aa: <p>I am completely new with PyTorch and run into an installation issues and need some help. Thank you for reading the following:<br>
Windows 10 Enterprise 64bit<br>
Latest Anacoda installed<br>
Python version is 3.9.13</p>
<p>I have installed Pytorch in c:\users\xxx\anaconda3\lib\site-packages<br>
When I use Qt Console, i can import pytorch and test basic function such as torch.randn<br>
that was good news.</p>
<p>However, when i test import torch in IDLE Shell (that is the only IDE i have now), i run into a problem.<br>
Originally the python cwd was:<br>
c:\Users\xxx\Anaconda3\Scripts<br>
I run into the following error:</p>
<p>============<br>
Traceback (most recent call last):<br>
File “&lt;pyshell#12&gt;”, line 1, in <br>
import torch<br>
File “C:\Users\XXX\Anaconda3\lib\site-packages\torch_<em>init</em>_.py”, line 223, in <br>
for name in dir(_C):<br>
NameError: name ‘_C’ is not defined</p>
<p>======<br>
Then i tried to change the cwd to other empty locations, including the one used by the Qt Console. I continue to have the same error.<br>
Anyone can suggest a solution? Thank you very much.</p>
Multilabel classification object detection | FasterRCNN | Annotation .csv file

smile_0: <p>Hello everyone, greetings!</p>
<p>I created an annotation file in .csv format for a multilabel classification object detection model. I am using the FasterRCNN algorithm for the training. I tried to annotate the image and labels in the .csv file in the format where columns are [image_id, class_name, class_id, x_min, y_min, x_max, y_max]. Since some of my image datasets have two objects in the image, the approach I tried is that I assigned two different rows in the .csv file and annotate the class and bbox info for each class for the same image. Here I think my model is considering the same image as two different independent images for their respective classes. (am I correct?)</p>
<p>After having the closer look at the validation output, I observed that the image with two objects inside only predicts one object in the image (and that is just Obj_B) whereas the ground truth says two objects (Obj_A &amp; Obj_B) inside the image. I am unable to investigate, why the model predicts only one object in the image (i.e.just Obj_B)? have I annotated the information in .csv in an incorrect way? Could anyone provide/show me a typical/sample .csv file that has been used for multilabel classification obj. detection case? I would be also thankful for your overall opinion or suggestion in this regard.</p>
<p>(wish you all a good day and sry for my bad english)</p>
Is it a good idea to utilize openmp when doing CPU PyTorch distributed training?

gc-fu: <p>The case is that we are doing distributed PyTorch training using the DDP model with CPU clusters.</p>
<p>I wonder if it is a good idea to set the <code>OMP_NUM_THREADS</code> as suggested by the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#utilize-openmp" rel="noopener nofollow ugc">tuning-guild</a>.</p>
<p>My worry is that the training is already a CPU-intensive workload and the use of <code>OpenMp</code> may contend the CPU, which may lead to even-worse performance.</p>
<p>Another question is that if the <code>OMP_NUM_THREADS</code> should be set, then is there a suggested value for it related to the number of logical cores used?</p>

fduwjj: <p>What backend are you guys using? If you are using Gloo this variable is not helping probably. If you are using MPI yes this can have some effects here.</p>

gc-fu: <p>Sorry for the late response.<br>
We are using GLOO backend.<br>
Currently, we set this value to the number of physical cores used for training.<br>
Why this variable is not helping for GLOO?</p>
Why there is no label setting in the Transfer learning tutorial?

you_lin: <pre><code class="lang-auto">data_dir = 'data/hymenoptera_data'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
</code></pre>
<p>In this code, why the author didn’t set a label to the image, but it can still work and extract the corresponding label?<br>
Thx.</p>

ptrblck: <p>I don’t fully understand the question since <code>ImageFolder</code> is creating the labels based on the image folder structure. The <code>DataLoader</code>s will thus return the image tensors and their corresponding targets.</p>
RuntimeError: .. torchaudio sox_effects .. requires sox

Emrah: <p>Hi,<br>
I’m trying to filtering and sample rate changing for Data Augmentation. Similar to the example in the link below :<br>
“<a href="https://pytorch.org/tutorials/beginner/audio_data_augmentation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Audio Data Augmentation — PyTorch Tutorials 1.12.1+cu102 documentation</a>”</p>
<p>But I am getting the following error message:<br>
File “C:.…\lib\site-packages\torchaudio_internal\module_utils.py”, line 143, in wrapped<br>
raise RuntimeError(f"{func.<strong>module</strong>}.{func.<strong>name</strong>} requires sox")<br>
RuntimeError: torchaudio.sox_effects.sox_effects.apply_effects_tensor requires sox</p>
<p>-My env: Python 3.7.8,  PyCharm, win10.<br>
-I have installed win sox. And i have added sox folder to my PATH from  Environment Variables<br>
I am sharing a sample code snippet below.</p>
<pre><code class="lang-auto">def codec_apply(waveform):
    # Apply filtering and change sample rate
    filtered, sample_rate2 = torchaudio.sox_effects.apply_effects_tensor(
        waveform,
        sample_rate,
        effects=[
            ["lowpass", "4000"],
            [
                "compand",
                "0.02,0.05",
                "-60,-60,-30,-10,-20,-8,-5,-8,-2,-8",
                "-8",
                "-7",
                "0.05",
            ],
            ["rate", "8000"],
        ],
    )
    codec_applied = F.apply_codec(filtered, sample_rate2, format="gsm")
    return codec_applied
</code></pre>
<p>Waiting for your kind help, thank you.</p>

Emrah: <p>I tried run it from the command line or another IDE, but I still got the error.<br>
I guess the incompatibility problem is related to Windows OS. I didn’t have a problem in Ubuntu.</p>

JosieZI: <p>Hi~ Do you solve this problem? I have the same trouble wuwuwu</p>

Emrah: <p>Unfortunately <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> , I couldn’t solve the problem on Windows, but it worked on Ubuntu</p>
How to know pytouch run on CUDA core or Tensor Core?

indramal: <p>How to know if a model is run in CUDA cores or Tensor Core?</p>
<p>When we give the following code, model goes to CUDA cores.</p>
<pre><code class="lang-python">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") 
model = model.to(device)
</code></pre>
<p>and then can check by using <code>next(model.parameters()).is_cuda</code>.</p>
<p>How to run a model in tensor cores? and how to check if it runs in tensor cores.</p>

Mahdi_Amrollahi: <p>You can benefit from <code>nvprof</code> by Nvidia to profile it.</p>

indramal: <p>Please provide any examples with nvprof</p>

Mahdi_Amrollahi: <p>Actually I haven’t tried yet. This package is provided by Nvidia and there are some tutorials on their developer website.</p>
<p><a href="https://youtu.be/yyR0ZoCeBO8" rel="noopener nofollow ugc">https://youtu.be/yyR0ZoCeBO8</a></p>

eqy: <p>ICYMI, please follow up in this thread here: <a href="https://discuss.pytorch.org/t/does-pytorch-use-tensor-cores-by-default/167676/2" class="inline-onebox">Does pytorch use tensor cores by default? - #2 by eqy</a></p>
Model parallelism vs. Data parallelism: Training data does not fit on GPU. batch_size=1

redsphinx: <p>During inference/prediction (not training) I can run the model with the data. However, when training, it throws a cuda out of memory error. Batch size = 1. I suspect that it happens due to the building of the compute graph maybe?<br>
I have 4 GPUs available to me and I want to use all 4 instead of only 1. Can I somehow pool the memory of all GPUs together?</p>
<p>As I understand, if I use DataParallel or DistributedDataParallel, this essentially copies the process multiple times over the GPUs. So in my case I will have 4 models running in parallel. But my data does not fit on 1 GPU, and I think DataParallel is not the solution here, right?</p>
<p>I saw that I can also use model parallelism by breaking down the model in different chunks and having each GPU process a separate chunk. If the compute graph was indeed the issue before, then this should solve it?</p>
<p>What is the best way to go about dealing with data that does not fit 1 GPU?</p>

ptrblck: <p>You could check <a href="https://github.com/pytorch/tau"><code>tau</code></a> as an approach of pipeline parallelism or e.g. <code>torch.utils.checkpoint</code> to trade compute for memory.</p>

redsphinx: <p>Oh I haven’t encountered pipeline parallelism before. Is this conceptually a combination of data and model parallelism?</p>

ptrblck: <p>I would see it as a model parallel technique, since the actual model is split onto different devices while the data stays untouched.</p>
Dealing with graphs with heterogeneous sizes

zaratruta: <p>Hi.<br>
I would like to use graph neural networks (from pytorch geometric) for dealing with a dataset of graphs, where each graph has a different number of nodes and edges.</p>
<p>I would like to test graf convolutional networks (GCN) and graph attention networks (GAT). Are these approaches able to deal with such kind of dataset (with a heterogeneous number of nodes and edges)? This is not clear for me.</p>
<p>I plan to use these two layers, specifically:<br>
<a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.GCNConv" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.GCNConv</a><br>
<a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.GATConv" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.GATConv</a></p>
Computation of weight matrix for the residual block of a Residual Neural Network (ResNet)

angelognazzo: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c79a4c64b52c593e73577860b6400f55a4e528c8.png" alt="image" data-base62-sha1="stLBafKHq98J8ThCjEUCJBlBPxS" width="678" height="362"><br>
I am trying to understand if I can “summarize” the operations in the residual block with a single weight matrix and bias.<br>
For example, consider the structure of the photo and suppose all the operations can be written via a linear form (don’t consider the relu). Then I can compute the two matrices that tell me how I can go from the input the output of path a and of path b. Then the real output should, from the construction of residual networks, the sum (pointwise) of what path a and path b produce.<br>
Then my intuition and some examples tell me that the matrix that tells me how the input changes when going through the residual block should be just the sum of the two for path a and path (i.e. every sum is done element wise).<br>
I would then implement this matrix using torch.add(mat_path_a, mat_path_b).<br>
Is this possible?<br>
Thanks in advance</p>
Validation Loss Not Decreasing, Local Minima

Thirumalai_Kumar: <p>I have been training a model with nearly 2L images and the validation loss seems to get stuck at 0.8. I tried many ways:</p>
<ol>
<li>Changed activation function</li>
<li>Used weighted CE instead of normal CE since the dataset is imbalanced.</li>
<li>Tried changing optimizers and lr_schedulers<br>
But nothing worked. I hope so it getting stuck with local minima. Is there anything else that I should try??</li>
</ol>
<p>Thanks in advance,<br>
Thirumalai Kumar</p>

ConvolutionalAtom: <p>Without seeing which strategies you already followed (your data you initial model), there is a variety of things you can follow, using more recently proposed activation functions (e.g. gelu) and optimizer etc.<br>
Maybe you can provide more info about what you already tried.<br>
You can also compare your own result to results obtained by others (e.g. literature) maybe you already reached the minimum feasible loss.</p>
<p>Another thing I would emphasize is that there is a difference between validation loss and validation metric. Usually, validation loss is loss function applied to validation data whereas validation metric would be a metric (appropriate for your problem ) applied to validation data.</p>
<p>For instance, if you were to experiment of cifar32 using NLL as loss function and accuracy as metric. Then, you will see the NLL for validation data will start increasing even though validation accuracy keep increasing (at a slower rate than initially) (There is a diverse set of ideas and papers on why this occurs)<br>
Maybe, you can check validation metric see if it is improving</p>

Thirumalai_Kumar: <p>actually it is the validation loss which is not decreasing after it reaches 0.8</p>