akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
SwitchNet - a fast transform based neural network

Cong_Chua_toc_may: <p>You might like to know about SwitchNet. It uses the fast Walsh Hadamard transform to replace the one-to-all connectivity of a matrix of weights. It then uses parametric (switching based) activation functions as the adjustable system.<br>
Then it uses a final fast transform as a set of set of statistical summary measures acting on the crude parametric function outputs.<br>
<a href="https://ai462qqq.blogspot.com/2023/04/switch-net.html" rel="noopener nofollow ugc">https://ai462qqq.blogspot.com/2023/04/switch-net.html</a><br>
Online code example:<br>
<a href="https://editor.p5js.org/siobhan.491/sketches/RvqZfikaE" rel="noopener nofollow ugc">https://editor.p5js.org/siobhan.491/sketches/RvqZfikaE</a></p>
Multiple optimizer issues

Ethan_Lee: <p>Hello, I’m trying to figure this problem of optimizing different part of the model with different data.</p>
<p>The setting I’m using is composed of data1 and data2 with model partA, partB, and partC.</p>
<p>The original code was like this:</p>
<pre><code class="lang-auto">optimizer = optim.Adam([partA, partB, partC], lr=0.03)

for input, target in data1:
     optimizer.zero_grad()
     # loss1 calculation with partA,B,C
     loss1.backward()
     
     input, target in data2:
     partA.requires_grad = False
     partC.requires_grad = False
     # loss2 calculation with only partB
     loss2.backward()
     partA.requires_grad = True
     partC.requires_grad = True
     
     optimizer.step()
</code></pre>
<p>What I want to change here is to train partC during the training of data2 with learning rate 0.0001</p>
<p>I planned to implement this code below but not sure if this is right.</p>
<pre><code class="lang-auto">optimizer1 = optim.Adam([partA, partB, partC], lr=0.03)
optimizer2 = optim.Adam({[partB], lr=0.03},{[partC], lr=0.0001})

for input, target in data1:
     optimizer1.zero_grad()
     optimizer2.zero_grad()
     # loss1 calculation with partA,B,C
     loss1.backward()
     
     input, target in data2:
     # loss2 calculation with only partB
     loss2.backward()
     
     optimizer1.step()
     optimizer2.step()
</code></pre>
<p>Especially, I’m not sure with the multiple optimizer part.</p>
<p>Thank you all!!</p>

srishti-git1110: <p>Hi,<br>
Multiple optimizers shouldn’t be an issue.<br>
Importantly, note that if you do not clear the gradients explicitly using <code>zero_grad()</code>, gradients from multiple backward calls are accumulated.</p>
<p>Hence, according to your pseudo code, after <code>loss2.backward()</code>, <code>grad</code> of B shall be the sum of gradients of <code>loss1</code> and <code>loss2</code> wrt B.</p>
<p>Also, from what you mention you want to calculate the gradient of <code>loss2</code> as well wrt C. The pseudo code on the other hand mentions that C isn’t involved in the calculation of <code>loss2</code>.</p>
Symbolically trace LSTM layers

Ahmed_Louati: <p>Is there any practical way to symbolically trace LSTM layers?<br>
Should I create my custom  LSTM with static sequence size?</p>

marksaroufim: <p>Yes that sounds about right</p>

Ahmed_Louati: <p>Thank you for your reply <a class="mention" href="/u/marksaroufim">@marksaroufim</a><br>
And then is it exportable to ONNX ?</p>
Separating out parts of pretrained model to send to separate processor

lorem: <p>I have a pretrained model that takes an input of vision and other state data and splits it. The vision data passes through an encoder before returning to the rest of the model, but this requires a fair amount of processing power and will be on relatively slow hardware. I have access to a separate processor that I want to offload the encoding task to, but for this, I would need to separate the model so that the encoder only gets the vision input, and the output is then fed to the rest of the model alongside the other state data inputs.</p>
<p>I haven’t found many sources that are helpful for this specific case and was hoping for some advice. I’ve already separated out the encoder by creating a new module for it, but my issue is saving the rest of the model in a way that it accepts the encoder output as an input.</p>
<p>Thank you in advance.</p>
Cross Hair filters from DeepVesselNet (memory not decreasing)

MLB9: <p>Hi all,<br>
I am using the Cross Hair filter approach from <a href="https://www.frontiersin.org/articles/10.3389/fnins.2020.592352/full" rel="noopener nofollow ugc">DeepVesselNet</a> for an instance segmentation project with UNet. The <a href="https://github.com/giesekow/deepvesselnet/blob/master/dvn/fcn.py" rel="noopener nofollow ugc">code</a> provided by DeepVesselNet is implemented for Keras and I am trying to do the same in PyTorch. I implemented the convolution with cross filters as follows:</p>
<pre><code class="lang-auto">class Conv3dCH(nn.Module):
    def __init__(self, in_channels: int, out_channels: int, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', device=None, dtype=None):
        super(Conv3dCH, self).__init__()
        self.convx = nn.Conv3d(in_channels, out_channels, (kernel_size[0],1,kernel_size[2]), stride=stride, padding=(padding, 0, padding), dilation=dilation, groups=groups, bias=False)
        self.convy = nn.Conv3d(in_channels, out_channels, (kernel_size[0],kernel_size[1],1), stride=stride, padding=(padding, padding, 0), dilation=dilation, groups=groups, bias=False)
        self.convz = nn.Conv3d(in_channels, out_channels, (1,kernel_size[1],kernel_size[2]), stride=stride, padding=(0, padding, padding), dilation=dilation, groups=groups, bias=False)

    def forward(self, x):
        cx = self.convx(x)
        cy = self.convy(x)
        cz = self.convz(x)
        out = torch.add(cx, cy)
        out = torch.add(out, cz)
        return out
</code></pre>
<p>Nevertheless, it seems that having 3 nn.Conv3d() with 2D filters uses more memory than a torch.nn.Conv3d(). Could this be true?</p>
<p>In this case, DeepVesselNets cross-hair’s advantage in memory and running time due to the decreased number of parameters would not happen. Could someone help with some insights on these and some better approach in PyTorch? Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="MLB9" data-post="1" data-topic="182682">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mlb9/48/60262_2.png" class="avatar"> MLB9:</div>
<blockquote>
<p>Nevertheless, it seems that having 3 nn.Conv3d() with 2D filters uses more memory than a torch.nn.Conv3d(). Could this be true?</p>
</blockquote>
</aside>
<p>Yes, I would assume this to be the case as the majority of the used memory would come from the intermediate forward activations while the actual filters are tiny. This of course depends on the input shape, but I also assume it’s larger than the kernel.</p>
<p>This also seems to be the same question as asked <a href="https://discuss.pytorch.org/t/unet-kernel-size-5-vs-kernel-size-3/182674/2">here</a>.</p>
Convert quantized model to ONNX format

Ahmed_Louati: <p>Hello,</p>
<p>I am working on quantizing a model using FX GraphModule mode.</p>
<ol>
<li>
<p>I wanna ask about the best methods to export it to ONNX format (if it is supported).<br>
Do I have to torchscript it ( torch.jit.trace OR torch.jit.export )? Or I just export it directly using torch.onnx.export API.</p>
</li>
<li>
<p>Are (dynamically) quantized LSTM/GRU layers/cells exportable to ONNX? (I saw that ONNX supports LSTM layers but not GRU)</p>
</li>
</ol>
<p>Looking forward to hearing from you!<br>
Peace,</p>

andrewor: <p>Hi Ahmed,</p>
<p>As far as I know exporting to ONNX is not officially supported, and we are not actively working on this integration. However, here’s a thread that you may find useful: <a href="https://discuss.pytorch.org/t/onnx-export-of-quantized-model/76884/32" class="inline-onebox">ONNX export of quantized model - #32 by tsaiHY</a>. I would guess more complex ops like LSTM/GRUs in particular are not well supported.</p>
<p>Best,<br>
-Andrew</p>

Ahmed_Louati: <p>Thanks a lot <a class="mention" href="/u/andrewor">@andrewor</a>  for your reply!</p>
<p>From what I have understood from the thread you sent me, QAT models are not exportable to ONNX.<br>
For the moment I am working with Post Training Quantization, it is still unsupported also?</p>
<p>Thank you!</p>

jerryzh168: <p>ONNX path is in general supported by our team, please see the response here: <a href="https://pytorch.org/docs/stable/quantization.html#frequently-asked-questions" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.0 documentation</a></p>
RuntimeError: quantized::conv(FBGEMM): Expected activation data type QUInt8 but got QInt8

Shunsheng_Lee: <p>Hello,<br>
I was trying to quantize a simple model with qint8 for both activations and weights, in a qconfig(2) way, because what I want to do is quantize-&gt;convert to onnx-&gt;deploy on tensorrt.</p>
<p>I replace<br>
(1)float_model.qconfig = torch.quantization.get_default_qconfig(‘fbgemm’)<br>
with<br>
(2)QConfig(activation=HistogramObserver.with_args(dtype=torch.qint8, qscheme=torch.per_tensor_symmetric), weight=default_per_channel_weight_observer)</p>
<pre><code class="lang-python">my model looks like:
class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.quant = torch.quantization.QuantStub()
        self.conv1 = nn.Conv2d(3, 16, 5)
        self.pool1 = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(16, 32, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(32*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
        self.relu1 = nn.ReLU()
        self.relu2 = nn.ReLU()
        self.relu3 = nn.ReLU()
        self.relu4 = nn.ReLU()
        self.flatten = nn.Flatten()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):            # input(3, 32, 32)
        x = self.quant(x)
        x = self.conv1(x)
        x = self.relu1(x)    #output(16, 28, 28)
        x = self.pool1(x)            # output(16, 14, 14)
        x = self.conv2(x)
        x = self.relu2(x)    # output(32, 10, 10)
        x = self.pool2(x)            # output(32, 5, 5)
#        x = x.view(-1, 32*5*5)       # output(32*5*5)
#        x = torch.reshape(x, (16, 32*5*5))
        x = self.flatten(x)
        x = self.fc1(x)
        x = self.relu3(x)      # output(120)
        x = self.fc2(x)
        x = self.relu4(x)      # output(84)
        x = self.fc3(x)              # output(10)
        x = self.dequant(x)
        return x
</code></pre>
<p>However, if I use (1)'s qconfig which is quint8, tensorrt will raise an error that it dosen’t support uint8</p>
<p>If I want to quantize on torch and deploy on tensorrt, what should I do? Could you tell me, please?<br>
Thanks!</p>

Nanlalalalalal: <p>Same problem here. I now use <a href="https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization" rel="noopener nofollow ugc">pytorch-quantization</a> instead. Waiting for solution.</p>

Nanlalalalalal: <p>It seems it is possible: <a href="https://github.com/pytorch/TensorRT/blob/main/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L37" class="inline-onebox" rel="noopener nofollow ugc">TensorRT/test_quant_trt.py at main · pytorch/TensorRT · GitHub</a><br>
However, it is Immature at this moment</p>

jerryzh168: <p>GPU inference is also one of our FAQs: <a href="https://pytorch.org/docs/stable/quantization.html#frequently-asked-questions" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.0 documentation</a></p>
Error in FX Graph Mode Quantization

Taehee_Jeong: <p>Hello,<br>
I tried to quantize opt model using FX Graph Mode Quantization.</p>
<h1>
<a name="model_fp-is-huggingface-opt-model-1" class="anchor" href="#model_fp-is-huggingface-opt-model-1"></a>model_fp is huggingface opt model.</h1>
<h2>
<a name="i-tried-to-follow-up-the-following-quantization-tutorial-2" class="anchor" href="#i-tried-to-follow-up-the-following-quantization-tutorial-2"></a>I tried to follow up the following quantization tutorial.</h2>
<p>model_to_quantize = copy.deepcopy(model_fp)<br>
model_to_quantize.eval()<br>
qconfig_mapping = QConfigMapping().set_global(torch.ao.quantization.default_dynamic_qconfig)</p>
<h1>
<a name="a-tuple-of-one-or-more-example-inputs-are-needed-to-trace-the-model-3" class="anchor" href="#a-tuple-of-one-or-more-example-inputs-are-needed-to-trace-the-model-3"></a>a tuple of one or more example inputs are needed to trace the model</h1>
<p>example_inputs = (input_fp32)</p>
<h1>
<a name="prepare-4" class="anchor" href="#prepare-4"></a>prepare</h1>
<h2>
<a name="model_prepared-quantize_fxprepare_fxmodel_to_quantize-qconfig_mapping-example_inputs-5" class="anchor" href="#model_prepared-quantize_fxprepare_fxmodel_to_quantize-qconfig_mapping-example_inputs-5"></a>model_prepared = quantize_fx.prepare_fx(model_to_quantize, qconfig_mapping, example_inputs)</h2>
<p>I got the following error.<br>
Could you help me how to fix this error?</p>
<p>File ~/transformers/models/opt/modeling_opt.py:625 in forward<br>
raise ValueError(“You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time”)</p>
<h2>
<a name="valueerror-you-cannot-specify-both-decoder_input_ids-and-decoder_inputs_embeds-at-the-same-time-6" class="anchor" href="#valueerror-you-cannot-specify-both-decoder_input_ids-and-decoder_inputs_embeds-at-the-same-time-6"></a>ValueError: You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time</h2>

HDCharles: <p>can you provide a repro or more details, that error seems unrelated to quantization given that its occuring in transformers/models/opt/modeling_opt directory</p>

jerryzh168: <p>I haven’t tried this, but I think you’ll need to use huggingface symbolic tracer to trace the model first, and then try quantizing with our API: <a href="https://github.com/huggingface/transformers/blob/main/src/transformers/utils/fx.py#L1209" class="inline-onebox" rel="noopener nofollow ugc">transformers/src/transformers/utils/fx.py at main · huggingface/transformers · GitHub</a></p>
Does pytorch eager QAT support LSTM network

yyd199948: <p>Does pytorch eager QAT support LSTM network? When I train LSTM with eager QAT, the training speed is more than ten times slower, but CNN, Linear training time has no effect。</p>

jerryzh168: <p>I think so, you can try following this: <a href="https://pytorch.org/docs/stable/quantization.html#quantization-custom-module-api" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.0 documentation</a></p>
HELP PLEASE! Please, where can I find the implementation of torch._C._nn._upsample_nearest_exact3d?

salvadeharo10: <p>Please, where can I find the implementation of torch._C._nn._upsample_nearest_exact3d???</p>

ptrblck: <p>Already answered <a href="https://discuss.pytorch.org/t/please-where-can-i-find-the-implementation-of-torch-c-nn-upsample-nearest-exact3d/182710/2">here</a>.<br>
Please don’t create double posts and use a proper title for your topics.</p>
Using torch.argsort in CNN

aditya_bhatt: <p>Output of my CNN network is log probabilities. I want to get 10 best indices of the output and further use the indices to get loss. Hence, I have to use torch.argsort. But using torch.argsort breaks gradients.<br>
Can anyone please help me on how do I proceed?<br>
Thank you very much for the help</p>

ptrblck: <p>That’s expected as <code>argsort</code> is not differentiable and returns indices.<br>
You could create a custom <code>autograd.Function</code> and implement the <code>backward</code> pass manually in case you want to “approximate” the gradient somehow.</p>

aditya_bhatt: <p>Will I need to code for the entire backward pass?<br>
That is will I have to code for backward pass of all the weights in CNN?</p>

ptrblck: <p>You would only need to implement the <code>backward</code> for <code>argsort</code> as it’s undefined.<br>
Note that the <code>backward</code> pass is undefined as <code>argsort</code> is returning the indices and thus the operation is not differentiable as small <code>eps</code> changes of the input will not result in any contiguous changes in its output, so the more interesting question is how you would implement the gradient.</p>
Is this code snippet thread safe with DataParallel?

Fei_Liu: <p>I have the following code snippet:</p>
<pre><code class="lang-python">
class Model(nn.Module):
  # omitting __init__

  # returns a scalar that can run backward on
  def expensive_call(self, x):
    # omiting detailed definition
    return some_scalar

  def forward(self, x):
    self.expensive_call(x).backward()

m = nn.DataParallel(Model().cuda())

# Is this thread-safe?
m(x)  # calling forward on input x, expecting a parallel backward

</code></pre>
<p>The motivation: I don’t care about the forward call return value and I specifically don’t want to gather the results back to device 0 and then do the backward. So I wrap the detailed logic into <code>expensive_call</code> and let <code>DataParallel</code> run backward inside forward for me (in a parallel fashion).</p>
<p>Despite the uncommon syntax, does this code actually work / is bug-free? I roughly know that GPU events are launched to cuda streams, and also there are some multithreading code under the hood of <code>DataParallel</code>. Are these cuda streams thread-safe?</p>
Extending FastRCNN model to use image + meta-data input

Georgiana_M: <p>I want to extend FastRCNN to use both image and metadata input (currently I have 3 features per image as meta-data). My VisitonDataset contains a tuple (image + metadata) and a tensor target (i.e. ((tensor_image, torch_meta), torch_target)).</p>
<p>I define model as:<br>
<strong>import torchvision.models.detection as models</strong><br>
<strong>model = models.fasterrcnn_mobilenet_v3_large_320_fpn(weights=“DEFAULT”)</strong><br>
<strong>model.roi_heads.box_predictor = models.faster_rcnn.FastRCNNPredictor(model.roi_heads.box_predictor.cls_score.in_features, len(classes))</strong></p>
<p>I tried adding a sequential module and an out module using:<br>
<strong>model.add_module(“MetaNet”, torch.nn.Sequential(collections.OrderedDict([</strong><br>
**                                (“lin1”, torch.nn.Linear(len(meta_features), 500)),**<br>
**                                (“batch1”, torch.nn.BatchNorm1d(500)),**<br>
**                                (“relu1”,torch.nn.ReLU()),**<br>
**                                (“dropout1”,torch.nn.Dropout(p=0.2)),**<br>
**                                (“lin2”,torch.nn.Linear(500, 250)),**<br>
**                                (“batch2”,torch.nn.BatchNorm1d(250)),**<br>
**                                (“relu2”,torch.nn.ReLU()),**<br>
**                                (“dropout2”,torch.nn.Dropout(p=0.2))])))**<br>
<strong>model.add_module(“OutNet”, torch.nn.Linear(4+250, 4))</strong></p>
<p>So not, apart form the general Fast RCNN modules, I get (at the end):<br>
<strong>…</strong><br>
<strong>(MetaNet): Sequential(</strong><br>
**    (lin1): Linear(in_features=3, out_features=500, bias=True)**<br>
**    (batch1): BatchNorm1d(500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)**<br>
**    (relu1): ReLU()**<br>
**    (dropout1): Dropout(p=0.2, inplace=False)**<br>
**    (lin2): Linear(in_features=500, out_features=250, bias=True)**<br>
**    (batch2): BatchNorm1d(250, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)**<br>
**    (relu2): ReLU()**<br>
**    (dropout2): Dropout(p=0.2, inplace=False)**<br>
**  )**<br>
**  (OutNet): Linear(in_features=254, out_features=4, bias=True)**</p>
<p>I want to concatenate the output of the FastRCNN with the Sequential model using metadata. How can I then update the forward (and backward) to train my classifier using the metadata.</p>
Please, where can I find the implementation of torch._C._nn._upsample_nearest_exact3d

salvadeharo10: <p>Please, where can I find the implementation of torch._C._nn._upsample_nearest_exact3d</p>

ptrblck: <p>The dispatching should be done <a href="https://github.com/pytorch/pytorch/blob/c16a28860f622a9c9793b2412c763d6d1e85d137/aten/src/ATen/native/UpSampleNearest3d.cpp#L176">here</a> which should then call into the backend. E.g. the CPU implementation can be found <a href="https://github.com/pytorch/pytorch/blob/c16a28860f622a9c9793b2412c763d6d1e85d137/aten/src/ATen/native/cpu/UpSampleKernel.cpp#L1707">here</a> and you could use the kernel names to check for other backends.</p>
Tensor mismatch error

Brooklyn_Amaro: <p>Hello. I am new to building neural networks. I am trying to build a neural network using Pytorch that has 11 inputs, 1 hidden layer with 11 neurons, and 2 outputs. Right now I am working on creating a tensor dataset with my given data but I’m having a hard time getting through the “AssertionError: Size mismatch between tensors”. Anything will help trying to get around this. My entire code so far is attached below:</p>
<p>import torch<br>
import numpy as np<br>
from torch import nn<br>
from torch import optim<br>
from torch.utils.data import TensorDataset<br>
import matplotlib.pyplot as plt<br>
import pandas as pd</p>
<p>‘’’<br>
Neural Network Structure:<br>
11 inputs<br>
11 neurons in the first hidden layer<br>
2 ouputs<br>
‘’’</p>
<h1>
<a name="get-data-from-tables-1" class="anchor" href="#get-data-from-tables-1"></a>Get data from tables</h1>
<h1>
<a name="split-data-up-into-training-and-testing-80-train-20-test-2" class="anchor" href="#split-data-up-into-training-and-testing-80-train-20-test-2"></a>Split data up into training and testing (80% Train, 20% Test)</h1>
<p>pnt_cont_data = pd.read_excel(‘C:\Users\bamar\Downloads\Chile_Research\Research_Resources\DoE_point_contact.xlsm’, sheet_name=‘Sample’, index_col = 0, names=[‘#’,‘E1’,‘E2’,‘v1’,‘v2’,‘Ap’,‘rho0’,‘mue0’,‘u1’,‘u2’,‘R’,‘Fn’,‘hmin’,‘hc’,‘p’])<br>
pnt_cont_data.drop(index = pnt_cont_data.index[0], axis = 0, inplace =True)<br>
del pnt_cont_data[‘p’]</p>
<h1>
<a name="turning-data-into-numpy-arrays-3" class="anchor" href="#turning-data-into-numpy-arrays-3"></a>Turning data into numpy arrays</h1>
<p>X = pnt_cont_data.to_numpy()[:,:-3]<br>
split = int(0.8*len(X))<br>
X_train_np, X_test_np = X[:split], X[split:]</p>
<p>y = pnt_cont_data.to_numpy()[:,11:]<br>
y_train_np, y_test_np = y[:split], y[split:]</p>
<p>‘’’<br>
print(X_train_np.shape, y_train_np.shape)<br>
print(X_test_np.shape, y_test_np.shape)</p>
<p>(800, 10) (800, 2)<br>
(200, 10) (200, 2)<br>
‘’’</p>
<p>X_train_float = X_train_np.astype(np.float32)<br>
X_train = torch.Tensor(X_train_float)</p>
<p>X_test_float = X_test_np.astype(np.float32)<br>
X_test = torch.Tensor(X_test_float)</p>
<p>y_train_float = y_train_np.astype(np.float32)<br>
y_train = torch.Tensor(y_train_float)</p>
<p>y_test_float = y_test_np.astype(np.float32)<br>
y_test = torch.Tensor(y_test_float)</p>
<p>train_dataset = TensorDataset(X_train, y_train)</p>

ptrblck: <p>The posted code doesn’t seem to be the entire code as the model is missing.<br>
Could you post a minimal and executable code snippet by wrapping it into three backticks ```?</p>
Every Module but Torch Will Import With Embedded Python

tempaccountforissue: <p>I’m trying to embed Python into a project I’m working on. Before embedding Python, the project is able to use my local install in this directory: <code>C:\Users\Name\AppData\Local\Programs\Python\Python39</code>. The project is able to run perfectly with this Python installation.</p>
<p>However, if I copy and paste the contents of the <code>Python39</code> folder into my project, every module except for Torch is able to be imported. If I try to import Torch, the application freezes.</p>
<p>What could be causing this, and how would I fix it?</p>
How to edit post or add more than 3 replys?

Robert_Grzelak: <p>How to edit post or add more than 3 reply?</p>
<p>You have reached the reply limit for this topic<br>
We’re sorry, but new users are temporarily limited to 3 replies in the same topic.<br>
Instead of adding another reply, please consider editing your previous replies, or visiting other topics.</p>
<p>Why I cannot edit or reply on…!!!</p>
<h1>
<a name="expected-tuple-but-got-stringhttpsdiscusspytorchorgtexpected-tuple-but-got-string179389-1" class="anchor" href="#expected-tuple-but-got-stringhttpsdiscusspytorchorgtexpected-tuple-but-got-string179389-1"></a><a href="https://discuss.pytorch.org/t/expected-tuple-but-got-string/179389">Expected Tuple but got String</a>
</h1>
<p>I don’t want to be new anymore!</p>

ptrblck: <p>You would need to participate more in this discussion board to reach the next trust level, which will remove some limitations.</p>

Robert_Grzelak: <p>Could you please share your code?</p>

Robert_Grzelak: <p>I used the 3 post limitation to paste the error trace. Now I cannot add a new one after deleting the old one as well as edit any of it.</p>
<p>Maybe you could help with this one:</p>
<aside class="quote quote-modified" data-post="1" data-topic="182516">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/robert_grzelak/48/61241_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/redisai-inference-attempt/182516">RedisAI inference attempt</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hello, I am getting redis.exceptions.ResponseError when trying to make inference on RedisAI using exported yolov5 torchscript  model, but got; 
isTuple()INTERNAL ASSERT FAILED at “…/aten/src/ATen/core/ivalue_inl.h”:1916, please report a bug to PyTorch. Expected Tuple but got String Exception raised from toTupleRef at …/aten/src/ATen/core/ivalue_inl.h:1916 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;)…
  </blockquote>
</aside>
Multiple Continuous Outputs from NN

DirkTheDS: <p>I need to create a NN with 8 continuous inputs (predictors) and 5 continuous outputs. The code below works great with one output. If I change the line “y = dataset[:,8]” to “y = dataset[:,8:10]” and resize the train and test sensor using “y = torch.tensor(y, dtype=torch.float32).reshape(-1, 2)” (so that there are two columns for outputs instead of one) then I obviously get errors. I’ve tried a zillion things, such as changing " y_pred = model(Xbatch)" to " y_pred1, y_pred2 = model(Xbatch)". Is there any best practice for this?</p>
<pre><code class="lang-auto">
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.model_selection import train_test_split
</code></pre>
<pre><code class="lang-auto">

# load the dataset, split into input (X) and output (y) variables
dataset = np.loadtxt('data.csv', delimiter=',')
X = dataset[:,0:8]
y = dataset[:,8]


X, x_test, y, y_test = train_test_split(X, y, test_size=0.2)
#x_test, x_val, y_test, y_val = train_test_split(x_test, y_test, test_size=0.5)
 
#train tensort
X = torch.tensor(X, dtype=torch.float32)
y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1)
 
#test tensor
x_test = torch.tensor(x_test, dtype=torch.float32)
y_test = torch.tensor(y_test, dtype=torch.float32).reshape(-1, 1)


# define the model
model = nn.Sequential(
    nn.Linear(8, 22),
    nn.ReLU(),
    nn.Linear(22, 22),
    nn.ReLU(),
    nn.Linear(22, 8),
    nn.ReLU(),
    nn.Linear(8, 1)#,
   # nn.Sigmoid()
)
print(model)
 
# train the model
loss_fn   = nn.L1Loss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
 
n_epochs = 200
batch_size = 100
 
for epoch in range(n_epochs):
    for i in range(0, len(X), batch_size):
        Xbatch = X[i:i+batch_size]
        y_pred = model(Xbatch)
        ybatch = y[i:i+batch_size]
        loss = loss_fn(y_pred, ybatch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>

KFrank: <p>Hi Dirk!</p>
<aside class="quote no-group quote-modified" data-username="DirkTheDS" data-post="1" data-topic="182599" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/bbe5ce/48.png" class="avatar"> DirkTheDS:</div>
<blockquote>
<p>I need to create a NN with 8 continuous inputs (predictors) and 5 continuous outputs.<br>
…</p>
<pre><code class="lang-auto">model = nn.Sequential(
    nn.Linear(8, 22),
    nn.ReLU(),
    nn.Linear(22, 22),
    nn.ReLU(),
    nn.Linear(22, 8),
    nn.ReLU(),
    nn.Linear(8, 1)#,
   # nn.Sigmoid()
)
</code></pre>
</blockquote>
</aside>
<p>Replace your final <code>Linear (8, 1)</code> layer with <code>Linear (8, 5)</code>.  That will produce<br>
5 continuous outputs for each sample you input.</p>
<p>Best.</p>
<p>K. Frank</p>

DirkTheDS: <p>Well that was easy. Thank you very much!</p>
How to know if an operation is blocking/synchronizing or not?

skandermoalla: <p>Where do we find documentation on whether an operation such as <code>cuda_tensor.item()</code>, <code>cuda_tensor.numel()</code>, <code>cuda_tensor.to("cpu")</code>, is blocking or not? (<a href="https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 2.0 documentation</a>).</p>
<p>Thanks!</p>

ptrblck: <p>You can set <code>torch.cuda.set_sync_debug_mode(mode)</code> to “warn” or “error” and execute the desired line of code.</p>

skandermoalla: <p>Awesome!!</p>
<pre><code class="lang-python">Python 3.10.11 | packaged by conda-forge | (main, May 10 2023, 18:58:44) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.set_sync_debug_mode("warn")
/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torch/cuda/__init__.py:762: UserWarning: Synchronization debug mode is a prototype feature and does not yet detect all synchronizing operations (Triggered internally at /opt/conda/conda-bld/pytorch_1682343967769/work/torch/csrc/cuda/Module.cpp:830.)
  torch._C._cuda_set_sync_debug_mode(debug_mode)
&gt;&gt;&gt; t = torch.arange(10, device="cuda")
&gt;&gt;&gt; z = t**2
&gt;&gt;&gt; z.numel()
10
&gt;&gt;&gt; x = z.float().mean()
&gt;&gt;&gt; m = x.item()
&lt;stdin&gt;:1: UserWarning: called a synchronizing CUDA operation (Triggered internally at /opt/conda/conda-bld/pytorch_1682343967769/work/c10/cuda/CUDAFunctions.cpp:148.)
&gt;&gt;&gt; 
</code></pre>
Convert image to a different image size

Oussama_Bouldjedri: <p>I am working on image classification using transformers; I have a data set of images with input size (128* 128),  I want to convert the images into a 224*224.</p>
<p>Thanks in advance</p>

ptrblck: <p>You could use <code>torchvision.transforms.Resize</code> to resize the images before feeding them into the model.</p>

Robert_Grzelak: <p>" You could use <code>torchvision.transforms.Resize</code> to resize the images before feeding them into the model."<br>
I agree!</p>
Numba vs jit vs compile

divinho: <p>Hi,<br>
If I need to write some code that runs on GPU and has for loops, should I be using numba or jit or torch compile in 2023?</p>
<p>When I search for info on this everything I find is outdated (pre pytorch 2.0).</p>

Robert_Grzelak: <p>Could you please share the code?</p>

divinho: <p>I want to optimize code like this: <a href="https://pytorch.org/audio/main/tutorials/forced_alignment_tutorial.html#find-the-most-likely-path-backtracking" class="inline-onebox" rel="noopener nofollow ugc">Forced Alignment with Wav2Vec2 — Torchaudio 2.1.0.dev20230622 documentation</a></p>
<p>If using <code>while</code> makes things a lot more difficult just assume I will find a way to use <code>for</code> instead</p>
UNet kernel_size=5 vs kernel_size=3

MLB9: <p>Hi all,</p>
<p>I am working on an image instance segmentation project. I have the exact same UNet model for kernel_size=3 (with padding=1) and for kernel_size=5 (with padding=2). I would expect that the model corresponding to the larger kernel required more memory (and computation time) due to the hight increase of number of trainable parameters, but this seems not to be happening significantly.</p>
<p>May my model be correct and not necessarily require more memory? Thank you in advance,</p>
<p>MLB9</p>

ptrblck: <p>The increase in memory usage between a kernel size of 3 and 5 might be tiny as the majority of the memory usage would come from the intermediate forward activations, which might have the same shape.</p>
How can I solve CUDA out of memory problem?

samm: <p>I got this problem and tried to reduce the batch_size but the error still<br>
<code>    labels=decoder_input_ids RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 5.80 GiB total capacity; 4.62 GiB already allocated; 2.56 MiB free; 4.72 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</code></p>
<p>the code is</p>
<pre><code class="lang-auto">dataloader = DataLoader(dataset, batch_size=4, shuffle=True)
num_epochs = 5


for epoch in range(num_epochs):
    model.train()
    total_loss = 0
    
    for batch in dataloader:
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        decoder_input_ids = batch['decoder_input_ids'].to(device)
        decoder_attention_mask = batch['decoder_attention_mask'].to(device)
        
        outputs = model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=decoder_attention_mask,
            labels=decoder_input_ids
        )
        loss = outputs.loss
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        total_loss += loss.item()
    
    avg_loss = total_loss / len(dataloader)
</code></pre>

ptrblck: <p>You would need to reduce the batch size further or if that’s not possible, you come or either use a model with a lower memory footprint, or could check e.g. <code>torch.utils.checkpoint</code> to trade compute for memory.</p>
Proper way to manage memory in multi-process scenario

Scolpe: <p>Dear Community,</p>
<p>I wanted to ask about proper methods of GPU memory management in a distributed scenario, especially 1) how to pass gradients between processes, 2) how to track the use of memory space for different processes, 3) the recommended method of releasing the reserved memory if the underlying code is extensively used deep copies of the original tensors.</p>
<p>I am currently tasked with writing a Python library for simulating a distributed learning process - and I use PyTorch as a boilerplate for neural networks training. I initialise a net, then deep copy it to n clients and perform a number of training rounds.</p>
<p>The executed code relies heavily on using a number of workers training simultaneously and then returning the model’s weights. As it is, for now, the code relies heavily on the deep copies of the gradients, as the PyTorch does not allow to pass the gradients from child processes otherwise (in short, I must perform a deep copy of gradients before sending them back to central orchestrator).</p>
<p>However, I have observed that I suffer from a ‘memory leak’ somewhere. Although it is not a real memory leak, it seems that my code is not releasing the reserved memory in a proper manner, and the reserved memory space is growing bigger over time. This can lead to a following error:</p>
<pre><code class="lang-auto">torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 23.65 GiB total capacity; 10.33 GiB already allocated; 45.06 MiB free; 11.40 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
</code></pre>
<p>It seems that my code is not working correctly at the level of realising properly shared CUDE tensors.</p>
<p>Therefore, I would like to ask for tips and recommendations for debugging this problem. Is there a way to inspect how much memory each device and process is currently reserving? Moreover, what would be the recommended way to initialize and pass tensors between child processes? I’ve inspected the documentation, and it seems that creating deep copies will increase memory usage - but it should not accumulate over time as it is now. Is there a was to ‘force-release’ memory from selected devices?</p>
<p>I am very grateful for all your help - as the usage of devices and memory allocation is much more tricky in a distributed scenario.</p>
Gpu memory gets accumulated during consecutive forward passes

mailcorahul: <p>I am running a densenet161 image classifier on gpu.</p>
<p>After continuous forward passes(basically 2-5 batches of images, batch_size=128), the memory gets accumulated and doesn’t get freed. Can someone tell me what is taking up so much memory?<br>
Btw, I am using <code>torch.no_grad()</code> during forward pass.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> It would also be helpful if you can tell the factors which determine memory usage during inference.</p>

Diego: <p>Could you post a snippet of the code where you do the forward pass, so that we can take a look?</p>

mailcorahul: <p>I am using an Inference class and the following code is one of its member function. Every batch is of size 128.</p>
<pre><code class="lang-auto">def predict(image_objs_batches):
    for batch in image_objs_batches:
        with torch.no_grad():
            _, outputs = self.model(batch)
</code></pre>

Diego: <p>Are you sure the memory leak is on the inference stage? If not could you post the training loop.  If you accumulate the losses on a list or something similar without calling <code>.item()</code> on it the gradients will accumulate.</p>

mailcorahul: <p>I am facing this memory accumulation issue only during inference. Not for training.</p>

rustagiadi95: <p>Hi there,</p>
<p>_, outputs = self.model(batch) is not within the context manager as per your snippet.<br>
Put a tab on the line to push it within the torch.no_grad() context manager.</p>
<p>Since you asked the other factors, one could be the batch size, in your case this might mot be the solution, but increasing the batch size means computing and keeping more gradients on your device, eventually eating up device memory. This is keeping in mind that you had a pretrained model and did not trained on your device, you are only inferencing.</p>
<p>Thanks</p>

mailcorahul: <p>Hey!<br>
Yes, there is a tab. Somehow got rearranged while copying code. Inference is being done with <code>torch.no_grad()</code> context manager.</p>
<p>I am pretty sure gradients are not computed, but gpu memory steadily increases during consecutive forward passes.<br>
Thanks.</p>

Gantavya_Bhatt1: <p>Hey, any update on this? I am facing the same issue, even with the with torch.no_grad() snippet, with the latest version 1.9.</p>

ptrblck: <p>Could you post a minimal and executable code snippet, which would show the increase in memory in PyTorch <code>1.9.0</code> using the <code>no_grad</code> guard, please?</p>

Gantavya_Bhatt1: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your response. I figured out that the issue is not related to the forward pass, but rather PyTorch was caching the memory, and therefore I saw an increased memory footprint on nvidia-smi. I have figured out the solution. Thanks once again.</p>

Zhiyi_Chen: <p>Hi, may I know what is your solution? I have the same issue at inference!</p>

Zige_Wang: <p>Hi, same issue here, does anyone know how to deal with it?</p>
Python Compatibility Issue with Intel Arc Graphics: Seeking Solutions and Insights

Stella_Gomez: <p>Hello everyone,</p>
<p>I hope you’re all doing well and enjoying your programming journeys. Today, I come seeking your expertise and insights regarding a compatibility issue I’ve encountered with Python on my <a href="https://www.lenovo.com/us/en/faqs/intel/intel-graphics/" rel="noopener nofollow ugc">Intel Arc Graphics</a>.</p>
<p>Recently, I upgraded my system to include the impressive Intel Arc GPU, eagerly anticipating a boost in performance for both gaming and deep learning tasks. However, upon diving into my Python projects, I noticed some unexpected hiccups. It appears that there might be compatibility issues between Python and the Intel Arc Graphics drivers.</p>
<p>Specifically, when running certain Python scripts or utilizing libraries that heavily rely on GPU acceleration, I’ve experienced crashes, unexpected behavior, or even system instability. As someone who heavily relies on Python for machine learning and data analysis, this is quite concerning.</p>
<p>I wanted to reach out to this knowledgeable community to see if any of you have encountered similar issues or have any insights on potential workarounds. Have you managed to successfully leverage Python with Intel Arc Graphics, or have you faced any hurdles along the way?</p>
<p>I’ve made sure to keep my system drivers up to date, including the Intel Arc Graphics drivers, but the issue persists. I’ve also experimented with different Python versions and libraries, but haven’t found a definitive solution yet.</p>
<p>If any of you have suggestions, workarounds, or troubleshooting tips, I would greatly appreciate your input. Perhaps there are specific configurations, driver versions, or patches that could alleviate this compatibility issue?</p>
<p>Let’s come together and brainstorm possible solutions. If you have any relevant experiences or knowledge to share, please do not hesitate to contribute to this discussion. Your insights might just be the key to helping me and others in the community overcome this Python-Intel Arc Graphics hurdle.</p>
<p>Thank you all in advance for your time and support. Let’s tackle this challenge together!</p>
<p>Best regards,<br>
Stella</p>
Is nn.utils.vector_to_parameters() not differentiable?

Konstantin_H: <p>Hi guys,</p>
<p>I have some difficulties with backpropagation and the function nn.utils.vector_to_parameters() .</p>
<p>Say I have a parameter mu, perform some operations on it and finally want to insert it as weights and biases into a network (net).<br>
Then I pass a batch of data x through the network, compute a loss and backpropagate the error back to mu (the original, flattened parameter).</p>
<p>Here is a minimal code example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

x = torch.ones((1, 8)) # input example

net = nn.Sequential(nn.Linear(8, 8))
size=sum(p.numel() for p in net.parameters() if p.requires_grad)
mu = nn.Parameter(torch.ones((size,))*0.05, requires_grad=True)
# perform some operations on mu

nn.utils.vector_to_parameters(mu, net.parameters())

y = net(x)
optimizer = optim.SGD([mu], lr=1e-3)
loss = y.sum()
loss.backward()
optimizer.step()

print(mu)
print(mu.grad)


</code></pre>
<p>After the update step, mu remains the same as before, i.e. it is not getting optimized. Also, mu.grad returns None.<br>
Is the function nn.utils.vector_to_parameters(mu, net.parameters()) stopping the gradient flow, and if so, is there an alternative way to insert mu as “external”, flattened parameter into the weights of the network?</p>
<p>Thank you all!</p>

ptrblck: <p>This method uses the <code>.data</code> attribute internally:</p>
<pre><code class="lang-python">param.data = vec[pointer:pointer + num_param].view_as(param).data
</code></pre>
<p>and thus Autograd won’t capture this operation.<br>
You could check if e.g. <code>torch.nn.utils.parametrize</code> might work as described in <a href="https://discuss.pytorch.org/t/averaging-models-with-differentiable-weighting-coefficients/177240/3">this post</a>.</p>

Konstantin_H: <p>Thanks, this solved the problem!</p>
<p>I also found another solution (with maybe a little less overhead) here <a href="https://discuss.pytorch.org/t/hypernetwork-implementation/152109/5" class="inline-onebox">Hypernetwork implementation - #5 by ID56</a> .<br>
Inserting the weights and biases manually via torch.nn.functional.linear also works!</p>
Adaptive Pooling with MPS Backend on Mac M1 Max: Input Sizes Must Be Divisible by Output Sizes

ControX: <p>Hello,</p>
<p>I’m trying to run a FastAI model on a Mac M1 Max using PyTorch with the MPS backend. However, I’m encountering an error with an adaptive pooling layer: “Adaptive pool MPS: input sizes must be divisible by output sizes.”</p>
<p>I understand this is due to a limitation of the MPS backend, but I’m wondering if there’s a workaround or solution that would allow me to use adaptive pooling without this restriction, while still utilizing the GPU on the M1 Max.</p>
<p>I’ve tried setting the <code>PYTORCH_ENABLE_MPS_FALLBACK=1</code> environment variable to enable CPU fallback for unsupported operations, but I’m still encountering the error.</p>
<p>Any advice or suggestions would be greatly appreciated:)</p>
<p>Kind regards Patrik</p>
Regarding the compilation of ibtorch dynamic libraries, I hope to receive an answer from pytorch's package management personnel

may_ming: <p>When I was porting a PC architecture project to Jetson, I encountered an issue with the lack of functions in the dynamic library of arm 64 libtorch. I used the official libtorch shared with deps 1.11.0% 2Bcu113.zip on the PC, and after decompressing this compressed package, the libtorch in lib_ Cuda_ Cu.so and libtorch_ Cuda_ Cpp.so has 2.7G, but I used build myself from the Pytorch project_ The two files compiled by libtorch. py only have 170MB. How can I adjust the environment variables of CMakeLists.txt to make these two files larger, or how can developers release libtorch shared with deps 1.11.0% 2Bcu113.zip</p>

may_ming: <p>help,please，This is really important to me</p>

ptrblck: <p>The binary build scripts are located in <a href="https://github.com/pytorch/builder"><code>pytorch/builder</code></a> and you could check the <code>libtorch</code> folder.</p>
How to use ResNet50_Weights from torchvision on an MLP model?

peony: <p>I have this idea where I want to use resnet weights <code>ResNet50_Weights</code> from torchvision to my custom MLP model. I am not very sure whether this make sense or if its possible. Is it possible to initialize the weights of my MLP model with <code>ResNet50_Weights</code>? I am asking here because there is no clue online on how to do this.</p>
<p>Any insight is welcome. Thank you in advance</p>

ptrblck: <p>You can certainly load parameters from one model to another one assuming their shape matches.<br>
In your use case you might be able to load some linear layer parameters, but would need to make sure these layers were initialized with the same in and out features.</p>