akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can't load the saved model

username: 1004dasom
cooked: <p>Hello i’m new in using PyTorch bcs my assignment</p>
<p>i had my model saved and load but the error wasn’t the same like when i run it first. The error was 7,5% but when i rerun the error has changed to 9%. Can someone help me? this is my code</p>
<pre><code class="lang-auto">model = get_model('gru', model_params)
model.load_state_dict(torch.load("model.pth")['model'])

loss_fn = nn.MSELoss(reduction="mean")
optimizer = optim.Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay)
optimizer.load_state_dict(torch.load("model.pth")['optimizer'])

opt = Optimization(model=model, loss_fn=loss_fn, optimizer=optimizer)
opt.train(train_loader, val_loader, batch_size=batch_size, n_epochs=n_epochs, n_features=input_dim)
opt.plot_losses()

predictions, values = opt.evaluate(
    test_loader_one,
    batch_size=1,
    n_features=input_dim
)
</code></pre>

username: deependujha
cooked: <p>btw, why are you loading optimizer state. May be you should once try to not load optimizer_state, or zero grad it.</p>
<p>My another guess, make sure your dataloader is consistent. Make use of random_seeds in places where there can be change in behaviour in each run.</p>
Guidance for RL course & torchRL

username: deependujha
cooked: <p>I’m very comfortable with pytorch, torchvision and torchtext.<br>
Now, I’m trying to get my hands on torchRL &amp; be a confident reinforcement learning engineer. My current roadmap is:</p>
<ul>
<li><a href="https://rail.eecs.berkeley.edu/deeprlcourse/" rel="noopener nofollow ugc">CS-285 course by Sergey Levine</a></li>
<li>RL book by Sutton &amp; Barto</li>
<li>and, then, torchRL documentation.</li>
</ul>
<p>Is this a good approach? Please give your feedback. Your small guidance may save me weeks.</p>
Failed to load image Python extension: Filename or extension too long

username: Ved_Patel
cooked: <p>Hi,</p>
<p>I am trying to build dream fusion in my local. I keep getting the following error.</p>
<blockquote>
<p>site-packages\torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: '[WinError 206] The filename or extension is too long: ‘C:\…\lib\site-packages\torchvision’'If you don’t plan on using image functionality from <code>torchvision.io</code>, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have <code>libjpeg</code> or <code>libpng</code> installed before building <code>torchvision</code> from source?</p>
</blockquote>
<p>I have a windows system and I have tried enabling <code>LongPathsEnabled</code> in the registory. I even tried creating symbolic link to shorter the path, but the error still persists.</p>

username: Ved_Patel
cooked: <p>I reinstalled the pytorch library from scratch and it started working.</p>
Why does ignore_index ignore the entire example and not the class?

username: Andrew_Holmes
cooked: <p>I was looking into the backward pass of the CrossEntropyLoss class and noticed that the ignore_index just zeros an entire row (example) instead of the column (classes) for the gradient. I’m wondering if this is intended behavior? To me it seems that the class that we don’t want to compute the gradient for should just have its column set to zero while leaving the others unaffected. But it seems the row that’s ignore_index is being ignored, and every single other class with it. My guess is, well you don’t want to compute the gradients for an example containing the class you want to ignore, thus you ignore the entire example. The thing is, don’t those gradients from the other classes matter still, that we’rent ignore_index?</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as torchf

  ignore_index = 0
  x = torch.arange(12).reshape(4, 3).float().requires_grad_()  # random data
  y = torch.tensor([1, 2, 0, 1])  # 3 classes
  loss = torchf.cross_entropy(x, y, ignore_index=ignore_index)
  print(loss)
  loss.backward()
  print(x.grad)
</code></pre>
<pre><code class="lang-auto">python3 tests.py
tensor(1.0743, grad_fn=&lt;NllLossBackward0&gt;)
tensor([[ 0.0300, -0.2518,  0.2217],
        [ 0.0300,  0.0816, -0.1116],
        [ 0.0000,  0.0000,  0.0000],
        [ 0.0300, -0.2518,  0.2217]])
</code></pre>
<p>Why not?</p>
<pre><code class="lang-auto">tensor([[ 0.0000, -0.2518,  0.2217],
        [  0.0000,  0.0816, -0.1116],
        [ 0.0000,  0.abcd,  0.wxyz],
        [ 0.0000, -0.2518,  0.2217]])
</code></pre>
<p>abcd and wxyz are just place holders for whatever those values would have been if they we’rent zeroed.</p>

username: srishti-git1110
cooked: <p>Hi Andrew,<br>
Please note that <code>ignore_index</code> is used to ignore the <strong>target class</strong> that we don’t want to contribute towards the gradient calculation. In your case, since you want to ignore class 0, the input gradients of the 3rd example in the batch became 0 as the 3rd example has class 0 as the ground truth.</p>
Another install/build thread

username: TheMegaWojas
cooked: <p>Hello<br>
Win10, libtorch cpu debug, visual studio 2022<br>
I have intalled and build LibTorch using instructions on website. The example-app builds fine, creates the example-app.exe which also works and prints random tensor. Cmake creates a solution that consists of 3 parts: Zero Check, example app and All build. When i open the solution with visual studio and try to run it it throws Cannot find file error and points to All Build. When i delete All build and Zero check from the solution the example-app runs fine in Visual stuido. I can even copy-paste an example program from example repo and it runs. I have also tried the visual studio template which creates a new project . Linking against the libtorch directory seems to be ok but if i try to run any libtorch code i get an error saying i have a wrong version of windows sdk.</p>
<ol>
<li>Do i have to use cmake and generate the example-app everytime i want to start a new project ?</li>
<li>Is it possible to make it work like for example opencv installed via vcpkg? I mean no configuration at all, just <span class="hashtag-raw">#include</span>?</li>
</ol>
<p>Sorry for noob questions and thanks in advance<br>
Happy Easter</p>
How to delete nodes in torchsctipt using pytorch

username: MartinZhang
cooked: <p>How to delete nodes in torchsctipt using pytorch？</p>
<p>Is there an interface that can be called?</p>

username: ptrblck
cooked: <p>No, I’m not aware of an interface to manipulate the <code>TorchScript</code> graph, but also note that <code>TorchScript</code> is in maintenance mode and the currently recommended approach is to use <code>torch.compile</code>.</p>
Cannot download EMNIST dataset via torchvision dataset using Google Colab

username: Justin_Goh
cooked: <p>As the topic suggests, I am trying to download the EMNIST dataset using via:</p>
<p>dataset = datasets.EMNIST(root=‘data/’,split=‘letters’, download=True)</p>
<p>However I get the following error: RuntimeError: File not found or corrupted.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d87d8f66bca748c11408ad2eea71780105c4873b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d87d8f66bca748c11408ad2eea71780105c4873b" title="Screenshot 2024-03-22 at 1.12.34 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d87d8f66bca748c11408ad2eea71780105c4873b_2_690x223.jpeg" alt="Screenshot 2024-03-22 at 1.12.34 PM" data-base62-sha1="uTa8ZsPp25UUjqRVl19Yzil3Akr" width="690" height="223" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d87d8f66bca748c11408ad2eea71780105c4873b_2_690x223.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d87d8f66bca748c11408ad2eea71780105c4873b_2_1035x334.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d87d8f66bca748c11408ad2eea71780105c4873b_2_1380x446.jpeg 2x" data-dominant-color="3D3D3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-03-22 at 1.12.34 PM</span><span class="informations">1920×623 80.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Wondering if anyone else has encountered a similar problem.</p>

username: Lev_Klem
cooked: <p>similar problem<br>
I tried to download it and put it in the folder /kaggle/input/emnist/EMNIST/raw/gzip<br>
(it try download it to /kaggle/working/data/EMNIST/raw/gzip.zip)<br>
but it didn’t work((<br>
Did you manage to find a solution?</p>

username: ptrblck
cooked: <p>Seems to be related to <a href="https://github.com/pytorch/vision/issues/5662">this older issue</a>.<br>
CC <a class="mention" href="/u/pmeier">@pmeier</a> as I can reproduce it.</p>
Computing Hessian

username: Mayank1
cooked: <pre><code class="lang-auto">x = torch.ones(1)
x = Variable(x,requires_grad=True)
y = x*x
y.backward()
torch.autograd.grad(x.grad,x)
</code></pre>
<p>Why does this method of computing hessian not work?</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

username: srishti-git1110
cooked: <p>Hi Mayank,<br>
That code will not work as the error message clearly says.</p>
<p>As I understand it, you want to calculate the double derivative of a function <code>y</code> wrt the input <code>x</code>. Here’s how it could be done:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

x = torch.ones(1, requires_grad=True)
y = x * x

first_grad = torch.autograd.grad(y, x, create_graph=True)[0] # first derivative. also note the use of create_graph=True is to ensure that the graph of the first derivative is constructed which is required compute higher order derivatives

second_grad = torch.autograd.grad(grad_y, x)[0] # this is what you need
</code></pre>
What implementation is used for `matrix_exp`?

username: SamRoss
cooked: <p>The documentation <a href="https://pytorch.org/docs/stable/generated/torch.linalg.matrix_exp.html#torch.linalg.matrix_exp" rel="noopener nofollow ugc">here</a> explains what the matrix exponential is, but not how it’s implemented. I know this is a massive area of research, so I’d like to know which particular implementation is used.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>I think you can find the implementation <a href="https://github.com/pytorch/pytorch/blob/7f18ef14c1fed4e4376a75d626d98ba3c074809c/aten/src/ATen/native/LinearAlgebra.cpp#L2296-L2402">here</a>.</p>

username: Lezcano
cooked: <p>For reference, we use the algorithm suggested in the paper <a href="https://www.mdpi.com/2227-7390/7/12/1174" class="inline-onebox" rel="noopener nofollow ugc">Mathematics | Free Full-Text | Computing the Matrix Exponential with an Optimized Taylor Polynomial Approximation</a></p>

username: SamRoss
cooked: <p>Thank you! Do I understand correctly that the bottleneck in this approximation is essentially a constant number of matrix multiplications?</p>

username: Lezcano
cooked: <p>Depends. Does your computation happen on CPU or CUDA? In the former, the algorithm can be understood as a number of mms. In the latter, you have to think about things like synchronisations and so on. The current implementation could do a better job when it comes to these really.</p>

username: luolanxinwc22
cooked: <p>Is it possible to add the pade-based matrix_exp method to pytorch? I found that the tylor-based method has poor computational accuracy when the matrix order exceeds 100.</p>

username: ptrblck
cooked: <p>Contributions are welcome, so you could create a feature request on GitHub describing your implementation and advantage, in case you are interested in implementing it.</p>

username: redwrasse
cooked: <p>Is there a reference somewhere for why the current implementation for matrix exponential was chosen? Have been going through issues like <a href="https://github.com/pytorch/pytorch/issues/9983" class="inline-onebox" rel="noopener nofollow ugc">[feature request] Add matrix functions · Issue #9983 · pytorch/pytorch · GitHub</a> to try to find where a discussion of the various choices of implementation may have existed (or still exist).</p>
<p>It may be worthwhile to explicitly add test-cases of the limitations of the current/given implementation, for example related to the above comments ‘taylor-based method has poor computational accuracy when the matrix order exceeds 100.’</p>
On reusing buffers passed through a queue

username: Claudio_dos_Santos_F
cooked: <p>Hello folks! I’m reading the multiprocessing documentation and found what seems to be conflicting information. Specifically, <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">this page</a> states that:</p>
<blockquote>
<p>Even if you have a pool of processes sending data to a single one, make it send the buffers back - this is nearly free and will let you avoid a copy when sending next batch.</p>
</blockquote>
<p>However, in my understanding, this <a href="https://pytorch.org/docs/stable/multiprocessing.html#multiprocessing-cuda-sharing-details" rel="noopener nofollow ugc">other page</a> is incisive in recommending the opposite:</p>
<blockquote>
<p>Don’t pass received tensors.</p>
<pre><code class="lang-auto"># not going to work
x = queue.get()
queue_2.put(x)

# you need to create a process-local copy
x = queue.get()
x_clone = x.clone()
queue_2.put(x_clone)
</code></pre>
</blockquote>
<p>Could someone help me understand what I’m missing here?</p>
How to add an additional layer on top of a pretrained model?

username: Kong
cooked: <p>How can I append an additional layer on top of an existing pretrained architecture ? I saw this snippet below from <a href="https://discuss.pytorch.org/t/using-freezed-pretrained-resnet18-as-a-feature-extractor-for-cifar10/19177">Using freezed pretrained resnet18 as a feature extractor for cifar10</a> but it modifies the last layer instead of appending a new one to it.</p>
<pre><code class="lang-auto">model = torchvision.models.vgg19(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
    # Replace the last fully-connected layer
    # Parameters of newly constructed modules have requires_grad=True by default
model.fc = nn.Linear(512, 8) # assuming that the fc7 layer has 512 neurons, otherwise change it 
model.cuda()
</code></pre>
<p>In a later part of the code, the model was called as follows when processing the input data</p>
<p><code>output = model(input)</code></p>

username: RicCu
cooked: <p>A very simple way is</p>
<pre><code class="lang-auto">model = nn.Sequential(models.vgg19(True), YourModule())
# Use it as usual
out = model(input)
</code></pre>

username: solsol
cooked: <aside class="quote no-group quote-modified" data-username="Kong" data-post="1" data-topic="21303">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kong/48/4215_2.png" class="avatar"> Kong:</div>
<blockquote>
<p>nn.Sequential</p>
</blockquote>
</aside>
<p>Hi,<br>
Also is it possible to add one conv layer to vgg19 in this way?<br>
Thanks!</p>

username: Phuc_Pham
cooked: <p>hi. What is “YourModule()”? Could you pls explain to me?</p>

username: RicCu
cooked: <p><code>YourModule()</code> refers to any function that returns a <code>nn.Module</code>. So that’s just a placeholder to say that you can insert whatever model you want in there.</p>

username: Aayush_Agrawal
cooked: <p>In this snippet how do we make sure that the output dimension <code>models.vgg19</code> is in sync with the input dimension of <code>YourModule()</code>?</p>

username: Yaying_Luo
cooked: <p>I have the same problem.</p>

username: pdc-quantum
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"> Thanks a  lot RicCu. I was stuck in trying to modify the original end-classifier outputting 600 features in a hybrid classical-quantum transfer learning model. I needed a binary output. I used : model_hybrid = nn.Sequential(model, DressedQuantumNet()), and it worked!</p>
How can I have both gloo and nccl backend in torch.distributed?

username: youkaichao1
cooked: <p>I have 2 processes in one node, and I want to set up two different communications between them: one with <code>gloo</code> to communicate tiny messages with CPU, and one with <code>nccl</code> to communicate large tensors for computation. Is it possible?</p>
<p>When I call <code>dist.init_process_group</code> for twice, once with <code>backend="gloo"</code> and <code>backend="nccl"</code>, I get an error saying that I cannot initialize it twice.</p>

username: youkaichao1
cooked: <p>This is what I tried, but it does not work:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.distributed as dist
import torch
dist.init_process_group(backend="gloo")

group1 = dist.new_group(backend="gloo")
data1 = torch.tensor([1] * dist.get_rank())
dist.all_reduce(data1, group=group1)
print(f"answer from rank {dist.get_rank()}: avg {data1.mean().cpu().item()}")

group2 = dist.new_group(backend="nccl")
torch.cuda.set_device(dist.get_rank())
data2 = torch.ones((1024, 1024, 1024), device=f"cuda:{dist.get_rank()}")
dist.all_reduce(data2, group=group2)
print(f"answer from rank {dist.get_rank()}: avg {data2.mean().cpu().item()}")
</code></pre>
<p>It prints out <code>answer from rank 0: avg nan</code> , and then hangs.</p>

username: youkaichao1
cooked: <p>It’s quite strange that this code works:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.distributed as dist
import torch
dist.init_process_group(backend="gloo")
group2 = dist.new_group(backend="nccl")
torch.cuda.set_device(dist.get_rank())
data2 = torch.ones((1024, 1024, 1024), device=f"cuda:{dist.get_rank()}")
dist.all_reduce(data2, group=group2)
print(f"answer from rank {dist.get_rank()}: avg {data2.mean().cpu().item()}")
</code></pre>
<p>But this code does not:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.distributed as dist
import torch
dist.init_process_group(backend="gloo")

group1 = dist.new_group(backend="gloo")
data1 = torch.tensor([1] * dist.get_rank())
dist.all_reduce(data1, group=group1)
print(f"answer from rank {dist.get_rank()}: avg {data1.mean().cpu().item()}")
</code></pre>
<p>Can I create arbitrary groups for arbitrary times?</p>
<p>I’m afraid that users might called <code>dist.init_process_group</code> themselves, and I don’t want to interfere with them. Therefore, I want to create a process group with <code>gloo</code> backend, no matter what users do to the <code>torch.distributed</code> module.</p>

username: youkaichao1
cooked: <p>Oh, there is a bug for my code: <code>torch.tensor([1] * dist.get_rank())</code> should be <code>torch.tensor([1.0 * dist.get_rank()])</code> . After fixing the bug, the code works fine.</p>
Streaming video and audio in torchaudio

username: miraodasilva
cooked: <p>Hi,</p>
<p>I’m trying to use torchaudio.io.StreamReader(), followed by add_basic_video_stream and add_basic_audio_stream, to stream video and audio from the webcam/microphone simultaneously <em>on Linux</em>.</p>
<p>There are a few tutorials out there on how to do this on Windows or Mac, but there seems to be none for Linux. Specifically, I’m confused about what I should select as format= and src= in StreamReader(). Through a bit of trial and error, I have figured out that I can use src=/dev/video0 and format=v4l2 to stream video, but can’t figure out what src and format I should use to have both streams - video and audio - simultaneously.</p>
<p>Any help would be greatly appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: miraodasilva
cooked: <p>I suppose the first question here would be how to stream audio using StreamReader, i.e., which src and format I should use (can’t seem to figure this out and there seem to be no examples online for Linux), and then the second question would be how to combine the video and audio srcs and formats.</p>

username: roboav8r
cooked: <p>I had this issue as well. First, I did a <code>pacmd list sources</code> to find my device card. I have a 6-channel array on card 4 that samples at 16000 Hz.</p>
<p>To get the StreamReader started, I did the following:<br>
<code>streamer = torchaudio.io.StreamReader(src="hw:4, format="alsa", option={"sample_rate":"16000","channels":"6"})</code></p>
<p>The actual option names come directly from the FFMPEG alsa options here: <a href="https://ffmpeg.org/ffmpeg-devices.html#Options" rel="noopener nofollow ugc">FFmpeg Devices Documentation</a></p>
Explosive gradient in DenseNet

username: Abhishek_Gupta
cooked: <p>Recently I am training my DenseNoet model for image classification for emotion detection. I am using Adam optimizer. Problem is that I have put learning rate as low as 1e-16 and max_norm as 10 in nn.clip_grad_norm_. and training is still showing abrupt change in training accuracy.</p>
<p>Anyone who has faced the same. Please help me.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff4858bc691302ea211148bab7943e02e984b70c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff4858bc691302ea211148bab7943e02e984b70c" title="Screenshot 2024-03-29 at 10.31.32 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff4858bc691302ea211148bab7943e02e984b70c_2_512x500.png" alt="Screenshot 2024-03-29 at 10.31.32 PM" data-base62-sha1="AqkKVuV5vrNupTzxXf39s7LvadC" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff4858bc691302ea211148bab7943e02e984b70c_2_512x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff4858bc691302ea211148bab7943e02e984b70c_2_768x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff4858bc691302ea211148bab7943e02e984b70c_2_1024x1000.png 2x" data-dominant-color="404143"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-03-29 at 10.31.32 PM</span><span class="informations">1256×1226 262 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Is it possible to freeze some filters/channels of a CNN and unfreeze other filters/channels?

username: pytorchuser101
cooked: <p>I am training a CNN model using Pytorch.<br>
I have identified filters with low variance/magnitude for my input dataset and want to train only these filters for new auxiliary data.<br>
I want to freeze the rest of the filters/channels in the give layer.</p>
<p>How could I go about achieving this ? I did not find any latest PyTorch implementation for the same</p>

username: aririri
cooked: <p>Set requires_grad for the parameters of your layer as false before you backprop your gradients:</p>
<pre><code class="lang-auto">for param in ~your layer~.parameters():
            param.requires_grad = False
</code></pre>
Question about quint8 and qint8

username: jackie_huang
cooked: <p>I have obtained quantization parameters through PyTorch quantization and now I want to perform inference based on these parameters. However, I have encountered an issue where the quantized result of a layer is greater than 128, for example, 200, and PyTorch represents this value using quint8. Additionally, some computed values result are 0, such as after the ReLU activation of negative numbers.</p>
<p>During the computation of the next layer, I should subtract the zero point from the input. However, since the zero point is not zero, let’s say it is 10, subtracting 10 from 0 results in -10, which cannot be represented using uint8 and requires int8 representation. On the other hand, subtracting 10 from 200 results in 190, which can be represented using uint8. This creates a contradiction, and I’m unsure how to perform the calculations correctly.</p>
<p>For example, in this particular layer, the input zero point (input_zp) is 58, but the computed minimum value is 0.</p>
<p>In the next layer, on the other hand, the input zero point (input_zp) is 0, and the computed maximum value is 144.</p>
<p>Thanks</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="jackie_huang" data-post="1" data-topic="199796">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jackie_huang/48/61840_2.png" class="avatar"> jackie_huang:</div>
<blockquote>
<p>In the next layer, on the other hand, the input zero point (input_zp) is 0, and the computed maximum value is 144.</p>
</blockquote>
</aside>
<p>looks like you might have some confusion around how quantization is done, can you take a look at <a href="https://intellabs.github.io/distiller/algo_quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization - Neural Network Distiller</a> to see if it helps</p>
Loss Not Changing - No Convergence

username: learner13
cooked: <p>Hi all,</p>
<p>I’ve been working on a basic model for some time now - it’s a multi-classification problem with images (10). Details are below:<br>
Target = One Hot Encoding || 10 possible outcomes &gt; numbers 0-9<br>
Images are 28x28. They have been normalised with /255 to give a float32 between 0-1.<br>
Training Data is shape [60000, 28, 28] – gray scale images.<br>
Model code is below</p>
<pre><code class="lang-auto">from torch import nn, optim
import torch.nn.functional as torch_func

## Set the device ot GPU is available
device = 'cuda' if torch.cuda.is_available() else 'cpu'

model = nn.Sequential(
            nn.Linear(784, 1024),
            nn.ReLU(),
            nn.Linear(1024, 512),
            nn.Tanh(),
            nn.Linear(512, 10),
            nn.Softmax(dim=1)
)

model = model.to(device) ## move it to GPU if available
</code></pre>
<p>Implementation of the model below:</p>
<pre><code class="lang-auto">learning_rate = 1e-3
 
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
 
loss_fn = nn.MSELoss()
 
n_epochs = 10

for epoch in range(n_epochs):
    for label, img in train_data:
      outputs = model(img.view(-1).unsqueeze(0))
      loss = loss_fn(outputs.squeeze(0), label)
      optimizer.zero_grad()
      loss.requires_grad = True
      loss.backward()
      optimizer.step()
    print("Epoch: %d, Loss: %f" % (epoch, float(loss)))

</code></pre>
<p>Output is:</p>
<pre><code class="lang-auto">Epoch: 0, Loss: 0.091024
Epoch: 1, Loss: 0.091024
Epoch: 2, Loss: 0.091024
Epoch: 3, Loss: 0.091024
Epoch: 4, Loss: 0.091024
Epoch: 5, Loss: 0.091024
Epoch: 6, Loss: 0.091024
Epoch: 7, Loss: 0.091024
Epoch: 8, Loss: 0.091024
Epoch: 9, Loss: 0.091024
</code></pre>
<p>It could be a data prep problem but I am open to any and all suggestions.</p>
<p>Many thanks in advance.</p>
<p>JM</p>

username: ptrblck
cooked: <p>For a multi-class classification use case I would recommend using <code>nn.CrossEntropyLoss</code> instead of <code>nn.MSELoss</code>. You would also remove the <code>nn.Softmax</code> at the end of your model as raw logits are expected.</p>
How to handle FunctionalTensor in C++ backend

username: ashari4
cooked: <p>I am using a custom torch backend written in C++ for the ORT dispatch key.</p>
<p>I noticed that torch dynamo creates a <code>FunctionalTensor</code> with device set to ORT and calls the <code>source_Storage_storage_offset</code> op on this tensor. But my backend is unable to handle this. Do you know how the C++ backend is supposed to implement ops for FunctionalTensors?</p>
<p>python call stack:</p>
<pre><code class="lang-auto">alias_non_inplace_storage (torch\utils\_python_dispatch.py:400)
_correct_storage_aliasing (torch\utils\_python_dispatch.py:413)
return_and_correct_aliasing (torch\utils\_python_dispatch.py:553)
__torch_dispatch__ (torch\_subclasses\functional_tensor.py:460)
_engine_run_backward (torch\autograd\graph.py:744)
grad (torch\autograd\__init__.py:412)
inner_fn (torch\_functorch\_aot_autograd\traced_function_transforms.py:240)
inner_fn_with_anomaly (torch\_functorch\_aot_autograd\traced_function_transforms.py:255)
_functionalized_f_helper (torch\_functorch\_aot_autograd\traced_function_transforms.py:387)
joint_helper (torch\_functorch\_aot_autograd\traced_function_transforms.py:521)
wrapped (torch\fx\experimental\proxy_tensor.py:651)
...
</code></pre>
DP training for text-to-image diffusion models

username: Raman_Dutt
cooked: <p>T2I models are quite popular at the moment. Differentially private training of T2I models is very useful for a number of domains including healthcare where preserving patient privacy is of utmost importance. Most papers on diffusion models these days use diffusers library from huggingface for implementation. Is there any tutorial on resource that shows how to properly do DP training using opacus and diffusers?</p>
Multiple inputs in shared weight layers

username: pizza_party
cooked: <p>Hello, I have some time series in input, that are of shape (batch_size, num_channels, input_size).<br>
I want to process each channel indipendently in some linear layers that share the same weights for each channel. Then I need to take these outputs, concatenate them, put them through some other linear layers and compute the loss, optimize it etc…<br>
Here’s a simplified version of what I’m asking with only two inputs:</p>
<pre><code class="lang-auto">class network(nn.Module):
    def __init__(self):
        super(network, self).__init__()
        self.shared_linear = nn.Linear(4, 3)

        self.final_linear = nn.Linear(6,3)

    def forward(self, x):
        for i in range(x.shape[1]):
            out_i = self.shared_linear(x[:,i,:]).unsqueeze(1)
            if i == 0:
                shared_out = out_i
            else:
                shared_out = torch.cat((shared_out, out_i), 1)
        
        concatenated_out = shared_out.view(shared_out.shape[0], -1)
        return self.final_linear(concatenated_out)

model = network()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)
optimizer.zero_grad()

target = torch.tensor([0])
x = torch.randn(1, 2, 4)
out = model(x)

loss = criterion(out, target)
loss.backward()
optimizer.step()
</code></pre>
<p>Is this approach correct? Does backward automatically accumulate the gradients or shall I do something else?</p>

username: ptrblck
cooked: <p>Your approach looks alright and the backward pass should work as no detaching operation was used.<br>
You can also verify it by checking the <code>.grad</code> attributes of trainable parameters before and after the <code>backward</code> call.</p>
Parameter number of convolutional layer right after pooling layer very high compared to other layers

username: dangmanhtruong1995
cooked: <p>Hi everyone, I have this following model created by using nn.Sequential, however when I used torchsummary, I noticed that right after the pooling layer, the number of parameters is very high compared to other layers. I don’t know why it is that way. Here is the output:</p>
<pre><code class="lang-auto">Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (1): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1), padding=same, dilation=(7, 7))
  (2): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1), padding=same, dilation=(3, 3))
  (3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), padding=same, dilation=(5, 5))
  (4): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), padding=same, dilation=(3, 3))
  (5): MaxPool2d(kernel_size=3, stride=3, padding=0, dilation=1, ceil_mode=False)
  (6): Conv2d(64, 64, kernel_size=(7, 7), stride=(1, 1), padding=same, dilation=(7, 7))
  (7): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 256, 256]           1,792
            Conv2d-2         [-1, 16, 256, 256]           1,040
            Conv2d-3         [-1, 64, 256, 256]           1,088
            Conv2d-4         [-1, 64, 256, 256]           4,160
            Conv2d-5         [-1, 64, 256, 256]           4,160
         MaxPool2d-6           [-1, 64, 85, 85]               0
            Conv2d-7           [-1, 64, 85, 85]         200,768
       BatchNorm2d-8           [-1, 64, 85, 85]             128
================================================================
Total params: 213,136
Trainable params: 213,136
Non-trainable params: 0
</code></pre>

username: ptrblck
cooked: <p>This is expected as the next conv layer has <code>64*64*7*7=200704</code> (+64 bias) trainable weights.</p>

username: dangmanhtruong1995
cooked: <p>Oh i understand now, it’s because of</p>
<blockquote>
<p>kernel_size=(7, 7)</p>
</blockquote>
<p>and</p>
<blockquote>
<p>dilation=(7, 7)</p>
</blockquote>

username: ptrblck
cooked: <p>The <code>dilation</code> does not change the parameter count.<br>
It’s thus because of the kernel size and the in/out channels of the kernel.</p>
Customized loss function while using opacus in personalized federated learning

username: raegan_yang
cooked: <p>Hi. I implement a mean-regularized multi-task learning framework using a loss function like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de56bfb32e2753d4bff17d27b7f8ffbae4e3b2e9.png" data-download-href="https://discuss.pytorch.org/uploads/default/de56bfb32e2753d4bff17d27b7f8ffbae4e3b2e9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de56bfb32e2753d4bff17d27b7f8ffbae4e3b2e9_2_690x287.png" alt="image" data-base62-sha1="vITR86fZAeChvitxNtLaDs8aHQR" width="690" height="287" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de56bfb32e2753d4bff17d27b7f8ffbae4e3b2e9_2_690x287.png, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de56bfb32e2753d4bff17d27b7f8ffbae4e3b2e9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de56bfb32e2753d4bff17d27b7f8ffbae4e3b2e9.png 2x" data-dominant-color="F4F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">861×359 37.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
While adding no noise, it works well: Average accuracy of clients increases and then decreases as lambda(the personalization degree) starts to increase from zero.<br>
When I make clients do DP-SGD using opacus, this rise-then-fall pattern doesn’t exist. The average accuracy always stays the same as if all clients train their models completely locally, which means the prox term in loss function somehow disappears. Anyone knows if the loss function is really the problem here?</p>
<p>BTW, when using opacus in FedAvg setting without personalization, the code works well. I will really appreciate any thougnts.</p>

username: raegan_yang
cooked: <p>I am guessing I need to tune the max_norm of DP-SGD…</p>

username: raegan_yang
cooked: <p>Now I know max_norm is kind of irrelevant. So I guess the prox_term is never added to any weight.grad_sample?</p>

username: raegan_yang
cooked: <p>Oh I see this is a bug from github…</p>

username: raegan_yang
cooked: <p>Will simply setting <code>grad_sample_mode="functorch"</code> in the call to <code>make_private()</code> solve this problem?</p>

username: raegan_yang
cooked: <p>no. and ‘‘ew’’ does not work too</p>

username: raegan_yang
cooked: <p>Hi, I manually add the prox term after <code>loss.backward()</code> and before <code>optim.step()</code> like this:</p>
<pre><code class="lang-auto">    for p, diff in zip(self.model.parameters(), n_w_diff):
        p.grad_sample.data = p.grad_sample.data + diff.data
</code></pre>
<p>Is there something I didn’t pay attention to? I got some weird results.</p>
Can we optimize the frequency by comparing the difference between 2D sine wave images?

username: icewired-yy
cooked: <p>I set a random target frequency <em>f1</em>, and another randomly initial frequency <em>f2</em>. Since the <em>f1</em> is unknown now, I want to use the gradient-based optimization to recover the <em>f2</em> by comparing the difference between the 2D sine wave image <code>m1</code> generated by <em>f1</em> and <code>m2</code> generated by <em>f2</em> in the same way. The 2D sine wave image looks like the following:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af826be146c0078c01b3d63ecaf2304713514e68.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/af826be146c0078c01b3d63ecaf2304713514e68" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af826be146c0078c01b3d63ecaf2304713514e68_2_690x369.jpeg" alt="image" data-base62-sha1="p2CZ07ydmPk0QcDvUYUuP7JQ6WQ" width="690" height="369" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af826be146c0078c01b3d63ecaf2304713514e68_2_690x369.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/f/af826be146c0078c01b3d63ecaf2304713514e68.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/f/af826be146c0078c01b3d63ecaf2304713514e68.jpeg 2x" data-dominant-color="848484"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">764×409 45.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I think this is an interesting experiment, but finally, I failed to recover the correct frequency. Loss functions such as MSE, VGG, and even FFT cannot lead the frequency to the correct one.</p>
<p>Is it possible to recover the frequency in this way?</p>
Does DataLoader work normally on Intel's 13th gen CPU?

username: TimandXiyu
cooked: <p>As said, I am considering upgrading my setup but I do wonder if the multi-processing of dataloader can fully utilize all the P-cores and E-cores on Intel’s new 13th gen.</p>
<p>Based on my reading of the code of DataLoader, it seems like the speed difference between P-core and E-core won’t trigger performance issues. Still, I am asking this here to make sure of it.</p>

username: nivek
cooked: <p>I don’t think DataLoader has been tested on those hardware yet.</p>
<p>Based on my little understanding on the new CPUs, I would think there shouldn’t be significant differences but it is hard to tell. It depends on the task at hand (whether the data loading processes are compute-bound or IO-bound), and how well the CPU schedules/balances work between the P-core and E-core.</p>
<p>Please let us know if you have compared different generations of Intel CPUs and found noticeable performance differences.</p>

username: iago-lito
cooked: <p>Hi <a class="mention" href="/u/nivek">@nivek</a>, I happen to have a personal i7 13th gen CPU at home and one i7 12th gen CPU at work, and I do observe significant (2x) slowdown of the same process on the 13th gen compared to the 12th gen.</p>
<p>I tried hard to explain this from code version differences this morning, but both machines are up-to-date arch installs with the same torch version <code>2.2.1</code>, so I ended up suspecting hardware. I’m not exactly sure how I can help you know further about the problem, but I’m happy to help ^ ^"</p>
<p>I used <a href="https://pytorch.org/tutorials/recipes/recipes/benchmark.html" rel="noopener nofollow ugc">this example</a> to benchmark:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

torch.manual_seed(12)
torch.set_num_threads(1)

def batched_dot_mul_sum(a, b):
    """Computes batched dot by multiplying and summing"""
    return a.mul(b).sum(-1)


def batched_dot_bmm(a, b):
    """Computes batched dot by reducing to ``bmm``"""
    a = a.reshape(-1, 1, a.shape[-1])
    b = b.reshape(-1, b.shape[-1], 1)
    return torch.bmm(a, b).flatten(-3)


# Input for benchmarking.
x = torch.randn(1_000_000, 640)

# Ensure that both functions compute the same output.
assert batched_dot_mul_sum(x, x).allclose(batched_dot_bmm(x, x)) 
</code></pre>
<p>On the i7 12th machine:</p>
<pre><code class="lang-auto">$ time python benchmark.py
real 4,82s
user 4,18s
sys 0,63s
</code></pre>
<p>On the i7 13th machine:</p>
<pre><code class="lang-auto">$ time python benchmark.py
real 10,57s
user 9,39s
sys 1,16s
</code></pre>
<p>What could be wrong?</p>
Question about requires_grad usage

username: cmitch29
cooked: <p>In the Llama recipes repository, <a href="https://github.com/meta-llama/llama-recipes/blob/37c8f722116493e69ea99420b3d73287905a46d0/src/llama_recipes/utils/train_utils.py#L335" rel="noopener nofollow ugc">there is a function called <code>freeze_transformer_layers</code></a>. This function <a href="https://github.com/meta-llama/llama-recipes/blob/37c8f722116493e69ea99420b3d73287905a46d0/src/llama_recipes/finetuning.py#L170" rel="noopener nofollow ugc">can be triggered when sharding the model via FSDP</a>.</p>
<p>However, the <a href="https://pytorch.org/docs/main/fsdp.html" rel="noopener nofollow ugc">nightly (or main) FSDP docs</a>, at time of writing, states:</p>
<blockquote>
<p>FSDP has some constraints on freezing parameters (i.e. setting <code>param.requires_grad=False</code>). For <code>use_orig_params=False</code>, each FSDP instance must manage parameters that are all frozen or all non-frozen. For <code>use_orig_params=True</code>, FSDP supports mixing frozen and non-frozen, but we recommend not doing so since then the gradient memory usage will be higher than expected (namely, equivalent to not freezing those parameters). This means that ideally, frozen parameters should be isolated into their own <code>nn.Module</code> s and wrapped separately with FSDP.</p>
</blockquote>
<p>Why are we able to freeze / unfreeze certain parameters in this way? Should we be wrapping the decoder blocks in FSDP separately?</p>
How do I sovle the detection problem when I am using static_quantization

username: RonaldYuren
cooked: <p>I can’t deal with my Post-traing-static-Quantization model.<br>
It came across the detection problem, I can’t get the right detection outcome after applying Pytorch FX-mode static quantization.<br>
It seems that the accuracy is 0 after I using the Quantized YOLOv7 to do inference job. (Didn’t find the object location? )<br>
What may the error come from ?  Or how do  I to test where the probably error come from?</p>
<p>I am using Pytorch 2.1.0 ，<br>
Using YOLOv7 &amp; YOLOv7-tiny model to doing my cumtom dataset training and inference job.</p>