akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Image Patching question

Ali_Youssef: <p>Hello everyone!</p>
<p>I had a a simple question regarding image patching.</p>
<p>For exmaple, if we are doign segementation or any other image processing task.</p>
<ul>
<li>
<p>If an image is (256,256) and during training nd testing, we split it to patches of size (32,32), why not just resize the image to (32,32) directly? Do we lose information if we do so?</p>
</li>
<li>
<p>I don’t really understand why apply patches in the first place, does it improve segementation performance or is it purely for computational power expenses?</p>
</li>
</ul>
<p>Thank you!</p>

ptrblck: <ol>
<li>Yes, resizing the image from <code>256x256=65536</code> pixels to <code>32x32=1024</code> will lose information.</li>
<li>Also yes, using the larger input images might cause OOM issues depending on the model and the used GPU. Segmentation results might thus benefit from a patch approach.</li>
</ol>
Unable to use torcheval `module_summary` with packed sequences for LSTM

Adamits: <p>I am getting an error when I try to compute model FLOPs with the torcheval <a href="https://github.com/pytorch/torcheval/blob/main/torcheval/tools/module_summary.py" rel="noopener nofollow ugc">module_summary</a>.</p>
<p>Specifically, the issue seems to be that <code>module_summary</code> calls the forward function on every submodule of my model (see <a href="https://github.com/pytorch/torcheval/blob/main/torcheval/tools/module_summary.py#L258" rel="noopener nofollow ugc">here</a>), unpacking the provided args and passing them through.</p>
<p>Since one of my submodules is an <code>nn.LSTM</code> that I pass a <code>PackedSequence</code> to, and a pytorch <code>PackedSequence</code> inherits from a named tuple, this gets unpacked (sorry this term is overloaded, I mean unpacked as an iterable into args to the LSTM forward function), and errors.</p>
<p>example:</p>
<pre><code class="lang-auto">        embedded = self.embeddings(source)
        # Packs embedded source symbols into a PackedSequence.
        packed = nn.utils.rnn.pack_padded_sequence(
            embedded, lengths, batch_first=True, enforce_sorted=False
        )
        # -&gt; B x seq_len x encoder_dim, (h0, c0).
        packed_outs, (H, C) = self.encoder(packed)
        encoded, _ = nn.utils.rnn.pad_packed_sequence(
            packed_outs,
            batch_first=True,
            padding_value=self.pad_idx,
            total_length=None,
        )
</code></pre>
<p>Causes <code>TypeError: forward() takes from 2 to 3 positional arguments but 5 were given</code> while profiling FLOPs.</p>
<p>Of course, I can simply pass <code>packed</code> inside an iterable, which solves the issue when profiling:</p>
<pre><code class="lang-auto">packed_outs, (H, C) = self.encoder((packed,))
</code></pre>
<p>but then the actual forward pass errors when I am not profiling with torch eval, since it expects a <code>PackedSequence</code>: <code>AttributeError: 'tuple' object has no attribute 'dim'</code></p>
<p>This seems like a really silly issue that someone else might have run into. I am probably being dense but is there a simple solution to this?</p>

ptrblck: <p>I would recommend posting this issue in the <code>torcheval</code> GitHub repository to track and fix it, as it’s still in its alpha stage.</p>
Merging 3 regression model at certain ratio

Abinas: <p>Hi! everyone<br>
I am currently working on a problem where i have 3 trained regression model, where each model gives output of  (1,300) length sequence. things i need to do is combine the output from the regression model at certain ratio a<em>x + b</em>y + c*z = final output (1,300). I want to train the ratio a,b,c. I don’t want the gradient to back propagate to the trained model only the ratio must be updated. suggest me how can i achieve this.</p>

ptrblck: <p>I’d you want to keep the models frozen and only train the scaling factors, initialize them as <code>nn.Parameters</code>, pass them to an optimizer, and update them in each iteration.</p>

Abinas: <p>did you mean like this ?</p>
<pre><code class="lang-auto">class combined_model(nn.Module):
    def __init__(self):
        super(combined_model,self).__init__()
        self.a = nn.Parameter(torch.tensor(1.0))
        self.b = nn.Parameter(torch.tensor(1.0))
        self.c = nn.Parameter(torch.tensor(1.0))

    def forward(self,x1,x2,x3):
        return self.a*x1+self.b*x2+self.c*x3
</code></pre>

ptrblck: <p>Yes, your approach looks valid.</p>
AutoEncoder transfer learning: how to freeze encoder / decoder

bornexmachina: <p>Hi, I am working on anomaly detection via Auto Encoders.<br>
So my network looks like this:</p>
<pre><code class="lang-auto">class net(nn.Module):
    def __init__()
        ...
    # from 32x128 -&gt; 4x4 -&gt; Flatten()
    self.encoder = nn.Sequential(...)
    # use a dense layer, which will be reshaped in the forward function
    self.embedding = nn.Linear(4*4, out_dim)
    # go from 4x4 -&gt; 32x128
    self.decoder = nn.Sequential(...)
    # encode, embed, reshape, decode
    def forward(...) ...
</code></pre>
<p>Now I would like to train my representation of the encoder and decoder and transfer it on a new data set.<br>
The only trainable layer should be the embedding.</p>
<p>I am a bit unsure how to do it</p>
<ol>
<li>What does it mean for my code in the abstract sense?
<ul>
<li>Should I have something like <code>pretrain_model</code> which fully trains and then something like <code>actual_model = pretrain_model</code> with some layers frozen?</li>
</ul>
</li>
<li>How do I actually freeze the encoder and decoder and let only the embedding learn</li>
<li>Is it actually a good idea to let only the embedding learn?</li>
</ol>

ptrblck: <ol>
<li>I don’t fully understand the question and am unsure why different variables are used.</li>
<li>You can iterate the <code>.parameters()</code> of submodules and set their <code>.requires_grad</code> attribute to <code>False</code> in order to freeze them.</li>
<li>I also don’t know as it would depend on your use case.</li>
</ol>
RuntimeError: Function 'ToCopyBackward0' returned nan values in its 0th output

S_M: <p>Hi all, I am working on ResNet101 with cross entropy loss loss.backward() give me this error when I switch on fp16 precision. I can not used fp32 because my GPU and memory resources are limit.<br>
I am confused about this error. I tried some optimization settings but they did not work.<br>
I would be really appreciated to any advice.<br>
Thanks</p>

ptrblck: <p>Note that invalid gradients are expected at the beginning of the training in <code>amp</code> using <code>float16</code> as well as sometimes during the training. In these cases the <code>GradScaler</code> will skip the parameter updates and reduce the loss scaling factor. If you are using anomaly detection from the beginning, you might need to disable it.</p>
How to perform K-Fold Implementation in Transfer Learning tutorial

SrishtiRaj: <p><a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html#model-training-and-validation-code" class="inline-onebox" rel="noopener nofollow ugc">Finetuning Torchvision Models — PyTorch Tutorials 1.2.0 documentation</a>.</p>
<p>I am using this tutorial to implement transfer learning on my private dataset. However, I want to implement K-Fold in this tutorial. I am new to all of these and unable to comprehend how to apply K-fold validation.</p>
<p>I would be grateful  if anyone could guide me with the changes I need to make to this tutorial</p>
How can I replace A model's params and get the grad correctly?

ziyue_xu: <p>I have a resnet model and I want to replace a Conv tensor parameter of it with my own tensor A and B.<br>
Here is the code:</p>
<pre><code class="lang-auto">model = resnet.resnet18()

# my_param = torch.nn.Parameter(torch.zeros(size=(64, 64, 3, 3)))

A = torch.nn.Parameter(torch.zeros(size=(64 * 64, 32 * 32))) # A is my desired leaf Parameter
B = torch.nn.Parameter(torch.zeros(size=(32, 32, 3, 3))) # B is my desired leaf Parameter

B.requires_grad_(True)
B.retain_grad()

B = B.view(-1, 9)

base = A @ B # matrix mm to create 'base'

base.requires_grad_(True)
base.retain_grad()

base = base.view(64, 64, 3, 3)

A.requires_grad_(True)
A.retain_grad()
B.requires_grad_(True)
B.retain_grad()

model.layer1[0].conv1.weight = torch.nn.Parameter(base) # replace the params with 'base'

# my random img
img = {}
img["img"] = torch.randn(size=(1, 3, 28, 28))
img["label"] = torch.ones(1).long()

out = model(img)
print(out)

out["loss"].backward() # I already got a loss func, the loss will be auto calculed 

print(A.grad, B.grad) # print NONE
</code></pre>
<p>But, the result of A.grad and B.grad is NONE…</p>
<p>Oh my god, How can I eventually get their grads correctly?</p>
<p>I have try many ways:</p>
<ol>
<li>
<p>Use model.load_state_dict(), still NONE</p>
</li>
<li>
<p>Use model.layer1[0].conv1.weight = base, but pytorch raise TypeError: cannot assign ‘torch.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected)</p>
</li>
<li>
<p>Use model.layer1[0].conv1.weight.data = nn.Parameter(base), still NONE</p>
</li>
</ol>

ptrblck: <p>That’s expected since you are creating a new <code>nn.Parameter</code> without a gradient history.<br>
You could check <a href="https://discuss.pytorch.org/t/averaging-models-with-differentiable-weighting-coefficients/177240/3">this post</a> to see if <code>parametrize</code> would work.</p>
Alexnet pretraining Model

MaryamRasool: <p>I am using a pre-trained model called Alexnet for my data. I got good accuracy on training and validation, but when I trained again on the same parameter, I did not get good accuracy. Please give me some suggestions.  How do I remove my model’s overfitting now?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/3/83d3fdbdf35ad70372ba7d6e9a9c290a90d9bc82.png" alt="pytorch1" data-base62-sha1="iOcJJtrHXyxrAyURfBjsukTtOxQ" width="618" height="335"></p>
Problems installing Pytorch 2.0.1

Tim3: <p>Hello,</p>
<p>I am trying to install Pytorch 2.0.1 using pip via:</p>
<pre><code class="lang-auto">pip install torch
</code></pre>
<p>I am using Python 3.9.6 (64-bit version) on Windows 10.<br>
The command should install the newest Pytorch version 2.0.1.</p>
<p>Nethertheless I am getting the following error message:</p>
<blockquote>
<p>No matching distribution found for torch</p>
</blockquote>
<p>Can anyone help?</p>

suraj.pt: <p>I see a matching distribution for what I think is your setup? <a href="https://files.pythonhosted.org/packages/48/f4/d0b61525a3d3db78636f1937d1bc24cbb39abc57484a545b72b6ab35c114/torch-2.0.1-cp39-cp39-win_amd64.whl">https://files.pythonhosted.org/packages/48/f4/d0b61525a3d3db78636f1937d1bc24cbb39abc57484a545b72b6ab35c114/torch-2.0.1-cp39-cp39-win_amd64.whl</a></p>
<p>Maybe try upgrading pip or use <code>python3 -m pip install torch</code> ?</p>

ptrblck: <p><code>pip install torch</code> is not a valid install command for Windows and <code>torch==2.0.1</code>.<br>
From the <a href="https://pytorch.org/get-started/locally/">install matrix</a> I see:</p>
<ul>
<li>
<code>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117</code> for CUDA 11.7</li>
<li>and <code>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</code> for CUDA 11.8</li>
</ul>
Tensors of variable size

kazem: <p>Assume I have a set of objects of variable size, for example cells of different size in a image, and that I don’t want to resize or pad them.<br>
One needs to define a custom collate_fn to be able create a dataset/dataloader that returns a list of tensors of variable size. Now, here is my question:<br>
Does torch.compile now allow users to pass/serve a list of tensors of variable size to a pytorch model, instead of the old solution:<br>
In the transform step pad each the same size,  and then concatenate them along the first dimension new dimension in the collate function, and feeding  the resulting tensor to the model where the first dimension is the batch size?</p>

marksaroufim: <p>You probably are interested in <a href="https://pytorch.org/docs/stable/nested.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nested — PyTorch 2.0 documentation</a></p>
DataDistributedParallel: How to re-initialize select layers

adamtupper: <p>I’m trying to figure out how I can periodically re-initialize certain layers of my network (for example the last <em>m</em> layers of a CNN, such as a ResNet) when using DataDistributedParallel.</p>
<p>In the non-distributed setting, I think I could achieve this by iterating over the modules. For example:</p>
<pre><code class="lang-auto">for i, module in iter(algorithm.model.modules()):
    if i &gt;= m:
        if isinstance(module, nn.Conv2d):
            nn.init.kaiming_normal_(
                module.weight, mode="fan_out", nonlinearity="leaky_relu"
            )
        elif isinstance(module, nn.Linear):
            nn.init.xavier_normal_(module.weight.data)
            module.bias.data.zero_()
</code></pre>
<p>However, when I try to integrate this into my code, which uses <code>DataDistributedParallel</code>, iterating over the modules raises the error:</p>
<pre><code class="lang-auto">TypeError: cannot unpack non-iterable DistributedDataParallel object
</code></pre>
<p>Looking more closely at the iterable returned by <code>model.modules()</code> when the model is wrapped by <code>DataDistributedParallel</code> I can see that it’s added more complexity.</p>
<p>Has anyone tried what I’m setting out to achieve? If so, have you got any tips on how to best achieve this?</p>
<p>Thanks in advance!</p>

fduwjj: <p>I am not sure if DDP supports that. cc: <a class="mention" href="/u/rvarm1">@rvarm1</a></p>
I have a question about the allocation of a fix batch size for different number of GPUs

zjhzshenhaozhan: <p>I set batch size as 80, and use 4 GPUs to train the model, its loss of the first several steps is as the red line in image<br>
However, when I use 5 GPUs to train, its loss turns into the green line in image<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7ca1300607cf60b640feebd6d7172d54f996e33.png" data-download-href="https://discuss.pytorch.org/uploads/default/d7ca1300607cf60b640feebd6d7172d54f996e33" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7ca1300607cf60b640feebd6d7172d54f996e33_2_537x500.png" alt="image" data-base62-sha1="uMXB5o35MpZb55UZmA0qbnku1VN" width="537" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7ca1300607cf60b640feebd6d7172d54f996e33_2_537x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7ca1300607cf60b640feebd6d7172d54f996e33_2_805x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d7ca1300607cf60b640feebd6d7172d54f996e33_2_1074x1000.png 2x" data-dominant-color="F4F3F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1274×1186 185 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My GPU is 40G-A100, and torch version is 1.10.0, and my loss function doesn’t depend on  other samples<br>
I have trained the model more epoches in different GPU numbers, which got a larger gap in Loss value and performance (5 GPUs is better than 4).<br>
I sincerely wonder an answer and explaintion for this phenomenon. Thanks in advance.</p>

zjhzshenhaozhan: <p>My batch size is fixed at 80 from the beginning to the end.</p>

zjhzshenhaozhan: <p>In theory, why would the same batch size produce different results?</p>

fduwjj: <p>Do you have a code for us to take a look? What do you use for model training? Did you use DistributedDataParallel, FSDP or? And what about data loader you were using?</p>
Pytorch on amd/rocm

rtadd: <p>Hello.<br>
First of all I’d like to clarify that I’m really new in all of this, not only pytorch and ML but even python.<br>
I’m learning to use this library and I’ve managed to make it work with my rx 6700 xt by installing both the amdgpu driver (with rocm) and the “pip install…” command as shown on the PyTorch website.</p>
<p>The thing is that my gpu isn’t supported according to amd’s documentation, so one extra step I had to take was to add an environment variable (os.environ[“HSA_OVERRIDE_GFX_VERSION”]= “10.3.0”) otherwise all my hardware was correcly detected, but anytime I tried to do anything with a tensor loaded on the gpu memory I would get this error: Segmentation fault (core dump)</p>
<p>Anyway, I managed to make it work. So far I’ve only tried to do simple linear regressions with the following code:</p>
<pre><code class="lang-auto">import torch as trc
from torch.utils.data import Dataset
import os
import time as tm
os.environ["HSA_OVERRIDE_GFX_VERSION"]= "10.3.0"

gpu = trc.device("cuda" if trc.cuda.is_available() else "cpu")

class Dataset_1(Dataset):
    def __init__(self, transf=None, dev=None):
        self.X = trc.arange(-3.0, 3.0, 0.1, device=dev).view(-1,1)
        self.Y = (-3 * self.X) + (0.1 * trc.randn(self.X.size(), device=dev)) - 2
        self.transf = transf
    
    def __getitem__(self, indice):
        dat = [self.X[indice], self.Y[indice]]
        if self.transf:
            dat = self.transf(dat)
        return dat

class reg_lin(trc.nn.Module):
    def __init__(self, num_ent, num_sal):
        super(reg_lin, self).__init__()
        self.lineal = trc.nn.Linear(num_ent, num_sal)

    def forward(self,x):
        val = self.lineal(x)
        return val

datos1 = Dataset_1(dev=gpu)

modelo1 = reg_lin(1, 1)
modelo1 = modelo1.to(gpu)

def criterio(yhat, y):
    return trc.mean( (yhat - y) ** 2 )

optim = trc.optim.SGD(modelo1.parameters(), 0.01)

def entrenar_modelo(iter):
    for epoch in range(iter):
        Yhat = modelo1(datos1[:][0])
        loss = criterio(Yhat, datos1[:][1])
        loss.backward()
        optim.step()
        optim.zero_grad()

t0 = tm.time()

entrenar_modelo(5)

t1 = tm.time()

print(modelo1.state_dict())

print(t1-t0)
</code></pre>
<p>What I found out is that running this code on the gpu is several times slower than running it on the cpu (by removing “dev=gpu” on datos1 and commenting out the “modelo1.to(gpu)” line).</p>
<p>Is that supposed to happen? Is there anything wrong with the code?</p>
<p>-EDIT-<br>
A bit more of information since I’m learning new stuff. Using the DataLoader reduces significantly the difference between the times of the GPU and CPU. Before de GPU took more or less twice the time, with the DataLoader is around 50% more time.</p>
Does an optimizer deallocate the gradients after it updates the weights of model?

raining_day513: <p>As title. During the backward pass, those tensors that <code>require_gradient</code> will be set so that their gradient can be used by the optimizer later on. Now I’m interested in whether or not an optimizer will deallocate the GPU memory of these gradients.</p>

ptrblck: <p>Yes, <code>optimizer.zero_grad()</code> will use <code>set_to_none=True</code> by default in newer PyTorch versions and will thus delete the <code>.grad</code> tensors to be able to reuse this memory.</p>

raining_day513: <p>In my current code I wrote:</p>
<pre><code class="lang-python">optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>would it be reasonable to change it to:</p>
<pre><code class="lang-python">loss.backward()
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>if I want to clear the GPU memory usage of gradient after each training loop. It seems that the first version might cause the <code>memory_allocated</code> called in the forward pass to also record those <code>grad</code>s. Am I correct?</p>

ptrblck: <p>Yes, if you delete the gradients late, i.e after the next forward pass was done, your memory footprint would be higher.</p>
How to change the params in one conv1d layer

Alva-2020: <pre><code class="lang-auto">
import torch
import torch.nn as nn
import pdb
import torch.nn.init as init
pdb.set_trace()
# Third convolutional layer
conv3 = nn.Conv1d(64, 128, kernel_size=3)

ab = conv3.weight
print("ab",ab[:,1:3,:])

new_params = torch.ones(128, 2, 3)  # Example: generate new parameters

ab.data[:, 1:3, :] = new_params
cd = conv3.weight
print("cd",cd[:,1:3,:])
print("ab",ab[:,1:3,:])
print(sum(cd - ab))

</code></pre>
<p>For the above code, I think ab is the previous params of conv3, and cd is the new replaced weights and they are different. But why they are the same value. Thanks for your reply.</p>

ptrblck: <p>Don’t use the deprecated <code>.data</code> attribute, but copy the new values directly to the <code>weight</code> in a <code>no_grad</code> context:</p>
<pre><code class="lang-python">with torch.no_grad():
    conv3.weight[:, 1:3].copy_(new_params)
</code></pre>
Unstructured global pruning with importance scores is not working

Goldname: <p>I am trying to prune a model using importance scores. As a test here is what I did:</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms
from torch.utils.data import Dataset, DataLoader
import torch.nn.utils.prune as prune

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
model = torchvision.models.vit_b_32(weights=torchvision.models.ViT_B_32_Weights.IMAGENET1K_V1)
model.to(device)
model.eval()

random_activation = {}

for k, v in dict(model.named_modules()).items():
    if ((len(list(v.children())) == 0) and (v._get_name().lower().startswith('linear'))):
        random_activation[k] = torch.rand(v.weight.size())

parameter_to_prune = [
    (v, "weight") 
    for k, v in dict(model.named_modules()).items()
    if ((len(list(v.children())) == 0) and (v._get_name().lower().startswith('linear')))
]

prune.global_unstructured(parameter_to_prune, prune.L1Unstructured, importance_scores = random_activation, amount = 0.3)
</code></pre>
<p>However, I am getting the same accuracy as if I use importance_scores = None. Am I using this incorrectly?</p>
Increasing size of numeric dataset

yari: <p>I am trying to increase size a numeric dataset that I have been working with. The dataset comprises the columns: [‘Name’, ‘O3_1hr’, ‘O3_4hr’, ‘O3_8hr’, ‘PM10 1 hr’, ‘PM10’, ‘AQI_O3_1hr’, ‘AQI_O3_4hr’, ‘AQI_PM10’, ‘AQI_Site’, ‘Date’, ‘Time’, ‘Latitude’, ‘Longitude’].</p>
<p>Previously, I attempted to use a GAN (Generative Adversarial Network) for data augmentation, which yielded an accuracy of 87%. This accuracy is not acceptable.<br>
I would greatly appreciate any guidance or recommendations in this regard. Specifically, I am interested in alternative methods of data augmentation or any other techniques that can effectively expand a numeric dataset while preserving the integrity and patterns that already exist in the data.</p>
Backward temperature Softmax implementation

DeepLearner17: <p>Hello,</p>
<p>Is there any forward/backward implementation  of Temperature softmax function ?</p>
<h1>softmax = e^(z/T) / sum_i e^(z_i/T)</h1>
<p>Thank you</p>

albanD: <p>Hi,</p>
<p>I don’t think there is a special one but you can use the regular softmax:</p>
<pre><code class="lang-auto">t = 0.1
inp = torch.rand(2, 10)

out = torch.softmax(inp/t, dim=1)
</code></pre>

wk910930: <p>Hi, <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>I am not very sure, but do we need to do <code>inp = inp / t</code> (aka. assign back) first in this case for the later gradient calculation?  Or which way is better?</p>
<p>Thanks,<br>
Nick</p>

albanD: <p>Hi,</p>
<p>The autograd will differentiate the whole code so you can arrange it whichever way you want. The gradient will be computed the same way <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

fjanoos: <p>This does not work as expected for T=0. The implementation produces nans when it should be producing an indicator vector.</p>

albanD: <p>Hey!<br>
Why would you have t=0?<br>
That would make the gradient 0 everywhere so you cannot train anything.</p>

fjanoos: <p>I just want to run forward mode with a one hot indicator on the most probable component of the vector. No back prop. Thanks</p>

albanD: <p>In that case I would just get the index with a max op. Or do <code>t == t.max()</code> if you need a one-hot encoding.</p>
Weights dont change during optimizer.step() for custom gradient calculation

hl465: <p>Hi,</p>
<p>I have problems with my weight updates when calling <code> optimizer.step()</code> after manually calculating and assigning the gradients. I checked that <code>.grad</code> for all parameters is not <code>None</code> or zero, but when I call <code>optimizer.state_dict()</code> I get the unusual output</p>
<pre><code class="lang-auto">{'state': {0: {'momentum_buffer': None}, 1: {'momentum_buffer': None}, 2: {'momentum_buffer': None}, 3: {'momentum_buffer': None}, 4: {'momentum_buffer': None}, 5: {'momentum_buffer': None}, 6: {'momentum_buffer': None}, 7: {'momentum_buffer': None}, 8: {'momentum_buffer': None}}, 'param_groups': [{'lr': 0.005, 'momentum': 0, 'dampening': 0, 'weight_decay': 0, 'nesterov': False, 'maximize': False, 'foreach': None, 'differentiable': False, 'params': [0, 1, 2, 3, 4, 5, 6, 7, 8]}]}
</code></pre>
<p>I share the code that I am using below:</p>
<pre><code class="lang-auto">from functorch import jacrev, vmap, make_functional, grad

def compute_centered_jacobian(model,samples):
    func, parameters = make_functional(model)
    def func_(*params):
        return func(params, samples)
    jac = torch.autograd.functional.jacobian(func_, parameters)
    jac_ampl = jac[0]
    jac_phase = jac[1]
    jac_ampl = torch.cat([it.reshape(it.size(0),-1) for it in list(jac_ampl)], axis=-1)
    jac_phase = torch.cat([it.reshape(it.size(0),-1) for it in list(jac_phase)], axis=-1)
    return jac_ampl, jac_phase


def _compute_gradient_with_curvature(Tinv, E, O):
    n_samples = Tinv.size(0)
    TinvE = torch.mv(Tinv, E)/n_samples
    δ = torch.einsum("ij,j", O.t(),TinvE)/n_samples
    return δ


def compute_gradient_with_curvature(Ore, Oim, E, model,**kwargs):
    T = torch.einsum("ij,jk", Ore, Ore.t())+torch.einsum("ij,jk",Oim,Oim.t())  
    Tinv = torch.linalg.pinv(T,rtol=1e-12)
    δ = _compute_gradient_with_curvature(Tinv, E.real, Ore)+_compute_gradient_with_curvature(Tinv, E.imag, Oim)
    return δ

def apply_grads(model,grad):
    i = 0
    for p in filter(lambda x: x.requires_grad, model.parameters()):
        n = p.numel()
        if p.grad is not None:
            p.grad.copy_(grad[i : i + n].view(p.size()))
        else:
            print("gradient = None. Please check whats going wrong!")
            p.grad = grad[i : i + n].view(p.size())
        i += 1

def run_sr(model, E, samples, optimizer, scheduler=None):
    Ore, Oim = compute_centered_jacobian(model, samples)
    grads = compute_gradient_with_curvature(Oim, Ore, E, model)
    print(grads)
    apply_grads(model,grads)
    optimizer.step()
</code></pre>
<p>I would be happy about any help!</p>
Dual Tensor contribution

panos: <p>I created a dual tensor implementation for pytorch for a research project. It is an alternative to autograd in some cases where the gradients are highly non linear.  The work is quite extensive and I would like to give it back to the community.  I believe creating a new type using dual numbers has a number of interesting applications and the current implementation is very bare.</p>
<p>How do I go about contributing it?</p>
Replicating switching allocator behavior natively

Velveret: <p>I’ve been playing around with LLMs on my laptop, I have rather limited VRAM so I had to get creative with memory allocation.<br>
Initially, I tried UVM, which gets the model running at about 8s per token.<br>
After some experimentation, I settled on a switching allocator that puts the model weights in page locked host memory and everything else in VRAM, which runs at about 3s per token, a substantial improvement!</p>
<pre><code class="lang-auto">#include &lt;cuda_runtime.h&gt;
#include &lt;iostream&gt;
#include &lt;unordered_set&gt;

bool host_alloc = true;

std::unordered_set&lt;void*&gt; host_ptrs;

extern "C" {
void* sw_alloc(ssize_t size, int device, cudaStream_t stream) {
    void *ptr;
    auto ec = host_alloc ? cudaMallocHost(&amp;ptr, size) : cudaMalloc(&amp;ptr, size);

    if (ec != cudaSuccess) {
        std::cout &lt;&lt; "sw_alloc failed: " &lt;&lt; cudaGetErrorString(ec) &lt;&lt; std::endl;
    }

    if (host_alloc) {
        host_ptrs.insert(ptr);
    }

    return ptr;
}

void sw_free(void* ptr, ssize_t size, int device, cudaStream_t stream) {
    cudaError_t ec;
    if (host_ptrs.contains(ptr)) {
        ec = cudaFreeHost(ptr);
        host_ptrs.erase(ptr);
    } else {
        ec = cudaFree(ptr);
    }

    if (ec != cudaSuccess) {
        std::cout &lt;&lt; "sw_free failed: " &lt;&lt; cudaGetErrorString(ec) &lt;&lt; std::endl;
    }
}

void switch_alloc() {
    std::cout &lt;&lt; "switching allocator" &lt;&lt; std::endl;
    host_alloc = !host_alloc;
}
}
</code></pre>
<p>Thoughts on some API that allows the user to choose what kind of CUDA memory they want to use for a particular tensor/module?</p>
Does Transforms.Compose apply all augmentation or random

SrishtiRaj: <p>This is the code I am using, I understand that if I hv 100 images, 100 images will be the output with transforms. So, all transforms will be applied to the images or it is done random?</p>
<p>data_transforms = {<br>
‘train’: transforms.Compose([<br>
transforms.Resize((299,299)),<br>
transforms.RandomRotation(degrees=15),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ColorJitter(),<br>
transforms.ElasticTransform(alpha=250.0),<br>
transforms.RandomPosterize(bits=2),<br>
transforms.RandomAdjustSharpness(sharpness_factor=2),<br>
transforms.RandAugment(),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
]),<br>
‘val’: transforms.Compose([<br>
transforms.Resize((299,299)),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
]),<br>
}</p>
<p>print(“Initializing Datasets and Dataloaders…”)</p>
<h1>
<a name="create-training-and-validation-datasets-1" class="anchor" href="#create-training-and-validation-datasets-1"></a>Create training and validation datasets</h1>
<p>image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) for x in [‘train’, ‘val’]}</p>
<h1>
<a name="create-training-and-validation-dataloaders-2" class="anchor" href="#create-training-and-validation-dataloaders-2"></a>Create training and validation dataloaders</h1>
<p>dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=batch_size, shuffle=True, num_workers=4) for x in [‘train’, ‘val’]}<br>
class_names = image_datasets[‘val’].classes</p>
<h1>
<a name="detect-if-we-have-a-gpu-available-3" class="anchor" href="#detect-if-we-have-a-gpu-available-3"></a>Detect if we have a GPU available</h1>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)</p>

juacamgo: <p>You can check it out here:</p>
<blockquote>
<p>Randomized transformations will apply the same transformation to all the images of a given batch, but they will produce different transformations across calls.<br>
<a href="https://pytorch.org/vision/stable/transforms.html" rel="noopener nofollow ugc">PyTorch transforms</a></p>
</blockquote>
<p>The transforms will be applied to all the images.</p>

SrishtiRaj: <p>Thank you So much. I am able to comprehend now</p>
StreamReader with frames_per_chunk less than a specific value gives partial audio signal when streaming

lokesh_v: <p>When frames_per_chunk is given a value less than a specific value, Streaming the audio only gives partial audio. Here if the <code>frames_per_chunk</code> less than 2297, then streaming stops without fully streaming the audio.</p>
<h2><a name="info-1" class="anchor" href="#info-1"></a>Info</h2>
<ul>
<li>Using sox_io as backend</li>
<li>Audio file (<a href="https://dl.espressif.com/dl/audio/gs-16b-1c-44100hz.flac" rel="noopener nofollow ugc">https://dl.espressif.com/dl/audio/gs-16b-1c-44100hz.flac</a>)</li>
<li>torchaudio - 2.0.0+cu118</li>
</ul>
<h2><a name="using-torchaudio-2" class="anchor" href="#using-torchaudio-2"></a>Using torchaudio</h2>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b72f71afc1817a8b4ccbb32ace7dab233c79d2a.png" alt="image" data-base62-sha1="1DhyYoyA3JMdbHJqLifJn1RLofM" width="587" height="102"></p>
<h2><a name="using-streamreader-with-frames_per_chunksample-rate-3" class="anchor" href="#using-streamreader-with-frames_per_chunksample-rate-3"></a>Using StreamReader with frames_per_chunk=sample rate</h2>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c0c8817ea6bf36e3d6fc79d82c7772d968519ae.png" alt="image" data-base62-sha1="mgtkIRq59Aa0sNVO7tqw7nkU9hc" width="623" height="203"></p>
<h2><a name="using-streamreader-with-frames_per_chunk1000-4" class="anchor" href="#using-streamreader-with-frames_per_chunk1000-4"></a>Using StreamReader with frames_per_chunk=1000</h2>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42644781e56e206bf40c27015bf0dadfde33348d.png" alt="image" data-base62-sha1="9tkmlOEGmEBnF8cEVhC6XDWaIfb" width="623" height="203"></p>

lokesh_v: <p>Changing buffer_chunk_size=-1 fixed this issue.</p>
My U-Net model is taking to much memory

Gruny: <p>Hello there. I am working on a school project where we segment medical images. We use x-ray images and need to use them at full resolution. We implemented U-Net with transposed convolution as upsampling (which proved to be superior to using interpolation). The images are 1068x847, which is quite big. However the memory it takes up is huge. We need around 8GiB. I accept that the layers have to save a lot of data to do backdrop, however this seem too much. Thus we are worried about implementation mistakes. Is it normal? Or is there a simple way to see how much space is each layer taking up or something to make sure that this memory usage is justified?</p>

ptrblck: <p>The memory usage might be expected and you could take a look at e.g. <a href="https://discuss.pytorch.org/t/resnet-50-takes-10-13gb-to-run-with-batch-size-of-96/117402/2">this post</a> to get some information about the size of the model parameters as well as the intermediate forward activations.</p>
Question about pytorch-nightly no torchvision/torchaurdio for CUDA 11.6

Bofei_Zhang: <p>I tried to install pytorch nightly with cu116. And I realized there is no suitable version of torchvision torchaudio for cu116.</p>
<pre><code class="lang-auto">pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu116

</code></pre>
<p>Apparently, there are no torchvision and torchaudio in <a href="https://download.pytorch.org/whl/nightly/cu116" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu116</a></p>
<p>How do you guys address this?</p>

ptrblck: <p>We don’t build nightlies for CUDA 11.6 anymore and the current support should include CUDA 11.8 and 12.1.</p>
Error about downing torchvision in official doc

jackwillsmith: <pre><code>I found a download error on your official website, I sent the picture in the attachment, please modify it as soon as possible. torchvision==0.15.1+cu116 should be torchvision==0.15.1+cu117
</code></pre>
<p>Thanks.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/5077cdd256af2e8e7fd4d53eafde9e621bc5d266.png" data-download-href="https://discuss.pytorch.org/uploads/default/5077cdd256af2e8e7fd4d53eafde9e621bc5d266" title="20230515181936"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5077cdd256af2e8e7fd4d53eafde9e621bc5d266_2_690x369.png" alt="20230515181936" data-base62-sha1="btQSlh37pGa99lXyHzeRQP5668u" width="690" height="369" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5077cdd256af2e8e7fd4d53eafde9e621bc5d266_2_690x369.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5077cdd256af2e8e7fd4d53eafde9e621bc5d266_2_1035x553.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5077cdd256af2e8e7fd4d53eafde9e621bc5d266.png 2x" data-dominant-color="F7F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20230515181936</span><span class="informations">1205×645 31.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Thanks for letting us know! We’ll take a look at it.</p>
CUDA L2 cache Set-Aside for Persisting Accesses Do not work in Pytorch_CUDA_EXTENSTION

fancy2509957: <p>I’m trying to use L2 cache Set-Aside for Persisting Accesses on my Nvidia device.<br>
My GPU is “NVIDIA RTX A6000” with   6291456 bytes L2 cache on CUDA 11.6.</p>
<p>Here is my code snippet:</p>
<pre><code class="lang-auto">    cudaStreamAttrValue stream_attribute;                                       // Stream level attributes data structure
    stream_attribute.accessPolicyWindow.num_bytes = 4 * 1024 * 1024;            // Number of bytes for persistence access
    stream_attribute.accessPolicyWindow.hitRatio = 1.0;                         // Hint for cache hit ratio
    stream_attribute.accessPolicyWindow.hitProp = cudaAccessPropertyPersisting; // Persistence Property
    stream_attribute.accessPolicyWindow.missProp = cudaAccessPropertyStreaming; // Type of access property on cache miss

    // Set the attributes to a CUDA stream of type cudaStream_t
    cudaStream_t stream;
    cudaStreamCreate(&amp;stream); // Create CUDA stream
    cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &amp;stream_attribute);
     .....
      myKernel&lt;&lt;&lt;grid,blk,0,stream&gt;&gt;&gt;&gt;();
     
</code></pre>
<blockquote>
<p>Starting with CUDA 11.0, devices of compute capability 8.0 and above have the capability to influence persistence of data in the L2 cache, potentially providing higher bandwidth and lower latency accesses to global memory.</p>
</blockquote>
<p>I’m trying to use L2 cache to accelerate my kernel</p>
<ol>
<li>my code works well compiled with NVCC without integrated into Pytorch. I can see some performance improvement using L2 cache</li>
<li>But ! When using pytorch cuda_extension and pybind11 to pack it as a python module i dont see any acceleration in kernel performance.With some tiny changes in code:</li>
</ol>
<pre><code class="lang-auto">torch::Tensor myApp(
    torch::Tensor input){
  cudaStreamAttrValue stream_attribute;                                       // Stream level attributes data structure
    stream_attribute.accessPolicyWindow.num_bytes = 4 * 1024 * 1024;            // Number of bytes for persistence access
    stream_attribute.accessPolicyWindow.hitRatio = 1.0;                         // Hint for cache hit ratio
    stream_attribute.accessPolicyWindow.hitProp = cudaAccessPropertyPersisting; // Persistence Property
    stream_attribute.accessPolicyWindow.missProp = cudaAccessPropertyStreaming; // Type of access property on cache miss

    // Set the attributes to a CUDA stream of type cudaStream_t
    cudaStream_t stream;
    cudaStreamCreate(&amp;stream); // Create CUDA stream
    cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &amp;stream_attribute);
    myKernel&lt;&lt;&lt;grid,blk,0,stream&gt;&gt;&gt;();

}
</code></pre>
<p>My Pytorch version is 1.13 with CUDA11.6<br>
If there are any conflict between pytorch and L2 cache setting?<br>
It would be helpful if you can provide some hints or suggestions here</p>
Select max q value between specific indexes

NHarris: <p>Hello,</p>
<p>At the moment I am using the code:</p>
<p>(np.flatnonzero(self.model.select_action(self.state) == np.max(self.model.select_action(self.state))))</p>
<p>to select the max Q-value from:<br>
(self.model.select_action(self.state))</p>
<p>However, I would like to select the max Q-value between a certain set of indexes each time depending on the values from the state space i.e. round(self.state[0]) &amp; round(self.stae[1])</p>
Train test split in torch library

jS5t3r: <p>I would like to shuffle my data (without dataloader).</p>
<p>Is there a function like in <code>from sklearn.model_selection import train_test_split</code>?</p>

ptrblck: <p>I would probably just use <code>train_test_split</code> directly to split the data indices and create <code>Subset</code>s using these shuffled and split indices.</p>
Compile pytorch 2.0 with backend support for opencl opengl and vulkan

apcameron: <p>Does pytorch 2.0 support any of the following backends and if so how do I enable them.</p>
<ol>
<li>OpenCL</li>
<li>OpenGL</li>
<li>Vulkan,</li>
</ol>
<p>When I try the version Installed I get the following<br>
python<br>
Python 3.11.3 (main, May  5 2023, 10:18:46) [GCC 12.2.1 20230304] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
t2=torch.randn(1,10).to(‘devices’)<br>
Error in cpuinfo: processor architecture is not supported in cpuinfo<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: devices<br>
t2=torch.randn(1,10).to(‘cpu’)<br>
print(t2)<br>
tensor([[ 1.4973, -0.7791, -0.4402, -1.6017,  0.4876, -0.7664, -0.7203,  0.2741,<br>
-2.4557,  0.5646]])<br>
t2=torch.randn(1,10).to(‘opengl’)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
RuntimeError: PyTorch is not linked with support for opengl devices<br>
t2=torch.randn(1,10).to(‘opencl’)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
RuntimeError: PyTorch is not linked with support for opencl devices<br>
t2=torch.randn(1,10).to(‘vulkan’)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
RuntimeError: PyTorch is not linked with support for vulkan devices</p>
</blockquote>
</blockquote>
</blockquote>
Where is the quantized param saved?

SenJia: <p>Hi, I am using the dynamic quantization on my model, and trying to compute the size reduced. I suppose the number of parameters before and after quantization should be the same, only the number of bits used is changed. However, when I check the model state_dict or parameters, the quantized is not there, and it is not in the buffer. I was trying to estimate the model size by numel * element_size, but it seems the quantized param cannot be found. I saw the document for pruning that I can compute the number of pruned params based on zeros in the _mask in the buffer, but I cannot find docs about how the quantized param is saved. Could anyone please help here, thanks.</p>

jcaip: <p>Hi <a class="mention" href="/u/senjia">@SenJia</a>, can you share the code you are using to quantize / save the model?</p>

HDCharles: <p>see this tutorial:</p>
<p><a href="https://pytorch.org/tutorials/advanced/dynamic_quantization_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/dynamic_quantization_tutorial.html</a></p>
<p>which includes a method for looking at the size of the model.</p>
<p>in general the quantized weight is not simply saved as a quantized tensor with X elements each having Y bits, rather it has to be saved as packedparams which include other intermediate values needed by the quantized matmul to speed up quantization.</p>
How to get numerical/analytical results from torch.autograd.gradcheck

AlphaBetaGamma96: <p><strong>TL;DR</strong> - Is there a way to access the <code>numerical</code> and <code>analytical</code> results from a <code>torch.autograd.gradcheck</code> call?</p>
<p>Hi All,</p>
<p>I’ve recently being trying to validate a custom <code>torch.autograd.Function</code> object via <code>torch.autograd.gradcheck</code>, however, my manually defined backward is incorrect. The <code>torch.autograd.gradcheck</code> function works well at telling me what parts of my jacobian are wrong, but annoyingly I can’t seem to access the <code>numerical</code> and <code>analtyical</code> results from the <code>torch.autograd.gradcheck</code> itself as they’re simply printed within the error function and not returned.</p>
<p>Is there a way to access the <code>numerical</code> and <code>analytical</code> tensors, so I can do an element-wise difference to check what values are correct/wrong?</p>
How to code in the real world for projects

ghanbrialireza770: <p>I have a question for you friends and professors:<br>
And how should coding be done in real world datasets? And what I want to know more is that from the first stage and starting to write code in machine learning and deep learning, how should we identify the next stage and code in such a way that the next stages proceed according to it, and what could the next stage be?</p>
Training and valdiation loss plots

Ali_Youssef: <p>Hello everyone,</p>
<p>I am doing semi-sueprvised binary sementatic segmentation, although the accuracy is pretty good, 95% and a dice score of 0.94, after plotting the running training loss and running validation loss, I was curious if the model is  overfitting.</p>
<p>The training data is 5100 images with augmentations (rotation, random resized crop, random horizontal flip, color jitter and gaussian blur) and 1400 validation images which are only center cropped.</p>
<p>Both data are normalised the same way.</p>
<p>Thank you!</p>
<p>The loss plots seem kind of weird to me and I don’t know how to interpret them.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/8674c60657d2939f3310233369fb7e69d0de38f9.png" alt="output" data-base62-sha1="jbsamEw7OGv1Z1JimgdTAACTkBb" width="565" height="418"></p>
Problem with several parameters to torch.autograd.jacobian()

hl465: <p>Hi,</p>
<p>I want to calculate the jacobian matrix w.r.t. the parameters of a model. This is how I am doing it at the moment:</p>
<pre><code class="lang-auto">from functorch import jacrev, vmap, make_functional, grad

def _compute_centered_jacobian(model,samples):
    func, parameters = make_functional(model)
    def func_ampl(params):
        return func(params, samples)
    def func_phase(params):
        return func(params, samples)[1]
    jac_ampl = torch.autograd.functional.jacobian(func_ampl, parameters)
    jac_phase = torch.autograd.functional.jacobian(func_phase parameters)
    return jac_ampl, jac_phase
</code></pre>
<p>but I am getting the error <code>TypeError: func_ampl() takes 1 positional argument but 9 were given</code> because there are 9 different weight matrices in the model. How can I resolve this problem?<br>
Thanks a lot for your help in advance!</p>