akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Difference between kaldi mfcc and torchaudio transform mfcc

binhtran: <p>I see that <code>torchaudio.transforms.MFCC</code> and <code>torchaudio.compliance.kaldi.mfcc</code> results are different:</p>
<pre><code class="lang-auto">import torchaudio
import torch

torch.manual_seed(0)
torch.set_printoptions(precision=3, sci_mode=False)

wave = torch.rand(1, 400)

# torchaudio mfcc
transform = torchaudio.transforms.MFCC(
    sample_rate=16000,
    n_mfcc=13,
    melkwargs={"n_fft": 400, "hop_length": 160, "n_mels": 23, "center": False},
)
ta_mfcc = transform(wave)[0].transpose(0, 1)

# kaldi compliance mfcc
kaldi_mfcc = torchaudio.compliance.kaldi.mfcc(wave * 2**15, window_type="hanning")

ft = torch.cat([ta_mfcc, kaldi_mfcc]).transpose(0, 1)

print(ft)
</code></pre>
<p>The result is:</p>
<pre><code class="lang-auto">tensor([[ 92.246, 115.379],
        [-10.815, -34.377],
        [  2.703, -11.685],
        [  0.333, -15.649],
        [  4.773,  -7.279],
        [  1.226, -13.743],
        [  2.976, -10.609],
        [  6.198,  -2.479],
        [  4.769,  -4.193],
        [  5.665,  -0.910],
        [  5.217,  -0.147],
        [  4.096,  -2.355],
        [  5.315,   1.021]])
</code></pre>
<p>Is there a way to configure <code>torchaudio.compliance.kaldi.mfcc</code> so that the result is the same as that of <code>torchaudio.transforms.MFCC</code>.</p>
RuntimeError: Placeholder storage has not been allocated on MPS device!

Fyu: <p>I am trying to train a CNN on a dataset loaded from local storage:</p>
<pre><code class="lang-auto">trainset = datasets.ImageFolder(root=train_dir, transform=train_transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True)
</code></pre>
<p>Note: Problem may be caused by the 4 lines at the bottom here. It didn’t crash with VGG16 after I removed those lines, but I need the code for outputing one of 22 labels</p>
<pre><code class="lang-auto">model = models.resnet18(weights='IMAGENET1K_V1')
model.to(device=device)

for param in model.parameters():
  param.requires_grad = False
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs,22)
</code></pre>
<p>I have set the device to be mps:0 (I have also tried mps and confirmed my laptop has mps)</p>
<pre><code class="lang-auto">device = torch.device("mps:0" if torch.backends.mps.is_available() else "cpu")
</code></pre>
<p>In the training loop, I send the inputs to the device (I have also tried images.to(device) instead of images = …):</p>
<pre><code class="lang-auto">for epoch in range(1):
  train_loss = train(model, trainloader, optimizer, criterion)
</code></pre>
<pre><code class="lang-auto">def train(model,trainloader,optimizer,criterion):
  model.train()

  total_loss = []
  for data in trainloader:
    images, labels = data
    images = images.to(device)
    labels = labels.to(device)
    optimizer.zero_grad()

    outputs = model(images)
    ...
</code></pre>
<p>The code fails at the outputs = model(images) step</p>
<p>Error stack:</p>
<pre><code class="lang-auto">File /opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py:1518, in Module._wrapped_call_impl(self, *args, **kwargs)
   1516     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1517 else:
-&gt; 1518     return self._call_impl(*args, **kwargs)

File /opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/module.py:1527, in Module._call_impl(self, *args, **kwargs)
   1522 # If we don't have any hooks, we want to skip the rest of the logic in
   1523 # this function, and just call forward.
   1524 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
...
File /opt/homebrew/lib/python3.11/site-packages/torch/nn/modules/linear.py:114, in Linear.forward(self, input)
    113 def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 114     return F.linear(input, self.weight, self.bias)

RuntimeError: Placeholder storage has not been allocated on MPS device!
</code></pre>

Fyu: <p>Ha… the issue is the missing device=device in nn.Linear()</p>
Two issues about using `torch.distributed` with GNU screen

ChristLBUPT: <p>The first issue is that it takes a lot of time between the time when the watchdog process catches collective timeout and when the entire process group is taken down (around 12hrs), as shown below:<br>
(Note that I’m launching distributed training through <code>huggingface accelerate</code>, in a <code>GNU screen</code> session. I launched it and went to sleep, leaving the session remaining attached, the next day when I woke my computer up, I saw the hanged progress bar and witnessed the process of taking the entire progress group down. Does it mean that the taking-down process is stuck due to writing block to screen stdout?)</p>
<pre><code class="lang-auto"> 22%|██████▉                         | 15200/70301 [1:40:12&lt;6:08:43,  2.49it/s, loss=0.602, lr=5e-5][E ProcessGroupNCCL.cpp:821] [Rank 1] Watchdog caught collective op
eration timeout: WorkNCCL(SeqNum=27532, OpType=ALLREDUCE, Timeout(ms)=30000) ran for 34143 milliseconds before timing out.
 22%|██████▎                      | 15215/70301 [12:06:01&lt;1174:02:40, 76.73s/it, loss=0.51, lr=5e-5][E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
</code></pre>
<p>The second problem is that when I launched my distributed code, and entered the <code>scroll mode</code> (by typing <code>ctrl+a esc</code>), and then scrolled up for a period more than the timeout duration I set, collective operations will timeout. I observed that processes might not be able to write to stdout during scroll mode, is that the reason for timing out?</p>
DDP inference issue

gaol: <p>Hello! I saved the model in the middle of one epoch and I could also load the model without any errors. However, the predicted results were quite different from what I saved during training. Any suggestions, thanks!</p>
<pre><code class="lang-auto">#training
for epoch in range(num_epochs):
	for seqOhe, hicMat in train_iter:
		model.train()
		i += 1
		outputs = model(seqOhe.to(rank))
		
		loss = ((torch.square(hicMat.to(rank) - outputs))/(256 * 256 * bs)).sum()

		if epoch % 5 == 0 and epoch &gt;= 5 and i == 500:
			if rank == 0:
				torch.save({'model_state_dict': model.state_dict(),
										'model': model.module.state_dict(),
										'optimizer_state_dict': optimizer.state_dict(),
										'seq': seqOhe,
										'pred': outputs,
										'real': hicMat,
										'loss': loss}, "model.pt") 


		optimizer.zero_grad()
		# computing gradients
		loss.backward()
		nn.utils.clip_grad_norm_(model.parameters(), 1.0)
		# accumulating running loss
		running_loss += loss.item()
		# updated weights based on computed gradients
		optimizer.step()

#inference
checkpoint = torch.load("model.pt", map_location=torch.device('cpu'))
new_state_dict = OrderedDict()
for key, val in checkpoint['model_state_dict'].items():
    name = key[7:]
    new_state_dict[name] = val

model.load_state_dict(new_state_dict)
model.to(device)
model.eval()

seqOhe    = checkpoint['seq']
with torch.no_grad():
    outputs = model(seqOhe.to(device, non_blocking=True))

</code></pre>
Transformer attention ignores boolean mask

cinna: <p>In the code for calculating attention, if the mask is boolean, it is not used:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html</a></p>
<p>The relevant lines are:<br>
if attn_mask.dtype == torch.bool:<br>
attn_mask.masked_fill_(attn_mask.logical_not(), float(“-inf”))<br>
else:<br>
attn_bias += attn_mask<br>
attn_weight = query @ key.transpose(-2, -1) * scale_factor<br>
attn_weight += attn_bias</p>
AOTInductor gpu inference

snakedude: <p>I currently use libtorch with torchscript to do gpu inference from c++ and wanted to understand what is the forthcoming replacement for this path. It seems tathat torch_.export.aoti_compile is likely the basis for generating libraries to call from native c++.</p>
<p>I am somewhat concerned however that this path is not generating source code that can be fed into my build system - it’s generating a built shared library. It seems that AOTInductor manages the toolchain. This is effectively a non-starter for me. I can’t control the arguments to the host compiler or nvcc to support different gpu architectures. I can’t cross compile. I can’t even set the host optimization flags. It definitely does not mesh well with usage of an existing c++/nvcc build system. What’s my future path forward here? Am I fundamentally misunderstanding something?</p>
How to get detail info of aten ops in a network

xinli: <p>Hi, I want to grab aten ops information in a model, such as input shapes and other arguments.<br>
I know two tools similarly:</p>
<ol>
<li>torch.profiler can log op input shapes, but do not include other arguments.</li>
<li>torch-summary can log module info, but do not include ops info.</li>
</ol>
<p>So how can I get these information?  Thanks a lot!</p>
Torch Compile Custom Op

Jerome_Ku: <p>If I register a custom op using the <code>torch.library.Library</code> api that calls a <code>triton.jit</code> kernel then compile a module containing this custom op with <code>cpp_wrapper</code> enabled, is the <code>cubin</code> of the <code>triton</code> kernel embedded in the generated CUDA extension?</p>
<p>How does this differ from a module with only (non-custom) <code>aten</code> ops that are compiled using <code>inductor</code> and lowered into <code>triton</code> kernels (through the <code>inductor</code> lowering pipeline) then output using the <code>cpp_wrapper</code> option?</p>
Testing image in image super-resolution task

LamKser: <p>Hi, I’m doing image super-resolution models. When training models, models use fixed image sizes such as 24x24 - 96x96, 32x32 - 128x128 (for LR-HR size) and when testing to evaluate their performance, models load the full image size or any size. How can models process at full size? Thank you so much.</p>
NaN in gradients when using jit.script

mpad: <p>Dear everyone,</p>
<p>I’m seing NaN errors in the gradient calculations of my loss function when using “jit.script” on my loss function.<br>
I’ve narrowed the problem down to a sqrt :</p>
<pre data-code-wrap="python"><code class="lang-python">@torch.jit.script
def myloss(...):
    ... complex operations ...
    aij2 = torch.clip( aij2, 1e-9, 1-1e-9)
    A_aij = torch.sqrt(1-aij2)
   ... more operations ...
</code></pre>
<p>Then the gradient of the loss systematically contains Nans.<br>
If I replace the <code>sqrt</code> by an other operation or remove <code>@torch.jit.script</code> or replace it by <code>@torch.compile</code> then the gradients are free of Nan (as far as my tests go).<br>
Is this expected ? fixable ?<br>
The full operation is quite heavy (and <em>requires</em> the sqrt) so I was hoping <code>jit.script</code> would help. But maybe <code>torch.compile</code> is a better option ? (it takes longer to start-up though…)</p>
<p>Thanks for any hint !</p>
ExportedProgram in libtorch

Donghun_Lee: <p>I’m looking for some example similar to <a href="https://pytorch.org/docs/master/jit.html" class="inline-onebox" rel="noopener nofollow ugc">TorchScript — PyTorch master documentation</a> for running torch.export.ExportedProgram (.pt2) with libtorch. Are there any existing apis for this or any plans supporting this?</p>

spiegelball: <p>I am very interested in this as well.</p>

sef43: <p>also interested in this</p>
How to use MKLDNN in Libtorch

zheng_wang: <p>I want to use mkldnn in Libtorch.1.9 with intel cpu.Like this:</p>
<pre data-code-wrap="C++"><code class="lang-C++">// 创建 Strided 布局的张量
torch::Tensor strided_tensor = torch::randn({ 2, 3 });

// 打印当前张量的布局
std::cout &lt;&lt; "Original Layout: " &lt;&lt; strided_tensor.layout() &lt;&lt; std::endl;

// 使用 to 方法将 Strided 布局的张量转换为 MKL-DNN 布局
torch::Tensor mkldnn_tensor = strided_tensor.to(torch::TensorOptions().layout(torch::kMkldnn));

// 打印修改后的布局
std::cout &lt;&lt; "Modified Layout: " &lt;&lt; mkldnn_tensor.layout() &lt;&lt; std::endl;
</code></pre>
<p>But is wrong . What should I do?</p>
<pre data-code-wrap="shell"><code class="lang-shell">to(options) doesn't support converting to a different layout, but got self.layout being Strided and options.layout set as Mkldnn
</code></pre>
Is there a way to compute weight gradients in Pytorch with no_grad

Nikolay_Blagoev: <p>I know the question seems like it answers itself (compute weight gradients without gradients?). But the issue I am trying to resolve is that I will need to change leaf variables before computing the backwards pass. Without no_grad pytorch will complain about in-place modifications.</p>
<ol>
<li>Define the model (it is split in two stages of a pipeline):</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">class Pipe1(nn.Module):
      def __init__(self):
        super(Pipe1, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

      def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2(x), 2))
        return x
 class Pipe2(nn.Module):
     def __init__(self):
        super(Pipe2, self).__init__()
        
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

     def forward(self, x):
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        
        x = self.fc2(x)
        return F.log_softmax(x)

</code></pre>
<ol start="2">
<li>Make the pipeline:</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">pipe = [Pipe1(), Pipe2()]

</code></pre>
<ol start="3">
<li>Run a micro-batch through the pipeline:</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    for batch_idx, (data, target) in enumerate(train_loader):
      output = pipe [1](pipe [0](data))
      
      loss = F.nll_loss(output, target)
      pipe[0].conv1.weight *= 3
</code></pre>
<p>And… what is four? Given the current loss I would ideally want to compute the derivatives with respect to the new updated weights. Is there any “proper” way of doing this in Pytorch? Would I need to compute the gradients myself?</p>

ptrblck: <p>The approach fails since the intermediate forward activations were not created by the manipulated weights and are thus wrong. You could try to directly pass the new gradients to this particular layer, by this sounds like a painfully manual approach.</p>

Nikolay_Blagoev: <p>I was afraid that would be the case. Thank you for your answer! What did you mean by pass the gradients manually exactly?</p>
What is the appropriate way to initialize a teacher model for distillation?

Hui_Liu: <p>What I want to do:</p>
<ol>
<li>load a teacher model,</li>
<li>during training, do forward pass using teacher model and calculate a L2 loss with the student model to update the student model</li>
</ol>
<p>What I did:</p>
<ol>
<li>construct the teacher model under <code>torch.no_grad()</code> block</li>
</ol>
<p>My question:</p>
<ol>
<li>do I still need to set require_grad = False for all teacher parameters to make sure teacher model is freezed and no backpropagation on teacher model ?</li>
<li>Or , should I put <code>output = teacher_model(input)</code> under <code>torch.no_grad()</code> block  too?</li>
</ol>
<p>Thanks!!</p>
Different results in same settings, only different GPU

TaintlessCupcake: <p>Hello,</p>
<p>I’m experiencing different losses in the same settings, with only different GPU, 4090 and T4.</p>
<p>I already setup the seed and library version issue. What should I do for same result?</p>
<p>All the settings I set for reproduction is below.</p>
<pre><code class="lang-auto">    torch.manual_seed(args.seed)
    torch.cuda.manual_seed(args.seed)
    torch.cuda.manual_seed_all(args.seed)
    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.backends.cuda.matmul.allow_tf32 = False
    torch.backends.cudnn.allow_tf32 = False
</code></pre>
<p>1st step loss output is<br>
1.4394590854644775 0.6447765231132507 0.0026639089919626713 0.0<br>
and<br>
1.4394253492355347 0.6326223611831665 0.002665554638952017 0.0.</p>
<p>It’s different slightly, but makes significant different for result.</p>
<p>Thanks.</p>

ptrblck: <aside class="quote no-group" data-username="TaintlessCupcake" data-post="1" data-topic="193169">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/taintlesscupcake/48/63023_2.png" class="avatar"> TaintlessCupcake:</div>
<blockquote>
<p>What should I do for same result?</p>
</blockquote>
</aside>
<p>You would have to use the same device as there is no guarantee the same algorithms will be selected between different GPU architectures.</p>
Print intermediate gradient values during backward pass in Pytorch using hooks

jsswoosh: <p>I am trying to print the value of each of the intermediate gradients during backward pass of a model, using register backward hooks:</p>
<pre><code class="lang-auto">class func_NN(torch.nn.Module):
    def __init__(self,) :
        super().__init__()
        self.a = torch.nn.Parameter(torch.rand(1))
        self.b = torch.nn.Parameter(torch.rand(1))

    def forward(self, inp):
        mul_x = torch.cos(self.a.view(-1,1)*inp)
        sum_x = mul_x - self.b
        return sum_x

# hook function
def backward_hook(module, grad_input, grad_output):
    print("module: ", module)
    print("inp: ", grad_input)
    print("out: ", grad_output) 

# Training
# Generate labels
a = torch.Tensor([0.5])
b = torch.Tensor([0.8])
x = torch.linspace(-1, 1, 10)
y = a*x + (0.1**0.5)*torch.randn_like(x)*(0.001) + b
inp = torch.linspace(-1, 1, 10)
foo = func_NN()
handle_ = foo.register_full_backward_hook(backward_hook)
loss = torch.nn.MSELoss()
optim = torch.optim.Adam(foo.parameters(),lr=0.001)

t_l = []
for i in range(2):
    optim.zero_grad()
    l = loss(y, foo.forward(inp=inp))
    t_l.append(l.detach())
    l.backward()
    optim.step()
handle_.remove()
</code></pre>
<p>But this does not provide the desired result.</p>
<p>My objective is to print the gradients of the non-leaf nodes like <code>sum_x</code> and <code>mul_x</code>.<br>
Please help.</p>

ptrblck: <p>You can print the gradient directly via its <code>.grad</code> attribute after calling <code>.retain_grad()</code> on these tensors.</p>

soulitzer: <p>You are using module backward hooks which only provide gradient information at the granularity of a module. If you want to look at the gradients wrt specific operators you can look at tensor/grad_fn hooks. See here <a href="https://pytorch.org/docs/stable/notes/autograd.html#backward-hooks-execution" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 2.1 documentation</a></p>
torch.fx.proxy.TraceError: symbolically traced variables cannot be used as inputs to control flow

RonaldYuren: <p>Hi, I am using the “prepare_fx” from torch.ao.quantization.quantize_fx, But I face the problem with "torch.fx.proxy.TraceError: symbolically traced variables cannot be used as inputs to control flow "<br>
I am conducting the model in YOLOv7, while its models/yolo.py in forward() has  “if-elseif”, it seems that Static-quantization can’t deal with the “if” statement, How can I solve this question?</p>
How to wrap layers that have shared weights in FSDP

mspanrin: <p>Hi all!</p>
<p>I am currently trying to wrap a model with a Transformer-like architecture in FSDP.<br>
Currently, I am using the <code>transformer_auto_wrap_policy</code> with <code>Block</code> being the Module to wrap.</p>
<p>However, I would like to also wrap the <code>embedding</code> and <code>lm_head</code> layers. Below you find a pseudo-code example of what I am currently doing:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyModel():

	def __init__(self, n_blocks):

		self.token_embedding = nn.Embedding()
		self.blocks: nn.ModuleList[Block] = create_blocks(n_blocks)
		self.lm_head = Linear()

		# tie weights 
		self.lm_head.weight = self.token_embedding.weight

	def forward(self, x):

		x = self.token_embedding(x)

		for block in self.blocks:
			x = block(x)

		return self.lm_head(x)


# Wrapping model in FSDP

block_auto_wrap_policy = functools.partial(
    transformer_auto_wrap_policy,
    transformer_layer_cls={Block},
)

model = MyModel(2)
model = FSDP(
    model,
    sharding_strategy=ShardingStrategy.FULL_SHARD,
    auto_wrap_policy=block_auto_wrap_policy,
    use_orig_params=True,
    device_id=torch.cuda.current_device(),
)
</code></pre>
<p>From what I can tell this wrap policy does only consider <code>self.blocks</code>. What do I need to change to wrap the other two layers?</p>
<p>Thank you very much in advance!</p>
How to use fixtures with `TestCase`

kunaltyagi: <pre data-code-wrap="py"><code class="lang-py">from torch.testing._internal.common_utils import TestCase
from torch.testing._internal.common_device_type import instantiate_device_type_tests
</code></pre>
<p>I’m wondering if it is possible to use pytest fixtures while creating test cases using pytorch helper functions.</p>
<p>Are there any other better options (apart from hand-rolling)?</p>
Why custom forward function is called 5 times in LayerIntegratedGradients?

IneedMrmeeseeks: <p>Hello,<br>
I use captum LayerIntegratedGradients class for attribution method on my bert classification model.<br>
I managed to get the attributions by following <a href="https://captum.ai/tutorials/Bert_SQUAD_Interpret" rel="noopener nofollow ugc">the great source</a>.</p>
<p>However, the IG method take some times like 5 times more than inference step.<br>
So I look into it, and find out that It calls the custom forward function 5 times.</p>
<pre><code class="lang-auto">def __custom_forward(self, input_ids, mask):
      logits = self.model(input_ids, mask)
      log.info(logits.size())
      log.info(' Custom forward function has been called !!!! ')
      return logits.max(-1).values
</code></pre>
<p>Above code is the custom forward function, and Below is the result.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edca37306424cfc6074629535956cce4ce460d50.png" data-download-href="https://discuss.pytorch.org/uploads/default/edca37306424cfc6074629535956cce4ce460d50" title="lig2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edca37306424cfc6074629535956cce4ce460d50.png" alt="lig2" data-base62-sha1="xVAp8i8hnXYvkfCBxd8U0WfVUeA" width="690" height="160" data-dominant-color="E4E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lig2</span><span class="informations">791×184 2.81 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I get that the (10, 3) shape tensor is because of that I choose n_step as 10. So it should be the delta shape. And first (1,3) is for the logits.</p>
<pre><code class="lang-auto">self.attr_method = LayerIntegratedGradients(self.__custom_forward, self.model.l1.embeddings)

attributions, delta = self.attr_method.attribute(inputs=ids,
                      baselines=ref_input_ids,
                      additional_forward_args=(mask),
                      n_steps=10,
                      return_convergence_delta=True) # B, input_ids, embed_dim
</code></pre>
<p>Then what the other left three of (1, 3) is for?</p>
<p>Thanks in advance!</p>
Images do not line up with labels after shuffling (ImageFolder)

Fyu: <p>I am training an image classifier and have a directory of local images. After loading via ImageFolder and shuffling, I go to print an image but the corresponding label is incorrect</p>
<p>Local file directory:</p>
<p>root<br>
| class1<br>
|- 1.png<br>
|- 2.png<br>
| class2<br>
|- 1.png<br>
|- 2.png<br>
| …</p>
<p>Data loader:</p>
<p>dataset = datasets.ImageFolder(train_dir, transform=train_transform)<br>
dataloader = torch.utils.data.DataLoader(trainset, batch_size = batch_size, shuffle=True)</p>
<p>Image printer:</p>
<p>for images, labels in dataloader:<br>
print(labels[5])<br>
plt.imshow(images[5].permute(1, 2, 0))<br>
plt.show()</p>
<p>The image prints but the output label is incorrect</p>
<p>It works as intended without shuffle=True but I need the randomization for better testing</p>

ptrblck: <p>Your <code>DataLoader</code> doesn’t use the specified <code>dataset</code> but an undefined <code>trainset</code> so make sure the right data is used.</p>

Fyu: <p>Thanks for your response - I renamed trainset to dataset when I was writing this question but missed that one - the naming is consistent in my actual code when I attempt to run it</p>

Fyu: <p>I found the solution. It turns out ‘<strong>for images, labels in dataloader</strong>’ returns the index as labels rather than the classname. To get the class, do <strong>classes[labels.numpy()[0]]</strong></p>
<p>I was confused because the classnames given to me were already numeric (ie, 1, 2, 3) rather than strings (ex ‘cat’, ‘dog’) so I confused the indexes as being the class names</p>
<p>All the best if anyone else encounters this (especially fellow SYDE 572 students!)</p>
Pytorch support for python 3.12

Harsha_J_K: <p>Hi All,</p>
<p>I am trying to use pytorch with python 3.12. However, I see that it is not yet supported (upon browsing the index url). Can someone let me know when is this support planned?</p>
<p>Thanks!</p>
<p><code>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121</code></p>
<blockquote>
<p>Looking in indexes: <a href="https://download.pytorch.org/whl/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu121</a></p>
<p>ERROR: Could not find a version that satisfies the requirement torch (from versions: none) ERROR: No matching distribution found for torch</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/9008cd6eb4a018e5f5ec6e810b3939f6f46a7381.png" data-download-href="https://discuss.pytorch.org/uploads/default/9008cd6eb4a018e5f5ec6e810b3939f6f46a7381" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9008cd6eb4a018e5f5ec6e810b3939f6f46a7381_2_690x357.png" alt="image" data-base62-sha1="kybC7RY6dZE9JRCZieCwzhelVw5" width="690" height="357" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9008cd6eb4a018e5f5ec6e810b3939f6f46a7381_2_690x357.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9008cd6eb4a018e5f5ec6e810b3939f6f46a7381_2_1035x535.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/0/9008cd6eb4a018e5f5ec6e810b3939f6f46a7381.png 2x" data-dominant-color="D3D2E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1328×688 80.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</blockquote>

ptrblck: <p>You can track the support issue <a href="https://github.com/pytorch/pytorch/issues/110436">here</a>.</p>
CNN prediction - CPU or GPU?

skyfail_ufg: <p>I finally finished tuning my model algorithm and it was running pretty fast, about 23 secs per epoch, now I want to get the model’s metrics, I concatenated every single batch results so I could do a quick math when the loops end, but now I’m getting 40 secs/epoch. I think it’s because a) I used torch.cat, b) it’s too small computation to be done on GPU. I’m not sure if I copy everything to CPU will make things faster, because I’ll have the overhead on I/O read/write.</p>
<p>mini-batch loop:</p>
<pre><code>
      if y_true == None:
        y_true = y
        y_regression = torch.argmax(y_hat, dim=1)
      else:
        y_true = torch.cat((y_true, y), dim=0)
        y_regression = torch.cat((y_regression, torch.argmax(y_hat, dim=1)), dim=0)
</code></pre>
<p>epoch loop, after batch loop ends:</p>
<pre><code>
    acc = torch.sum(torch.eq(y_true,y_regression))/y_true.numel()
    print(f'Epoch: {epoch}, loss: {loss}, accuracy: {acc}')
</code></pre>

ptrblck: <p>Instead of calling <code>torch.cat</code> in each iteration, <code>append</code> the temp. results in a <code>list</code> and create the tensor after the loop finished, which should speed up the code as only one <code>copy</code> will be triggered.</p>

skyfail_ufg: <p>It reduced about 1,5 sec/epoch, not a great increase, but anything is worth it! Thanks for your help!</p>
New GPU with same memory is OOM

Lelouch1: <p>HI,</p>
<p>I replaced my old 1080 TI 12 GB with a 2080 Ti 12 Gb.</p>
<p>But now my GPU (2080ti) is out of memory on the same task my 1080ti was able to do (even if the memory was almost full on the older GPU). And they have the same VRAM.</p>
<p>How can we explain this ? Is there a way to solve this problem ?</p>
Kaggle TPU roberta Finetuning

Gurveer_Singh_Virk: <p>I have pretrained roberta-base on dna promoter sequences of plants (working on a project). I am currently trying to finetune it on a downstream task of predicting gene expression values, basically a list of 8 values (corresponding to various tissues) from a single promoter sequence.</p>
<p>This wasn’t possible on kaggle’s gpu (due to memory restrictions), so I tried to do the same on TPU using pytorch-xla (figured that was the best option). The link to the notebook as well as the datasets used are as follows:</p>
<ol>
<li><a href="https://www.kaggle.com/code/gurveersinghvirk/florabert-2/" rel="noopener nofollow ugc">Main Kaggle Notebook</a></li>
<li><a href="https://www.kaggle.com/datasets/gurveersinghvirk/florabert-base" rel="noopener nofollow ugc">Dataset containing code and data</a></li>
</ol>
<p>Version 43 is the one using the pytorch-xla code (as far as I could figure out). The data’s format is as follows:</p>
<p>sequence \t labels<br>
dna_promoter_seq_here list_of_8_values_here</p>
<p>eg: CTCAAGCTGAGCAGTGGGTTTGCTCTGGAGGGGAAGCTCAACGGTGGCGACAAGGAAGAATCTGCTTGCGAGGCGAGCCCTGACGCCGCTGATAGCGACCAAAGGTGGATTAAACAACCCATTTCATCATTCTTCTTCCTTGTTAGTTATGATTCCCACGCTTGCCTTTCATGAATCATGATCCTATATGTATATTGATATTAATCAGTTCTAGAAAGTTCAACAACATTTGAGCATGTCAAAACCTGATCGTTGCCTGTTCCATGTCAACAGTGGATTATAACACGTGCAAATGTAGCTATTTGTGTGAGAAGACGTGTGATCGACTCTTTTTTTATATAGATAGCATTGAGATCAACTGTTTGTATATATCTTGTCATAACATTTTTACTTCGTAGCAACGTACGAGCGTTCACCTATTTGTATATAAGTTATCATGATATTTATAAGTTACCGTTGCAACGCACGGACACTCACCTAGTATAGTTTATGTATTACAGTACTAGGAGCCCTAGGCTTCCAATAACTAGAAAAAGTCCTGGTCAGTCGAACCAAACCACAATCCGACGTATACATTCTGGTTCCCCCACGCCCCCATCCGTTCGATTCA	[54.679647, 60.646678, 54.9113, 78.878474, 21.326259, 27.973276, 17.419968, 40.465529]</p>
<p>There’s 7,22,000 examples of this kind, ~722 mb in total divided into ~400 mb train, 200 mb test and 100 mb eval. When running the code “finetune.py”, all goes well till the training starts (datasets are loaded, processed, etc). But, the latest run took 3+ hrs to get to the next step and the RAM usage kept on increasing. It looked the TPU run was very slow and the run then crashed as it ran out of memory. I have tried accelerate and trainer but those efforts were in vain.</p>
<p>Few questions:</p>
<ol>
<li>Is my approach correct?</li>
<li>What changes should I make?</li>
<li>Can I run this code using HuggingFace Trainer (was originally used in the code)? If so, how?</li>
<li>Is the RAM usage normal?</li>
<li>Should it take this long?</li>
</ol>
<p>If I pass the model as an arg to xmp.spawn, I end up seeing either of “Check failed: data()-&gt;tensor_data” or “RuntimeError: Function AddcmulBackward0 returned an invalid gradient at index 1 - expected device xla:1 but got xla:0”. Why?</p>
<p>Kindly guide.</p>
How to deploy a pytorch detector built by pyinstaller in ubuntu to another host with cuda of different version

proshm: <p>I build a pytorch detector in ubuntu using pyinstaller by the command:<br>
<em><strong>pyinstaller --onefile --name=start_dector start_dector.py</strong></em><br>
The version of cuda on the host is 11.8. And I want to deploy the built file start_dector to another host where cuda10.2 installed. It can execute normally by running on GPU. but if I want to run on GPU, it report an warning msg:<br>
<em><strong>torch/cuda/<strong>init</strong>.py:138: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at /opt/conda/conda-bld/pytorch_1695392026823/work/c10/cuda/CUDAFunctions.cpp:108.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0</strong></em><br>
And it failed to run on GPU. I also tried to copy the file libcudart.so of version 11.8 to the directory where start_dector is located, but it didn’t work</p>

ptrblck: <p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or custom CUDA extensions, since the PyTorch binaries ship with their own CUDA dependencies. You would need to install an NVIDIA driver, which supports the corresponding CUDA runtime libraries.</p>

proshm: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="192842" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or custom CUDA extensions, since the PyTorch binaries ship with their own CUDA dependencies. You would need to install an NVIDIA driver, which supports the corresponding CUDA runtime libraries.</p>
</blockquote>
</aside>
<p>it seems that nvidia driver support cuda is a component of cuda toolkit and it can not install standalonely. So I uninstall the old cuda toolkit and install cuda toolkit v11.8. But it also didn’t work after that and the result was no change.</p>

ptrblck: <p>The CUDA toolkit has the option to install the driver. However, you can download and install the NVIDIA driver standalone.</p>

proshm: <p>Thank you for your help. The version of nvidia driver in src host is 535.98. I found no CUDA toolkit on nvidia’s webside contains the same version of cuda. So I tried the driver of version 535.104.05 but it didn’t work. Where can I get nvidia cuda driver of version 535.98 from?</p>

ptrblck: <p>You wouldn’t need the exactly same driver version and should check the compatibility matrix <a href="https://docs.nvidia.com/deploy/cuda-compatibility/#minor-version-compatibility">here</a>.</p>
Export encoder and decoder model to import in Android Studio

Raelynn_w: <p>I have follow this tutorial to train the language translation model: <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
<p>I would like to ask how can i export the model into encoder and decoder based on the tutorial’s code above. After export I want to put in to Android studio using kotlin language. Or is there any method to train the language translation and import into Android Studio using Kotlin.</p>
<p>Thanks for helping <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=12" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:" loading="lazy" width="20" height="20"></p>
Compute gradients of weights with respect to variable in loss function

JonathanBe: <p>Hello everyone,</p>
<p>I’m currently working with PyTorch and have a question about the <code>optimizer.step()</code> function and autograd. I understand that <code>optimizer.step()</code> performs a single optimization step (parameter update) based on the gradients that have already been computed and stored in the <code>.grad</code> attribute of the parameters during the backward pass. However I also need something different which seems to be just the opposite.</p>
<p><strong>Here is my question:</strong></p>
<p>I’m interested in understanding how a specific variable in my loss function influenced the weights during the update (optimizer.step). Is there a way to compute the gradients of the weights with respect to a variable in the loss function of an ANN?</p>
<p><strong>To understand the background of why I need this, here is a detailed explanation:</strong></p>
<p>I am building a combination of two ANNs. The 2nd ANN computes an output (psi) that is used inside the loss function of the 1st ANN. The loss function of the 2nd ANN is derived from the output of the 1st ANN. Currently, PyTorch is not able to compute the gradient of the loss function of the 2nd ANN with respect to its parameters because they do not influence the output of the 1st ANN (which is used inside the loss of the 2nd ANN) directly but only via a training step that updates the parameters of the 1st ANN depending on the output of the 2nd ANN.<br>
Apparently PyTorch does not track the influence of the loss function on the weights during the optimizer.step() in the autograd graph.</p>
<p>Any insights or suggestions would be greatly appreciated. Thank you in advance!</p>
<p>Best, Jonathan</p>

soulitzer: <blockquote>
<p>Apparently PyTorch does not track the influence of the loss function on the weights during the optimizer.step() in the autograd graph.</p>
</blockquote>
<p>You can enable this tracking by specifying <code>differentiable=True</code> in <a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD" class="inline-onebox" rel="noopener nofollow ugc">SGD — PyTorch 2.1 documentation</a></p>

JonathanBe: <p>Thank you! I just found TorchOpt which also allows differentiation through the optimization path. But I will also try this.</p>
Self-built of pytorch wheel without having to install mkl in the target environment

osllogon: <p>I have built from tag 1.13.1 my own wheel for installation. I tried to use the same features as the official version for cpu (1.13.1+cpu) and that is why I have used mkl for compiling. However, when I try to install my pytorch version in a new environment this error is raised:</p>
<p><code>OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or director</code></p>
<p>I know that this will be solved by installing mkl, but how can I create the wheel without this being a requirement, since with the official version this is not required</p>

kandarp_sharda: <p>Have you found solution for it?</p>
Does torch.compile support gradient operation in the model?

wuchiz: <p>I need to calculate the gradient of the output of network with respect to input in my model. When model is not compiled, the training is normal otherwise report an error when calculating the gradient</p>
<blockquote>
<p>unsupported operand type(s) for *: ‘Tensor’ and ‘NoneType’<br>
File “/home/yklei/practice/mlmm_energy/test/debug/mlmm/model/model_pl.py”, line 148, in <br>
f = grad(<br>
File “/home/yklei/practice/mlmm_energy/test/debug/mlmm/model/model_pl.py”, line 89, in forward<br>
for key in g.ndata.keys():<br>
File “/home/yklei/practice/mlmm_energy/test/debug/mlmm/model/model_pl.py”, line 220, in training_step<br>
results = self(g_qmmm, cell = cell)<br>
File “/home/yklei/practice/mlmm_energy/test/debug/mlmm_main_pl.py”, line 38, in <br>
cli = MyLightningCLI(LitMLMM, Molecule_DataModule)#, subclass_mode_model=True)<br>
TypeError: unsupported operand type(s) for *: ‘Tensor’ and ‘NoneType’</p>
</blockquote>
<p>I write a simple code to check whether gradient operation is supported when model is compiled</p>
<pre><code class="lang-auto">import torch
def fn(x, y):
    x.requires_grad_()
    y.requires_grad_()
    a = torch.cos(x).cuda()
    b = torch.sin(y).cuda()
    fmm = torch.autograd.grad(
                    a + b,
                    [x,y],
                    grad_outputs=torch.ones_like(a+b),
                    create_graph=True,
                    retain_graph=True )
    return a + b, fmm
new_fn = torch.compile(fn, backend="inductor")
input_tensor = torch.randn(10000).to(device="cuda:0")
a,fmm = new_fn(input_tensor, input_tensor)
print(a)
print(fmm)
</code></pre>
<p>it actually reported an error</p>
<blockquote>
<p>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
</blockquote>
<p>is there any way to solve this?</p>

bdhirsh: <p>Late reply, but:</p>
<p>(1) Any chance you’ve tried this on a more recent nightly? I believe this should work now</p>
<p>(2) The high level problem is that torch.compile works best when capturing large backward graphs, but suffers a bit when you want to do more fine-grained surgery on the backward graph. In your example, you’re changing the requires-gradness of your inputs halfway through your function, which is a pain to support. So as of this PR: <a href="https://github.com/pytorch/pytorch/pull/110053" class="inline-onebox" rel="noopener nofollow ugc">[dynamo] Add graph break on requires_grad_() by int3 · Pull Request #110053 · pytorch/pytorch · GitHub</a>, we now graph break on <code>x.requires_grad_(True)</code></p>
<p>For better performance, I’d recommend moving the requires_grad_() (and probably also the .cuda()) calls outside of the compiled region, and letting torch.compile just directly handle the compute (cos/sin) and the backwared calls.</p>

wuchiz: <p>Thank your for your replies! As you suggested, I use new version, then it works. But it seems that it doesn’t suport the module with external library such as deep graph library.</p>
Comparing Accuracy: Single GPU vs. 8 GPUs

External_happy: <p>Would PyTorch yield different accuracy when executed on 8 GPUs (distrusted mode) compared to running on 1 GPU? Is it expected to observe variations in results? For instance, the accuracy on a single GPU for the DTD dataset is 50.1%, whereas when utilizing 8 GPUs, it is reported as 54.1% using ViT-B/16.</p>

eqy: <p>That difference sounds to be large to be expected. Could you explain more details of the setup (the batch size in each case, the types of GPUs used, whether quantization or mixed-precision is used), and whether you are observing this difference in training, validation, or both?</p>
How to train custom made dataset for beam engineering problem?

Bluegirl123: <p>Hi!</p>
<p>Greetings! In my machine learning problem, I have an input, which is a Gaussian beam, The input Gaussian beam has to pass through different layers and finally produce a beam of desired shape. My model looks something like this,</p>
<blockquote>
<p>E= A* torch.exp(-((x)**2+(y)<strong>2)/(wa</strong>2)); # This is a analytical formula for gaussian beam<br>
G=torch.fft.fft2(E) # Taking Fourier transform of it.</p>
</blockquote>
<h1><a name="propagate-through-the-1st-layer-1" class="anchor" href="#propagate-through-the-1st-layer-1"></a>Propagate through the 1st layer</h1>
<p>elayer1=torch.fft.ifft2(torch.fft.ifftshift((torch.fft.fftshift(G))*H0)); # Multiplying it with the transfer function</p>
<h1><a name="multiplying-elayer1-with-phase1-values-2" class="anchor" href="#multiplying-elayer1-with-phase1-values-2"></a>Multiplying Elayer1 with phase1 values</h1>
<p>elayer1=elayer1<em>torch.exp(1j</em>(Phase1));<br>
predictedbeam=torch.abs(elayer1)**2</p>
<p>In the first line, I have given the Gaussian beam. I want to train this model, with many number of gaussian beams that are tilted slightly (E= A* torch. exp(-((x-a)*<em>2+(y-b)<strong>2)/(wa</strong>2)). So that the model becomes robust even if there is a slight tilt in the Gaussian beam position.  I have made a custom-made dataset. I classified the Gaussian beam based on how they are titled (E= A</em> torch. exp(-((x-a)**2+(y-b)<strong>2)/(wa</strong>2))). Now I want to train them. In the place of E, in the code, I want to give a folder that has many Gaussian beams. I don’t know how to go about it. Can anyone kindly help me with this?</p>
Updating to latest CuDNN without building pytorch from source

zplizzi: <p>I want to update to CuDNN 8.9.6; the version included in a pip build of torch 2.1.1 is 8.9.2. Nowadays, CuDNN is installed as part of a torch pip install using a pip dependency - with torch 2.1.1, the dependency is <code>nvidia-cudnn-cu12==8.9.2.26</code>. There’s an updated version of this package with the version I want - <code>nvidia-cudnn-cu12==8.9.6.50</code>. Can I just install this version and expect that it’ll work properly? Or does the version need to exactly match the version that pytorch was built with?</p>
<p>I have tried updating it, and nothing seems broken, but I don’t know how to know if it’s actually using the new version or not.</p>

ptrblck: <p>Updating libs manually by overwriting the files is not supported and might break. If you want to update specific libs, source builds are the recommended approach.</p>
Relationship of batch size and Variational AutoEncoder training

pumplerod: <p>I’ve been attempting to implement a Variational AutoEncoder, and my test example (MNIST) works quite well.  However, my actual data is rather memory intensive and I’m required to limit the batch size to something like 5 images.</p>
<p>I’m wondering if the smaller batch size has any effect when computing the KL_Loss.  Since, as I understand it, that loss on the latent vector space is trying to insure values are spread across the Gaussian domain.  Is it possible that for a VAE to be adequately trained I would need a reasonably large batch size?</p>
<p>The “reparameterization” is adding Gaussian noise to the latent vector and I’m wondering if this has a detrimental effect with a small batch size.</p>
Compiler support for aarch64 CPU inference

yd2102: <p>Hi, I’m wondering how complete the Dynamo/Inductor support currently is for aarch64? It seems compiler has good coverage for x86 but not aarch64.</p>
<p>For example, if I pick the hugging-face “bert-base-uncased” model and compile it with default options, I got 2.5x worse latency on aarch64 (25ms) than on x86 (10ms). To be more specific, I’m comparing the the c6i (Intel Xeon) and c7g (ARMv8.4-a) instance types using AWS EC2.</p>
<p>So my questions are: Is there any near future plan on aarch64 support/optimization for Dynamo/Inductor? Any roadmap so far?</p>
<p>Thanks!</p>

snadampal: <p>Hi <a class="mention" href="/u/yd2102">@yd2102</a>, please check this <a href="https://github.com/pytorch/pytorch/pull/115037" rel="noopener nofollow ugc">PR</a> I’ve pushed today, this enables the mkldnn passes for aarch64 when it’s compiled with ACL backend. I have observed up to 5.8x performance improvement on AWS c7g instance for bert base uncased inference with torch.compile().</p>
Segmentation fault passing tensor and using as type const float*

Louisex: <p>Hi I want to enable gradients for the tan_fovx and tan_fovy variables in this library: <a href="https://github.com/graphdeco-inria/diff-gaussian-rasterization/tree/59f5f77e3ddbac3ed9db93ec2cfe99ed6c5d121d" class="inline-onebox" rel="noopener nofollow ugc">GitHub - graphdeco-inria/diff-gaussian-rasterization at 59f5f77e3ddbac3ed9db93ec2cfe99ed6c5d121d</a><br>
I modified their types in rasterize_points.cu to const torch::Tensor&amp;, which were originally float. And referenced them with tan_fovx.contiguous().data() so they can be used as const float* in other places. In the actual computations I used *tan_fovx to refer to them.<br>
I was able to get the library to build but it segfaulted in run time. What am I doing wrong? Thanks.</p>

ptrblck: <p>Check the stacktrace via <code>gdb</code> and try to narrow down which memory access fails. Maybe the underlying data was already freed when you are trying to access the <code>float*</code>.</p>