akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can i display image that wrong predicted?

borik: <p>i finding y_pred and y_true for build a confusion matrix but this part i want them to display image that my model predicted is wrong while i looping my dataloader.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1368118616723f9c0d01c1deb853359bc21d0cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/a1368118616723f9c0d01c1deb853359bc21d0cc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1368118616723f9c0d01c1deb853359bc21d0cc_2_690x318.png" alt="image" data-base62-sha1="n09E9rXANTqje8YHAmZo1fILm3i" width="690" height="318" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1368118616723f9c0d01c1deb853359bc21d0cc_2_690x318.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1368118616723f9c0d01c1deb853359bc21d0cc_2_1035x477.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1368118616723f9c0d01c1deb853359bc21d0cc.png 2x" data-dominant-color="252525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1262×582 49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

eqy: <p>You could collect the tensors via a list comprehension:<br>
<code>incorrect = [inputs[i] if output[i] == labels[i] for i in range(len(labels))]</code><br>
Then you could use something like <code>imshow</code> to visualize the tensor<br>
<a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.imshow.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.imshow.html</a></p>
Need Torch compiled with CUDA enabled

pauljurczak: <p>I have CUDA 12.0 installed on Jetson AGX Orin (Aarch64). Following instruction on <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>, I run:</p>
<pre><code class="lang-auto">pip3 install cuda-python
pip3 install torch torchvision torchaudio
</code></pre>
<p>I expected to have Torch compiled with CUDA, but this script:</p>
<pre><code class="lang-auto">import cuda, torch

print(cuda.__version__)
print(torch.__version__)
print(torch.cuda.is_available())
print(torch.cuda.current_device())
</code></pre>
<p>produces:</p>
<pre><code class="lang-auto">12.0.0
1.13.1
False
Traceback (most recent call last):
  File "/home/nick/yolo/cuda-2.py", line 6, in &lt;module&gt;
    print(torch.cuda.current_device())
  File "/home/nick/yolo/yolo-pip/lib/python3.8/site-packages/torch/cuda/__init__.py", line 552, in current_device
    _lazy_init()
  File "/home/nick/yolo/yolo-pip/lib/python3.8/site-packages/torch/cuda/__init__.py", line 221, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
</code></pre>
<p>which contradicts the installation instructions. How can I get Torch with CUDA enabled?</p>

ptrblck: <p>The built binaries mentioned on the PyTorch website do not support Jetson Orin and you should use <a href="https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform/index.html">these instructions</a>.</p>

pauljurczak: <p>Thank you. I will give it a try. Is there a note about no support for <code>Jetson Orin</code> on PyTorch page?</p>

pauljurczak: <p>It works with some caveats, see: <a href="https://forums.developer.nvidia.com/t/wrong-instructions-in-installing-pytorch-for-jetson-platform/237369" class="inline-onebox" rel="noopener nofollow ugc">Wrong instructions in Installing PyTorch for Jetson Platform - Jetson AGX Orin - NVIDIA Developer Forums</a>.</p>

ptrblck: <p>Thanks for letting me know you are seeing issues with the install instructions. We’ll take a look at it.</p>

pauljurczak: <p>Thank you. Can you take a look at a related question about Torchvision: <a href="https://forums.developer.nvidia.com/t/is-there-a-pip-installable-version-of-torchvision-for-jetson-agx-orin/237372" class="inline-onebox" rel="noopener nofollow ugc">Is there a Pip installable version of TorchVision for Jetson AGX Orin? - Jetson AGX Orin - NVIDIA Developer Forums</a>?</p>
Intuition behind adding MultiheadAttention block under activation.py

100deep1001: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Why is the implementation of MultiheadAttention a part of the <code>pytorch/torch/nn/modules/activation.py</code> ?</p>

ptrblck: <p>I don’t know, but <a class="mention" href="/u/zhangguanheng66">@zhangguanheng66</a> might know the reason as he has implemented it in <a href="https://github.com/pytorch/pytorch/pull/18334">this PR</a>.</p>
Way to use FASTERRCNN_RESNET50_FPN model in place of resnet101, Need solution as i am not able to cut last 2 layers and take feature map vector for decoder for Image captioning like resnet() pretrained models

Biswajit_Patra: <p>class Encoder(nn.Module):<br>
def <strong>init</strong>(self, enc_image_size=14):<br>
super(Encoder, self).<strong>init</strong>()<br>
self.enc_image_size = enc_image_size<br>
resnet = torchvision.models.resnet101(pretrained=True)<br>
modules = list(resnet.children())[:-2]                                      # remove linear and pool layers<br>
self.resnet = nn.Sequential(*modules)<br>
self.adaptive_pool = nn.AdaptiveAvgPool2d((enc_image_size, enc_image_size)) # resize image to fixed size using adaptive pool for variable size input image<br>
self.fine_tune()</p>
<pre><code>def fine_tune(self, fine_tune=True):
    for param in self.resnet.parameters():
        param.requires_grad = False
    for child in list(self.resnet.children())[5:]:
        for param in child.parameters():
            param.requires_grad = fine_tune

def forward(self, images):
    out = self.resnet(images)                                                  # (batch_size, 2048, image_size/32, image_size/32)
    out = self.adaptive_pool(out)                                              # (batch_size, 2048, enc_image_size, enc_image_size)
    out = out.permute(0, 2, 3, 1)                                              # (batch_size, enc_image_size, enc_image_size, 2048)
    return out
</code></pre>
Training Loss = 0.0, Validation Loss = nan

PandaKata: <p>Hello, I am training a model, but the training loss is zero and the validation loss is nan. This only happened when I switched the pretrained model from t5 to mt5.</p>
<p>I don’t know what’s wrong because it was working with t5.</p>
<pre><code class="lang-auto">args = Seq2SeqTrainingArguments(
    model_dir,
    evaluation_strategy="steps",
    eval_steps=100,
    logging_strategy="steps", 
    logging_steps=100,
    save_strategy="steps",
    save_steps=200,
    learning_rate=4e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    weight_decay=0.01, 
    save_total_limit=3, 
    num_train_epochs=6, 
    predict_with_generate=True, 
    fp16=True,
    load_best_model_at_end=True,
)

trainer = Seq2SeqTrainer(
    model_init=model_init,
    args=args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aae07a7682068e9803d714ec6747469b8f793232.png" alt="Bildschirm­foto 2022-12-16 um 16.17.08" data-base62-sha1="onE6Z5RiCCA6Q7RMgbCaIyFeTHc" width="670" height="492"></p>

dreidizzle: <p>These models use different data to pretrain. Also, they have different vocabulary tokens, with mT5 being about 8 times as large in its vocabulary (250 k to 32 k). How are you tokenizing? I think you might need to tokenize in different ways and then feed the data to specific models … Maybe this is the issue?</p>

PandaKata: <p>Hey Andrei, thank you for getting back to me. I actually found the solution this morning: Apparently with mt5 there are some problems with fp16 - I had to set it to false and now it is working.</p>

dreidizzle: <p>Oh gotcha. This should not an issue in the latest no? I saw some threads on this issue but it seems from 2021 …</p>

PandaKata: <p>Yes, I also read that. Apparently it is still an issue? I also had the problem with flan-t5 which was only released a few weeks ago, so that’s really weird…</p>
How Adam optimizer influence the learning rate?

laro: <p>I read some papers about how <code>ADAM</code> optimizer works, and there are some issues which seems that are confusing:</p>
<p><code>ADAM</code> equations are:</p>
<p><a href="https://i.stack.imgur.com/GyMqA.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/GyMqA.png" alt="enter image description here" width="690" height="477"></a></p>
<ol>
<li>
<p>In the second formula, we squared the gradient.</p>
<p>1.1 Why we are doing it ?</p>
<p>1.2 What is the gain from this squared?</p>
<p>1.3 Are there any situations that the gradient squared (in the denominator) will give us less performance ?</p>
</li>
</ol>
<p>I read that the disadvantage of <strong><code>AdaGrad</code></strong> algorithm is that <code>as the number of iteration becomes very large learning rate decreases to a very small number which leads to slow convergence</code></p>
<p>It seems to me that <code>ADAM</code> algorithm has the same drawback (at a lower dose): we are dividing the learning rate with the second equation (which has the squared gradient).</p>
<p>With a lot of epochs and iterations the learning rate decreases to a very small number and we will meet the same drawback, no ?</p>
<ol start="2">
<li>Am I wrong ?</li>
</ol>
<p><code>ADAM</code> uses equations 1 and 2 to tune the learning rate.<br>
On the other hand, I saw some examples using <strong>OneCycleLR</strong> scheduler with ADAM.</p>
<p>3.1 Does it seem that if that 2 algorithms (<code>ADAM</code> and <code>OneCycleLR</code>) work together to tune the learning rate on the same time, they may collide with each other ?<br>
are they ?</p>
<p>3.2 Is it wrong to use OneCycleLR with <code>ADAM</code> ?</p>
Most efficient way to execute Linear layers in parallel

Quasiordnung: <p>Hello there,</p>
<p>I have a lot of linear layers (up to 12000) with variable input (1-100) and an output of 1. I currently see two ways to tackle this:</p>
<ol>
<li>Loop through each layer (Low memory consumption ,slow)</li>
<li>Merge them to a big layer using <code>torch.block_diag</code> on all weight matrices (High memory consumption, fast inference but slow when calculating gradients)</li>
<li>Find a middle ground and combine some of the layers</li>
</ol>
<p>The problem with 2 is that I cant directly create a sparse matrix and so I have to first create this big &gt; 100000x100000 matrix with a lot of zeros.</p>
<p>Does anyone here know of a more efficient/better solution for this problem?<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <blockquote>
<p>Loop through each layer (Low memory consumption ,slow)</p>
</blockquote>
<p>I guess you are trying to launch a lot of small kernels in a loop, which would then suffer from the actual dispatching and kernel launch overheads. The CPU would thus not be able to run ahead and schedule all kernels fast enough which would show short periods of high GPU utilization with drops.<br>
If so, then consider using CUDA graphs as it could capture all your calls and replay them in a single go, freeing CPU resources and thus avoiding the CPU-limt.<br>
Have a look at <a href="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/">this blog post</a> for more information and <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">these docs</a> on how to use it.</p>
How does automatic mixed precision handle input downscaling

raduacosma: <p>Hello,</p>
<p>I was wondering whether for automatic mixed precision, Pytorch would benefit from/requires the input to be in the most precise format (e.g. float32) rather than in the smaller one (e.g. float16). The reason I am asking is that storing the input to my model can be very expensive in terms of storage and if I could downscale the input to float16 and operate on that I would. Of course, I could just run the whole model in fp16 but that might give less accuracy and that is why I am interested in automatic mixed precision.</p>
<p>Therefore, does Pytorch downscale the input to fp16 before applying e.g. a linear layer on it in torch.autocast()? In this case, if Pytorch already does that then could I store the fp16 representation myself and use automatic mixed precision to still have fp32 accumulation and train as if the input was fp32? Otherwise, would Pytorch require that the input be fp32 or would it benefit (in terms of performance, perhaps in the backpropagation) from it?</p>
<p>Thank you in advance!</p>

ptrblck: <p>It depends on your actual use case and the used layers. In your particular use case the first layer would be interesting. If it’s accepting <code>float16</code> inputs, <code>autocast</code> will cast the inputs down for you and keep the outputs also in <code>float16</code>. The next layer could then reuse them or cast them back to <code>float32</code> if needed.</p>
<p>A manual cast could work, but you should double check it with your actual model.</p>
<p>Here is a small example which shows the logged operations:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
from torch.testing._internal.logging_tensor import LoggingTensorMode, capture_logs

# setup
lin = nn.Linear(16, 16).cuda()
x = torch.randn(1, 16).cuda()

# default
with capture_logs(is_mode=True) as logs, LoggingTensorMode():
    with torch.cuda.amp.autocast():
        out = lin(x)
for l in logs:
    print(l)
# $1 = torch._ops.aten._to_copy.default($0, dtype=torch.float16)
# $3 = torch._ops.aten._to_copy.default($2, dtype=torch.float16)
# $5 = torch._ops.aten._to_copy.default($4, dtype=torch.float16)
# $6 = torch._ops.aten.t.default($3)
# $7 = torch._ops.aten.addmm.default($1, $5, $6)

# manual cast
x = x.half()
with capture_logs(is_mode=True) as logs, LoggingTensorMode():
    with torch.cuda.amp.autocast():
        out = lin(x)
for l in logs:
    print(l)
# $0 = torch._ops.aten._to_copy.default($0, dtype=torch.float16)
# $1 = torch._ops.aten._to_copy.default($2, dtype=torch.float16)
# $2 = torch._ops.aten.t.default($1)
# $4 = torch._ops.aten.addmm.default($0, $3, $2)
</code></pre>
<p>As you can see, the second approach works and avoids one cast.</p>
Any advice for storing and loading datasets with too many small files

mushan: <p>Hi there,</p>
<p>I’m currently training a model on a large dataset that contains too many small files (~1.5TB and all files are pure tensors of 50~200kb).</p>
<p>Is there any advice on how to store and load this kind of dataset so that I/O limitations have as little of an effect as possible?</p>
<p>I have tried LMDB and it does not work well in distributed training (multiprocess read). Other people suggest SQLite, is that a good choice?</p>
<p>Thanks in advance.</p>
Use of dlib models in pytorch transforms

ksasi: <p>Dear Community,</p>
<p>I have usecase where dlib based models need to used to perform face detection and face alignment. Is there a way to use these models to perform face detection and alignment as part of pytorch transforms?</p>
<p>Please suggest and possibilities or resources.</p>
<p>Thanks</p>

ptrblck: <p>In the common use case you would apply any transformation in the <code>Dataset.__getitem__</code> method on each sample. Assuming your <code>dlib</code> transformation works on a single numpy array (or any other object you could transform into a PyTorch tensor) you could just apply it in the <code>__getitem__</code>, too.<br>
Additionally, you could also write a custom <code>transform</code> class which expects to initialize internal states in the <code>__init__</code> and apply the transformation in its <code>__call__</code> method. Take a look at e.g <a href="https://github.com/pytorch/vision/blob/0dceac025615a1c2df6ec1675d8f9d7757432a49/torchvision/transforms/transforms.py#L107-L138"><code>torchvision.transforms.ToTensor</code></a> for an example. This would allow you to add this custom transformation into e.g. <code>torchvision.transforms.Compose</code>, but make sure the expected dtype is used.</p>

ksasi: <p>Thanks for these details. These are helpful.</p>
'int' object is not iterable

JimmyMa99: <p>When running on a single GPU, it works fine, but when running on multiple GPUs with DataParallel, it reports an error: ‘int’ object is not iterable.</p>
<p>It doesn’t seem to tell me where to report an error.<br>
‘int’ object is not iterable<br>
File “.py”, line 326, in train_eps<br>
cam, pred, _ = model(img)<br>
File “.py”, line 548, in <br>
train_eps(train_loader, model, optimizer, max_step, Q_model, eval_, args)</p>

JimmyMa99: <p>I found the reason, here is my network forward:</p>
<pre><code class="lang-auto">    def forward(self, x):
        d = super().forward_as_dict(x)
        
        cam = self.fc8(d['conv6'])
        n, c, h, w = d['conv6'].size()
        
        f8_3 = self.f8_3(d['conv4'].detach())
        f8_4 = self.f8_4(d['conv5'].detach())
        x_s = F.interpolate(x, (h, w), mode='bilinear', align_corners=True)
        f = torch.cat([x_s, f8_3, f8_4], dim=1)

        x_att = self.PCM(d['conv6'], f)
        cam_att = self.fc8(x_att)
        
        self.featmap = cam + cam_att

        _, _, h, w = cam.size()
        pred = F.avg_pool2d(self.featmap[:, :-1], kernel_size=(h, w), padding=0)

        pred = pred.view(pred.size(0), -1)
        

        return self.featmap, pred,_
</code></pre>
<p>Here, ‘_’ is ‘int’,so I change the code:</p>
<pre><code class="lang-auto">    def forward(self, x):
        d = super().forward_as_dict(x)
        
        cam = self.fc8(d['conv6'])
        n, c, h, w = d['conv6'].size()
        
        f8_3 = self.f8_3(d['conv4'].detach())
        f8_4 = self.f8_4(d['conv5'].detach())
        x_s = F.interpolate(x, (h, w), mode='bilinear', align_corners=True)
        f = torch.cat([x_s, f8_3, f8_4], dim=1)

        x_att = self.PCM(d['conv6'], f)
        cam_att = self.fc8(x_att)
        
        self.featmap = cam + cam_att

        _, _, h, w = cam.size()
        pred = F.avg_pool2d(self.featmap[:, :-1], kernel_size=(h, w), padding=0)

        pred = pred.view(pred.size(0), -1)

        return self.featmap, pred
</code></pre>
<p>Then it work, but I still don’t know why it can run in a single GPU.</p>
Concatenate tensors with only one matching dimension

shrbrh: <p>Is it possible to concatenate two tensors of size torch.Size([1, 1, 16, 16]) and torch.Size([1, 19, 1, 1])?<br>
I tried torch.cat() but it’s giving the following error:<br>
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 16 but got size 1 for tensor number 1 in the list.</p>

J_Johnson: <p>It’s not possible unless you first buffer both tensors to match on the other dims not being concatenated.</p>
<p>Example:</p>
<pre><code class="lang-auto"># initial tensors
x = torch.rand((1,1,16,16))
y = torch.rand((1,19,1,1))

# buffered tensor shells
buffered_x = torch.empty((1,19,16,16))
buffered_y = torch.empty((1,19,16,16))

# assign buffered tensors
buffered_x[:,0:1,:,:] = x
buffered_y[:,:,0:1,0:1] = y

# concat
buffered_xy=torch.cat([buffered_x, buffered_y])

torch.Size([2, 19, 16, 16])
</code></pre>
Problem not enough memory: you tried to allocate 6940492000 byte

TTTT: <p>Need some help !<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/069d288604f66bcaf8f0265ba787b30162c9d7b2.png" data-download-href="https://discuss.pytorch.org/uploads/default/069d288604f66bcaf8f0265ba787b30162c9d7b2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/6/069d288604f66bcaf8f0265ba787b30162c9d7b2.png" alt="image" data-base62-sha1="WvzDYLAerOeS8mXQzbP0FGCkn0" width="690" height="204" data-dominant-color="F5F4F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1131×335 18.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on the error message you are running out of host RAM while trying to allocate ~6.5GB.<br>
Reduce the <code>train_x</code> size or make sure your system has enough RAM.</p>
Error while installing from source

Ashwin_Raju: <p>Hello,<br>
I am trying to build from source and I followed all the instructions mentioned in the GitHub</p>
<p>I am getting this error while building</p>
<pre><code class="lang-auto">FAILED: caffe2/CMakeFiles/torch_cuda_cu.dir/__/aten/src/ATen/native/cuda/GcdLcmKernel.cu.o

./aten/src/ATen/native/cuda/GcdLcmKernel.cu:20:721: error: the value of ‘at::native::gcd_name’ is not usable in a constant e
xpression
../aten/src/ATen/native/cuda/GcdLcmKernel.cu:17:12: note: ‘at::native::gcd_name’ was not declared ‘constexpr’
 const char gcd_name[] = "gcd";
            ^
../aten/src/ATen/native/cuda/GcdLcmKernel.cu: In lambda function:
../aten/src/ATen/native/cuda/GcdLcmKernel.cu:20:1388: error: the value of ‘at::native::gcd_name’ is not usable in a constant expression
../aten/src/ATen/native/cuda/GcdLcmKernel.cu:17:12: note: ‘at::native::gcd_name’ was not declared ‘constexpr’
 const char gcd_name[] = "gcd";
            ^
../aten/src/ATen/native/cuda/GcdLcmKernel.cu: In lambda function:
../aten/src/ATen/native/cuda/GcdLcmKernel.cu:20:2050: error: the value of ‘at::native::gcd_name’ is not usable in a constant expression
../aten/src/ATen/native/cuda/GcdLcmKernel.cu:17:12: note: ‘at::native::gcd_name’ was not declared ‘constexpr’
 const char gcd_name[] = "gcd";
            ^
../aten/src/ATen/native/cuda/GcdLcmKernel.cu: In lambda function:
../aten/src/ATen/native/cuda/GcdLcmKernel.cu:20:2698: error: the value of ‘at::native::gcd_name’ is not usable in a constan
</code></pre>
<p>CUDA: 11.3<br>
CUDNN: 8.2.0<br>
CMAKE version: 3.22.0</p>
<p>Any suggestions on how to deal this issue ?</p>

ptrblck: <p>Could you provide instructions how to reproduce this issue?<br>
In particular, which GCC version are you using, which PyTorch commit etc.?</p>

Jean_Brian: <p>Meet the same problem.<br>
Here is my environment:<br>
gcc 5.5.0<br>
CUDA 10.2<br>
pytorch source tag: 793621984bc192f2e290dfcaa4f9e035f8b81cb8<br>
cmake 3.22.1</p>

Jean_Brian: <p>I solve this problem by upgrading gcc from 5.5 to 7.5</p>

Babar_Mehta: <p>This same errors occurs for me in <code>AbsKernel.cu</code>, and I have tried GCC 9 and 11 with no success. CUDA is set to 11.3 and PyTorch is the release version 1.12.1. Any ideas what might be going on here?</p>
<pre><code class="lang-auto">/aten/src/ATen/native/cuda/AbsKernel.cu: In lambda function:
/aten/src/ATen/native/cuda/AbsKernel.cu:25:2440: error: the value of ‘at::native:
:abs_name’ is not usable in a constant expression
 /aten/src/ATen/native/cuda/AbsKernel.cu:18:12: note: ‘at::native::abs_name’ was n
ot declared ‘constexpr’
   const char abs_name[] = "abs_kernel";
</code></pre>
Scheduler for momentum?

mrartemev: <p>I know that one can use something like torch.optim.lr_scheduler.LambdaLR to schedule lr.<br>
Is there any way to schedule momentum hyperparameter, like in SGD for example?</p>

timesler: <p>With a bit of work, the pytorch schedulers can be extended to handle momentum as well. I’ve had a go implementing a version of LambdaLR that includes momentum, along with a few other scheduler types.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/timesler/lr-momentum-scheduler" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/13936043?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/timesler/lr-momentum-scheduler" target="_blank" rel="nofollow noopener">timesler/lr-momentum-scheduler</a></h3>

<p>Pytorch implementation of arbitrary learning rate and momentum schedules, including the One Cycle Policy - timesler/lr-momentum-scheduler</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Is this the sort of thing you are looking for?</p>
<p>Alternatively, the pytorch/ignite project seems to cover this use case: <a href="https://github.com/pytorch/ignite" rel="nofollow noopener">https://github.com/pytorch/ignite</a></p>

akshay_23: <p>A bit late to the post , but is there any advantage of using a momentum scheduler?</p>
<p>currently checking out <a href="https://towardsdatascience.com/why-0-9-towards-better-momentum-strategies-in-deep-learning-827408503650" rel="noopener nofollow ugc">Demon</a> (paper: <a href="https://arxiv.org/pdf/1910.04952v4.pdf" rel="noopener nofollow ugc">link</a>) where the momentum starts at a very high value such as 0.9 and then reduces to a very low value towards the end.</p>
<p>Sometimes a combination of LR scheduler and momentum scheduler is useful I think , In other <a href="https://www.kaggle.com/code/residentmario/one-cycle-learning-rate-schedulers/notebook" rel="noopener nofollow ugc">articles</a> they suggest using low momentum for high learning rate and towards the end of the epoch increase the momentum and decrease the learning rate.</p>
<p>If there are advantages , Pytorch should include a feature for a momentum scheduler which we could add similar to <em>lr_scheduler.step()</em> we can add a <strong>momentum_scheduler.step()</strong>. A code example would be</p>
<pre><code class="lang-auto">lr_scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
momentum_scheduler = StepM(optimizer, step_size=30, gamma=0.1)
&gt;&gt;&gt; for epoch in range(100):
&gt;&gt;&gt;     train(...)
&gt;&gt;&gt;     validate(...)
&gt;&gt;&gt;     lr_scheduler.step()
&gt;&gt;&gt;     momentum_scheduler.step()
</code></pre>
Unable to build a dll using libtorch + cuda

nguigs: <p>I am trying to build a shared library to then call it from python scripts using ctypes. I had everything working with no cuda support, but did not manage to make it work on the GPU. Below is a minimal example:</p>
<p>I downloaded libtorch-win-shared-with-deps-1.13.1+cu117 from <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a> and unzipped it to C: I then used the following cmake files to build both an executable and a shared library:</p>
<pre data-code-wrap="cmake"><code class="lang-cmake">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
list(APPEND CMAKE_PREFIX_PATH "C:\\libtorch")
project(example-app)

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_library(example-lib SHARED example-app.cpp example-app.h)
target_link_libraries(example-lib "${TORCH_LIBRARIES}")
set_property(TARGET example-lib PROPERTY CXX_STANDARD 14)

add_executable(example-app example-app.cpp example-app.h)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

if (MSVC)
  file(GLOB TORCH_DLLS "${TORCH_INSTALL_PREFIX}/lib/*.dll")
  add_custom_command(TARGET example-app
                     POST_BUILD
                     COMMAND ${CMAKE_COMMAND} -E copy_if_different
                     ${TORCH_DLLS}
                     $&lt;TARGET_FILE_DIR:example-app&gt;)
endif (MSVC)
</code></pre>
<p>The example-app.cpp script is</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

#include "example-app.h"

int test() {
  torch::Tensor tensor = torch::rand({3, 2});
  std::cout &lt;&lt; tensor &lt;&lt; std::endl;
  return 0;
}

int main() {
	test();
	return 0;
}
</code></pre>
<p>and the header</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">extern "C" { __declspec(dllexport) int test(); }
</code></pre>
<p>I then want to call <code>test</code> in a python script with</p>
<pre data-code-wrap="python"><code class="lang-python">import os
import pathlib
import ctypes

libname = pathlib.Path("C:/Users/Matrix/PycharmProjects/torch_test/buil/Release")
os.add_dll_directory(str(libname))
c_lib = ctypes.CDLL(str(libname / "example-lib.dll"), winmode=0)

print(c_lib.test())
</code></pre>
<p>But I get the following error <code>Procedure entry point could not be located in the dynamic link library</code><br>
(sorry for the french):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/551f48ea9c00d45eaa7c68ae6c0eeb60781ecf24.png" alt="Screenshot from 2022-12-16 15-52-05" data-base62-sha1="c91BYavegaeFE7rIe63iSwUY87O" width="419" height="189"></p>
<p>Note that calling the executable works as desired. Any idea what could be causing the problem?<br>
Configurations: Windows 10, MSVC, CUDA 11.7<br>
Thanks for any help</p>
Unclear how Dataloader works

SO_K: <p>I am working on a classification problem for fruits. I’m loading the dataset using pytorch dataloder but i don’t know how it picks up the samples from the dataset. length of the train set and test set (found using len(train_set) …) are reported as 836 and 208 respectively but length of the train loader and test loader is reported as 105 and 26 respectively. Why is it so. I’ve used the batch size of 8 so, the length of train loader and test loader should be 8. But, clearly, i’m wrong. Can anyone explain me or direct me to resource where this is already explained.<br>
Below is the code that i’m using:</p>
<pre><code class="lang-auto">import os
import pandas as pd
import torch
from torch.utils.data import Dataset
from skimage import io
import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.transforms as transforms
import torchvision
from torch.utils.data import DataLoader
from sklearn.metrics import f1_score
import numpy as np

class kinoo(Dataset):
  def __init__(self, csv_file, root_dir, transform=None):
    self.annotations = pd.read_csv(csv_file)
    self.root_dir = root_dir
    self.transform = transform

  def __len__(self):
    return(len(self.annotations))

  def __getitem__(self, index):
    image_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
    image = io.imread(image_path)
    y_label = torch.tensor(int(self.annotations.iloc[index,1]))
    if self.transform:
      image = image.copy()
      image = self.transform(image)
    return (image, y_label)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

#hyperparameters
in_channel = 3
learning_rate = 1e-3
batch_size = 8
num_epochs = 50

#transforms
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
    transforms.Resize((640,640))
])

#load data
dataset = kinoo(csv_file = '/content/drive/MyDrive/aza-exp/aza-fruit-regression-v2_csv/fruit_classificaiton.csv', root_dir = '/content/drive/MyDrive/aza-exp/aza-fruit-regression-v2', transform = transform)

train_set, test_set = torch.utils.data.random_split(dataset, [0.8, 0.2])
train_loader = DataLoader(dataset = train_set, batch_size = batch_size, shuffle = True)
test_loader = DataLoader(dataset = test_set, batch_size = batch_size, shuffle = True)
</code></pre>
<p>Now, when i enquire the length, i get the following output:<br>
len(train_set)<br>
836</p>
<p>len(test_set)<br>
208</p>
<p>len(test_loader)<br>
26</p>
<p>len(train_loader)<br>
105</p>

ptrblck: <p>The length of the <code>DataLoader</code> is defined as the number of batches.<br>
Given a batch size of 8, the expected values are thus:</p>
<ul>
<li>for the training dataset: <code>836 /  8 = 104.5 = 105</code> batches where the last batch will only contain 4 samples</li>
<li>for the test dataset: <code>208 / 8 = 26</code> batches</li>
</ul>
<p>which fits the outputs you’ve posted.</p>

SO_K: <p>Thank you. It’s clear now.</p>
Expected 3d or 4d in conv2d while removing last layer in InceptionV3

DickyA: <p>I’m currently trying use inceptionv3 as a feature extractor but while trying to remove the last layer an error showed up which is the expected 3d or 4d in conv2d, here’s my code to remove the last layer</p>
<p><strong>download and initiate the inceptionv3 (this works)</strong></p>
<pre><code class="lang-auto">modelTuning = torchvision.models.inception_v3(pretrained=False)
modelTuning.to(device)
summary.summary(modelTuning, input_size=(3, 299,299))
</code></pre>
<p><strong>removing the last linear layer (this doesn’t work)</strong></p>
<pre><code class="lang-auto">newClassifier = nn.Sequential(*list(modelTuning.children())[:-2])
modelTuning = newClassifier
</code></pre>
<p>any solution or maybe there’s something that i need to know?</p>

ptrblck: <p>Wrapping modules into <code>nn.Sequential</code> containers will drop all functional API calls from the original <code>forward</code> method.<br>
In your case you would need to check if the layers are still called in the same order as seen <a href="https://github.com/pytorch/vision/blob/0dceac025615a1c2df6ec1675d8f9d7757432a49/torchvision/models/inception.py#L103-L155">here</a> (of course the <code>aux</code> path will be missing).</p>

DickyA: <p>so what i need to do is create the model using the code and then remove the fc manually?</p>

ptrblck: <p>If you only want to remove the <code>self.fc</code> layer you could instead replace it with an <code>nn.Identity</code> without any further modification to the model.<br>
If your use case is more complicated, derive a custom <code>nn.Module</code> from the base model and rewrite the <code>forward</code> as you wish.</p>

DickyA: <p>okay, thankyou for the answer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Parallel indexing inside voxels

Alexandre_Dang: <p>Hello,</p>
<p>How can I (if I can) vectorize those loops ?</p>
<p>(I’m trying to compute the eigenvalues of each pixel of a volume, I saw that torch.eigenvalsh can do that but it doesn’t work on GPU if my volume is too big so I’m trying to make my own implementation, if you guys have any clue on what can I do on that topic I would gladly take any help <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20">)</p>
<pre><code class="lang-python">    A = torch.randn(200, 200, 200, 3, 3) # Hessian matrices of each voxel of input volume
    R = torch.zeros(200, 200, 200, 3, 3) # Temp matrices for each voxel of input volume
    ij = torch.randint(0, 2, size=(200, 200, 200, 2)) # Some indexes for each voxel

    def f(value: torch.float) -&gt; torch.float:
        return value


    img_shape = A.shape[:-2]
    # For each voxel
    for x in range(img_shape[0]):
        for y in range(img_shape[1]):
            for z in range(img_shape[2]): 
                # Do some math
                i = ij[x, y, z, 0]
                j = ij[x, y, z, 1]
                R[x, y, z, i, j] = f(A[x, y, z, i, j])
</code></pre>
<p>I want to be able to do this operation on GPU (as I understand I cannot really use for loops if I want to do that).</p>
Multiple batch dimensions

Ryan_Keathley1: <p>Some of pytorch’s build in modules have support for multiple ‘batch’ dimensions.<br>
For example, the input of the nn.TransformerEncoder module has shape (n_words, batch, dim_in).<br>
Although the first dim is not <em>really</em> a batch dim, it does not change the number of params of the model and can be modified after instantiation of the model.<br>
I want similar functionality for the nn.Linear module. That is, I have input with shape (n_words, batch, dim_in) and I want dimensions 0 and 1 thought of as batch dimensions.<br>
The simplest way to achieve this is to flatten the batch dimensions into 1, run the default forward, and then view/reshape the output back into the original shape.</p>
<p>Is there a better/faster way to do this? Will this even have a performance cost?</p>

ptrblck: <p><code>nn.Linear</code> accepts inputs in the shape <code>[batch_size, *, in_features]</code> where <code>*</code> denotes additional dimensions.</p>

Ryan_Keathley1: <p>Oh god I didn’t even try. Thanks!</p>
Changing labels in Dataloader

S_M: <p>Hi all I have used the following code to select a piece of data. Everything goes well before the data is passed to the Dataloader. For example I want to select the data from ‘cat’, ‘dog’ and ‘deer’ classes with target labels 3, 4 and 5. The target data had selected successfully before passing to the Dataloader after passing the labels changed to the 0, 1 and 2. Is there any problem here?</p>
<p><a href="https://gist.github.com/Miladiouss/6ba0876f0e2b65d0178be7274f61ad2f" rel="noopener nofollow ugc">code</a></p>

ptrblck: <p>Your description of selecting a subset of classes and remap them to <code>[0, nb_classes-1]</code> sounds valid. Do you see any issues running your code? If so, could you post the error messages and stacktraces, please?</p>

S_M: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your reply. There is not any error or warring.</p>
Converting class target to regression targets

Estabi: <p>Hello everyone,<br>
I am sorry in advance if this question is too basic.</p>
<p>But, I am stucked on this problem. I am trying to convert a class target to a regression-like target.</p>
<p>For context, say I have images 224 x 224 RGB belonging to a class corresponding to any element of this array [0, 1, 2, 3, 4] I would like to convert the corresponding class to a target of float type. Given that my images are now in torch tensors of values between 0 and 1.</p>
<p>Is there any pytorch function I can use to do this? If not can someone kindly suggest a way I can simply do this?</p>
<p>Thank you very much in advance.</p>

nwn: <p>Hello,<br>
I am not sure what you exactly mean:</p>
<ol>
<li>If your question is about casting data types, take a look at <a href="https://discuss.pytorch.org/t/how-to-cast-a-tensor-to-another-type/2713">Casting data types</a>
</li>
<li>If you want to run some kind of classification, maybe try to use CrossEntropyLoss or something closely related which should take care of that.</li>
</ol>
<p>If you are just starting, I would recommend to do the PyTorch introduction and use a dataset with less dimensions, e.g. MNIST. From my experience, using those small datasets is much more convenient, since most of the basic methods can easily be transferred to more complex data and models, without requiring a lot of hardware.</p>

Estabi: <p>Hi Luis, thank you for your response.  I understand your responses, but they are not what I am looking for.</p>
<p>What I am looking for is a way to normalize my categories/classes ([0, 1, 2, 3, 4]) where each element represent a class; given an image of say size 224 x 224 (could as well be any size).  Ordinarily, this fits perfectly as a classification problem.  But, I am meaning to convert it to a regression problem, such that my target labels will no longer be categories but regression targets (continuous float) with the same range of my image tensor.</p>

nwn: <p>Hello Tiamiyu,<br>
as far as I remember correctly, when the CE loss is applied, you do not need to convert the classes to floats in advance. In the sense of deep learning you usually do regression even when you want to solve a classification task. When you apply the softmax function a probability score is returned, e.g.<br>
[0.25, 0.15, 0.3, 0.2, 0.1] for a 5-class (0, 1, 2, 3, 4) problem. In this example your network would have predicted that class 2 with a probability score of 0.3 is the most reasonable one.<br>
If you just want to scale your targets, you can do that e.g. with <a href="https://stackoverflow.com/questions/48109228/normalizing-data-to-certain-range-of-values" rel="noopener nofollow ugc">Normalize data to range</a>. Note that you have to check your input range at first, in order to apply the method presented by the attached URL. Does this resolve your question?</p>
Feed LSTM multiple thought vectors

lanka: <p>hi,<br>
is it possible to feed every time step initial thought vector to predict output instead of hidden input in LSTM cell? or concatenated with hidden input. this is for image captioning (refer to image)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/3596573a7080d2fe5ccb3261fc6c5fc6d82113e4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3596573a7080d2fe5ccb3261fc6c5fc6d82113e4" title="pytorch_lstm_captioning"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3596573a7080d2fe5ccb3261fc6c5fc6d82113e4_2_690x353.jpeg" alt="pytorch_lstm_captioning" data-base62-sha1="7E3po17xv9kGNSKY48MB1lH2VOA" width="690" height="353" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3596573a7080d2fe5ccb3261fc6c5fc6d82113e4_2_690x353.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/3/5/3596573a7080d2fe5ccb3261fc6c5fc6d82113e4.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/5/3596573a7080d2fe5ccb3261fc6c5fc6d82113e4.jpeg 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_lstm_captioning</span><span class="informations">907×465 31.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AbdulsalamBande: <p>I think it is possible. During training using the LSTM cell (which is in a for loop), just re-fed the same vector instead of the hidden that the model returns as in <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTMCell.html" rel="noopener nofollow ugc">here</a>.</p>

lanka: <p>any sample code pls…</p>

dreidizzle: <p>I mean when you do <code>ht, (hn, cn) = lstm(input, (h0, c0))</code> and if you do <code>batch_first=True</code> then output is all the hidden states and it is of dimensions B X L X D (with padding). So, one batch for example is ht[0, : , :] and this is the full sequence of hidden states. ht[0, 0, :] is the first hidden sate in batch 1, ht[0, 1, :] is the second in batch 1, etc. If you have another vector <code>kt</code> then you can do something like <code>zt = torch.concat((ht, kt), axis=-1)</code> to make this concatenated so that each hidden state in each batch is appended a vector so the dimension of the above is B X L X (D + D’) where D’ is the dimension of each of your kt batch vectors. Then, you can just apply <code>wt = nn.Linear(zt)</code> to go to a new dimension B X L X D’’ (D’’ is your wanted output dimension) and then you can do a softmax on the last output <code>nn.Softmax(axis=2)(wt)</code> to finally get the outputs per timestep of lstm … Basically, you are adding something to each time step of each batch, hidden states you had in the LSTM …</p>
What is training seed for dnn training? and how to use training seed?

nikiguo93: <p>Can anybody tell me what is training seed for dnn training and how to use it with pytorch?<br>
additionally, before I load a checkpoint and test my model in the evaluation mode, should I remove all the dropout layers in the dnn architecture?</p>

J_Johnson: <p>A random seed is set only when you want reproducible results. For instance, if you’re running an experiment and want to have the same starting parameters, you’d set the seed. Otherwise, leave it alone.</p>
<p>After you’ve trained a model, to turn off dropout and batchnorm layers, use:</p>
<p><code>model.eval()</code></p>
<p>You will also want to turn off gradient descent:</p>
<pre><code class="lang-auto">with torch.no_grad():
    model.eval()
    output = model(data)
    ...
</code></pre>
<p>To change it back to training, use:</p>
<pre><code class="lang-auto">model.train()
</code></pre>
Unbalanced memory usage when using Data Parallelism

Tm4: <p>For using data parallelism, When I’m using data parallelism, I face with the error that CUDA memory is not enough. But one of the GPUs’ memory is occupied around 70 percent and the other one’s memory is occupied 10 percent. I think the main reason is related to the way that I move the tensors to the device because the default device is only one of the devices. I use the following scenario for data parallelism and also for moving the tensors to device:</p>
<pre><code>dev = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = nn.DataParallel(model)

model = model.to(dev)
</code></pre>
<p>How can I use the both of the GPUs’ memory and how can I fix my problem?</p>

ptrblck: <p>That’s a know disadvantage of <code>DataParallel</code> besides the communication overhead, which is why we recommend using <code>DistributedDataParallel</code> as it won’t suffer from these issues.</p>
Is torch.sparse_coo_tensor conversion to torchscript for running on mobile device supported?

Jannis_Spiekermann: <p>Hi,</p>
<p>I tried to run a model containing a <code>torch.sparse_coo_tensor</code> compiled with torchscript on an Android device but got the following error message:</p>
<pre><code class="lang-auto">CppException: Could not run 'aten::_sparse_coo_tensor_with_dims_and_tensors' with arguments from the `SparseCPU` backend. 
This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 
`aten::_sparse_coo_tensor_with_dims_and_tensors` is only available for these backends: [BackendSelect, Functionalize, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy].

BackendSelect: registered at /home/anonymized/git/pytorch/build_android_arm64-v8a/aten/src/ATen/RegisterBackendSelect.cpp:726 [kernel]
</code></pre>
<p>Now I’m wondering, are sparse tensors generally unsupported on mobile devices?<br>
Or is there something else I have to do to get the sparse_coo_tensor running on device?</p>
<p>Thank you!</p>
LSTM with cuDNN

eriknikulski: <p>Hi,</p>
<p>I have A100 GPUs. In the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">documentation</a> it is explicitly stated that cuDNN is used when GPUs are V100. Does this mean that LSTMs performe worse on A100 GPUs? Can this be improved?</p>

ptrblck: <p>No, the docs need to be updated. Thanks for pointing this out.</p>