akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Two questions about arrow operator ("->") in torch

hynkang: <p>the code i want to use has an arrow operator ("-&gt;") on their definition of the function<br>
I understand what this operator meaning as the pre-specifiation of in/output variable type.<br>
but i don’t know how to call this to refer/search the document. the first question is (1) what is the name of this operator or this rule?</p>
<p>and as my code described below, this function has different number of outputs depending on the name of the class. the error msg is<br>
*** TypeError: dropout(): argument ‘input’ (position 1) must be Tensor, not tuple<br>
i think the reason is pre-specifiation of the output as only “Tensor”. the second question is … (2) how can i fix it?</p>
<pre><code class="lang-auto">    def forward(self, inputs: Tensor, masks: Optional[Tensor] = None) -&gt; Tensor:
        if self.module.__class__.__name__ is 'CLASSNAME_1':
            out_x, out_y  = self.module(inputs, masks) 
            return out_x, out_y 
        else:
            out = self.module(inputs)
            return out
</code></pre>

eqy: <p>I’m not sure about the exact name of the operator, but these are type annotations for the function that specify input and output types. The error is referring to the type of <code>inputs</code>, if it is a tuple then either the type annotation needs to be changed (seems unlikely) or the caller should make sure that <code>inputs</code> is indeed a <code>Tensor</code>.</p>
Creating Custom Loss For Multiple Labels

thelastspark: <p>Hi all,</p>
<p>I am trying to setup a multi-regression problem. Essentially, I have an input of 3 values: [2, 3, 4] and 2 outputs (ground truth/labels): [ 9, 24]. The first label is the sum, the second label is the product of all the inputs.</p>
<p>I am trying to setup a loss function so that it takes the MSE of the two labels seperately and adds them.</p>
<p>I am not sure if this is the best approach to setting up the loss for this kind of problem, so any advice on how to create the loss how I’m thinking, or a better approach would be greatly appreciated!</p>
<p>P.S.<br>
I have seen this on the forum somewhere:</p>
<pre><code class="lang-auto">def my_loss(output, target):
    loss = torch.mean((output - target)**2)
    return loss
</code></pre>
<p>Which is really close, however, how would I extract the individual labels and have their own torch.mean calls that I can then add and return result of?</p>

AbdulsalamBande: <p>You can create different functions.</p>
<pre><code class="lang-auto">def mse(output, target):
    loss = torch.mean((output - target)**2)
    return loss
</code></pre>
<pre><code class="lang-auto">def my_loss(outputs, targets):
    loss_one = mse(outputs[0],targets[0])
    loss_two = mse(outputs[1],targets[1])
    return loss_one + loss_two
</code></pre>
<p>Then, In the training loop do something like this for averaging</p>
<pre><code class="lang-auto">loss_one = mse(outputs[0],targets[0]) #index the outputs and the targets, take average later
loss_two = mse(outputs[1],targets[1]) #index the outputs and the targets,  take average later
total_loss = my_loss(outputs ,targets)
total_loss.backward()
</code></pre>

thelastspark: <p>That seems simple code wise but i have a few questions:</p>
<ol>
<li>Why are we soung the mse twice in the beginning and kever doing anything with them?</li>
<li>How is it okay to split the output tensor into seperste values and still have it be okay for backward propogstion? The solution seems to be exactly what im looking for, but i fear there is something missing?</li>
</ol>
<p>Thanks in advance!</p>
How to load a model for pytorch1.6 using pytorch1.11

HappyRaymond: <p>hello，I have a model that runs successfully on  pytorch 1.6, but it gets an error on pytorch 1.11:</p>
<pre><code class="lang-auto">TypeError: the first argument must be callable
</code></pre>
<p>Looking forward to your reply</p>

ptrblck: <p>Could you post a minimal, executable code snippet reproducing this issue in 1.11, please?</p>
Should I add empty_cache() after each batch?

giaminhbk: <p>Hello everyone.</p>
<p>I try training my model however I met an OOM problem, I tried to reduce the batch and I want to add empty_cache() after each batch. Here is my question:<br>
If i add <code>torch.cuda.empty_cache()</code> function after each batches, will it affect to  ability to evaluate model?<br>
Will model weights, loss keep updated if I add empty_cache() function</p>

ptrblck: <p>Calling <code>empty_cache()</code> will free the unused memory from the internal caching allocator and make is usable for other processes. You will not avoid OOM issues besides slowing down your code as the memory would have to be re-allocated with synchronizing <code>cudaMalloc</code> calls in the next iteration.<br>
Besides that no effects on e.g. Autograd etc. will be visible.</p>
Does a pretrained Inception V3 that has multioutput for binary classification tasks produce better results than separate networks of pretrained Inception V3 for each of these binary classification tasks? Why?

Mona_Jalal: <p>If I have 5 separate binary classifiers that are using a pre-trained Inception V3 each separately would it provide less accurate results if I modify an inception V3 to create multi-output results as shown in the paper below?</p>
<p>Is there a scientific proof or article that shows multioutput classification of binary classification tasks produces better results than separately considering each as a binary task? Is this because of weight sharing?</p>
<p>Here’s the article I saw this in:</p>
<p><strong>Predicting gene mutational status from whole-slide images</strong>. We next focused on the LUAD slides and tested whether CNNs can be trained to predict gene mutations using images as the only input. For this purpose, gene mutation data for matched patient samples were downloaded from TCGA. To make sure the training and test sets contained enough images from the mutated genes, we only selected those which were mutated in at least 10% of the available tumors. From each LUAD slide, only tiles classified as LUAD by our classification model were used for this task in order to avoid biasing the network to learn LUAD-specific versus LUSC-specific mutations and to focus instead on distinguishing mutations relying exclusively on LUAD tiles. Inception v3 was modified to allow multioutput classification (Methods): training and validation was conducted on ~212,000 tiles from ~320 slides, and testing was performed on ~44,000 tiles from 62 slides. Box plot and ROC curve analysis (Fig. 3a,b and Supplementary Fig. 5) show that six</p>
<p>frequently mutated genes seem predictable using our deep-learning approach; AUC values for serine/threonine protein kinase 11 (STK11), EGFR, FAT atypical cadherin 1 (FAT1), SET binding protein 1 (SETBP1), KRAS and TP53 were between 0.733 and 0.856 (Table 1). Availability of more data for training is expected to substantially improve the performance.</p>
<p><a href="https://www.nature.com/articles/s41591-018-0177-5" rel="noopener nofollow ugc">Classification and mutation prediction from non–small cell lung cancer histopathology<br>
images using deep learning</a></p>
TorchText load JSON for sentiment analysis

teohsinyee: <p>I want to load JSON files for training. Referring to this <a href="https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/A%20-%20Using%20TorchText%20with%20Your%20Own%20Datasets.ipynb" class="inline-onebox" rel="noopener nofollow ugc">pytorch-sentiment-analysis/A - Using TorchText with Your Own Datasets.ipynb at master · bentrevett/pytorch-sentiment-analysis · GitHub</a></p>
<p>I got an error for cell [3]: AttributeError: ‘list’ object has no attribute ‘TabularDataset’</p>
What is a good way to apply WeightDecay on bias terms?

songyuc: <p>Hi, guys,<br>
I heard that WeightDecay might not be appropriate to be applied on all the bias terms. Is that true? And what is a good way to apply WeightDecay on bias terms?</p>
<p>Your answer and guide will be appreciated!</p>

KFrank: <p>Hi Songyuc!</p>
<aside class="quote no-group" data-username="songyuc" data-post="1" data-topic="151583" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a698b9/48.png" class="avatar"> songyuc:</div>
<blockquote>
<p>I heard that WeightDecay might not be appropriate to be applied on all the bias terms.</p>
</blockquote>
</aside>
<p>I haven’t experimented with this in any detail, but I have used weight<br>
decay on entire networks, including the bias terms, without any problems.</p>
<p>It is true – at least according to my intuition – that the bias terms have<br>
less redundancy in them than do, for example, the weight terms of<br>
fully-connected layers.  So it could be that weight decay could be less<br>
beneficial for bias terms (and maybe more likely to cause problems).</p>
<aside class="quote no-group">
<blockquote>
<p>what is a good way to apply WeightDecay on bias terms?</p>
</blockquote>
</aside>
<p>You could either use the <code>weight_decay</code> feature of pytorch’s <code>Optimizer</code>s<br>
and use <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options" rel="noopener nofollow ugc">parameter groups</a> to apply <code>weight_decay</code> to all of the <code>Parameter</code>s<br>
except bias terms or only apply <code>weight_decay</code> to some bias terms or apply<br>
a weaker <code>weight_decay</code> to some bias terms, and so on.</p>
<p>You can also implement weight decay by adding an explicit sum-of-squares<br>
(L2) penalty to your loss function, e.g.,</p>
<pre data-code-wrap="python"><code class="lang-python">loss_weight_decay_penalty = alpha * (fc1.weight**2).sum()
</code></pre>
<p>This would apply weight decay of “strength” <code>alpha</code> to the <code>weight</code><br>
<code>Parameter</code> of <code>Linear</code> layer <code>fc1</code>, but not to <code>fc1</code>’s <code>bias</code> <code>Parameter</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

songyuc: <p>Thanks sincerely! I think the <em>Per-parameter options</em> is a good way to realize this.<br>
By the way, in my opinion, I think only the weights parameters of Conv layers in CNN need WeightDecay.</p>
Low metric scores for CNN

siddharth_MV: <p>I am training a CNN on an image dataset. When I print the classification report, these are the results. I’m not sure why it is not predicting the labels correctly for a few classes. There are sufficient number of samples in each class. Sometimes precision, recall and f1-score become 0.00 for some classes. Any help would be appreciated!</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/552b923f619803129ec39daebcfa008438b68cff.png" alt="image" data-base62-sha1="c9rW24MtxjhkU5mCy9QANWhSsG3" width="607" height="394"></p>

eqy: <p>I would try out some common debugging steps from e.g., <a href="http://karpathy.github.io/2019/04/25/recipe/" rel="noopener nofollow ugc">A Recipe for Training Neural Networks (karpathy.github.io)</a></p>
<p>such as using a very small training set and also using it as the test set to verify that the model is capable of overfitting a small dataset (e.g., reaches 100% accuracy perfect scores). If that doesn’t happen, then it indicates that there is a bug somewhere in the training loop (e.g., data augmentations not compatible between training/eval, label mismatches, etc.).</p>

siddharth_MV: <p>Okay thanks will try it out!</p>
Ragged tensors for list of variable shape 2D tensors in PyTorch in order to be able to feed data of variable shape in a batch size > 1

Mona_Jalal: <p>I have a list of 2D tensors in which the first dimension is varied while the second dimension is not varied.</p>
<p>I came across ragged tensor concept and searching for equivalents came across this answer by <a class="mention" href="/u/ptrblck">@ptrblck</a> for 1D tensors of different shape.</p>
<aside class="quote" data-post="5" data-topic="38699">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/nested-list-of-variable-length-to-a-tensor/38699/5">Nested list of variable length to a tensor</a> 
  </div>
  <blockquote>
    This code might work: 
target = [[[1,2,3], [2,4,5,6]], [[1,2,3], [2,4,5,6], [2,4,6,7,8]]]
max_cols = max([len(row) for batch in target for row in batch])
max_rows = max([len(batch) for batch in target])
padded = [batch + [[0] * (max_cols)] * (max_rows - len(batch)) for batch in target]
padded = torch.tensor([row + [0] * (max_length - len(row)) for batch in padded for row in batch])
padded = padded.view(-1, max_rows, max_cols)
  </blockquote>
</aside>

<p>How can I extend this to a nested list of 2D tensors?<br>
Please consider I have a batch size of 4 and my tensors are of these sizes:<br>
[ 300x512, 1000x512, 11x512, 8000x512]</p>
<p>Is this what you would do when you have different sized tensors in a batch? Because when I don’t have the same size tensors, I get error and vision transformer requires me to have all of my intermediate representation of the same dimension.</p>
Why is my Feed Forward Network predicting NAs?

Alex_628: <p>I did not see a category for noob question or tabular data. Let me know if I need to be in a different category.</p>
<p>I am just learning the basics of Pytorch. I am confused as to why I am getting NAs or poor predictions. My input data was generated from a simple polynomial.</p>
<p>The code is on github <a href="https://github.com/alexhallam/try_torch/blob/d6fdc4678befb85c4827955e1e9e8200f75a1261/main.py#L37" rel="noopener nofollow ugc">here</a></p>
<p>The data is on github <a href="https://github.com/alexhallam/try_torch/blob/main/simple_data.csv" rel="noopener nofollow ugc">here</a></p>

eqy: <p>The learning rate seems like it could be too high here, setting it to <code>0.0002</code> seems to help.</p>

Alex_628: <p>That is interesting. I guess I would have expected error rate problems to result in bad predictions not NAs.</p>
<p>How to people generally adjust their learning rate given their architecture?</p>

eqy: <p>Learning rate scheduling is more of an art than a science in my opinion, but some kind of schedule that decays by an order of magnitude every some number of epochs is an okay baseline to start from. I’d look up the canonical graphs of “learning rate too high” (loss not really decreasing or exploding), “learning rate too low” (loss decreasing steadily/slowly for a long time but not leveling off) as well.</p>
Unsupported or missing image file

A_Shamsadini: <p>openslide.lowlevel.OpenSlideUnsupportedFormatError: Unsupported or missing image file</p>
<p>I got error when I ran this command to generate the corresponding patches how can I solve it? thank you!<br>
|2022-05-19|10:45:09.732| [INFO] … Process: A1<br>
|2022-05-19|10:45:10.526| [ERROR] Crash<br>
Traceback (most recent call last):<br>
File “/content/drive/MyDrive/hover50/infer/wsi.py”, line 746, in process_wsi_list<br>
self.process_single_file(wsi_path, msk_path, self.output_dir)<br>
File “/content/drive/MyDrive/hover50/infer/wsi.py”, line 470, in process_single_file<br>
self.wsi_handler = get_file_handler(wsi_path, backend=wsi_ext)<br>
File “/content/drive/MyDrive/hover50/misc/wsi_handler.py”, line 201, in get_file_handler<br>
return OpenSlideHandler(path)<br>
File “/content/drive/MyDrive/hover50/misc/wsi_handler.py”, line 108, in <strong>init</strong><br>
self.file_ptr = openslide.OpenSlide(file_path)  # load OpenSlide object<br>
File “/usr/local/lib/python3.7/dist-packages/openslide/<strong>init</strong>.py”, line 160, in <strong>init</strong><br>
self._osr = lowlevel.open(filename)<br>
File “/usr/local/lib/python3.7/dist-packages/openslide/lowlevel.py”, line 129, in _check_open<br>
“Unsupported or missing image file”)<br>
openslide.lowlevel.OpenSlideUnsupportedFormatError: Unsupported or missing image file<br>
|2022-05-19|10:45:10.532| [INFO] … Process: A10</p>

eqy: <p>Could you verify the format/validity of the image file? e.g., print the path of the problematic image and inspect it?</p>
Quantization of weights after QAT training

Madhumitha_Sakthi: <p>I am trying to manually set the weight values after QAT training on each conv layer to 4 unique values.<br>
I printed the number of unique values after this operation using the below code:<br>
for name,param in quantized_model.named_parameters():<br>
print(name, len(np.unique(param.cpu().detach().numpy())))<br>
if name == ‘model_fp32.layer1.0.conv1.weight’:<br>
print(param, np.unique(param.cpu().detach().numpy()))</p>
<p>This gave 4 unique weight values per layer.<br>
But, when I do torch.quantization.convert(quantized_model, inplace=True)<br>
with qconfig as torch.quantization.QConfig(activation=torch.quantization.MinMaxObserver.with_args(qscheme=torch.per_tensor_affine, dtype=torch.quint8), weight=torch.quantization.MinMaxObserver.with_args(qscheme=torch.per_tensor_affine, dtype=torch.qint8))</p>
<p>I get 4 unique uint8 values per channel per layer although I chose qscheme as torch.per_tensor_affine.<br>
Can someone please let me know what I am doing wrong?</p>
<p>The end goal is to get 4 unique weight values per layer in the int8 model as well.</p>

andrewor: <p>Hi Madhumitha,</p>
<p>Do you mind sharing the code where you manually set the weights? The call to convert swaps some layers in the model so it’s possible this interfered with the weight values. Have you tried setting the weights manually after convert?</p>
<p>Best,<br>
-Andrew</p>

Madhumitha_Sakthi: <h2>
<a name="hi-andrew-thank-you-for-the-reply-i-found-the-issue-though-i-initially-set-it-to-4-unique-values-per-layer-however-the-batchnorm-folding-performed-just-before-quantization-changes-each-channels-weights-by-the-batchnorm-params-and-therefore-i-saw-4-unique-values-per-channel-thanks-for-your-input-though-1" class="anchor" href="#hi-andrew-thank-you-for-the-reply-i-found-the-issue-though-i-initially-set-it-to-4-unique-values-per-layer-however-the-batchnorm-folding-performed-just-before-quantization-changes-each-channels-weights-by-the-batchnorm-params-and-therefore-i-saw-4-unique-values-per-channel-thanks-for-your-input-though-1"></a>Hi Andrew,<br>
Thank you for the reply! I found the issue though.<br>
I initially set it to 4 unique values per layer. However, the batchnorm folding performed just before quantization changes each channel’s weights by the batchnorm params and therefore I saw 4 unique values per channel.<br>
Thanks for your input though!</h2>
<p>Madhu</p>
Autocast causes GRU memory to explode on AMD ROCm

xkrant: <p>I have an example where the gpu memory used by the GRU explodes on the AMD ROCm but not on a NVIDIA gpu. The issue seems to be caused by autocast, with autocast disabled the memory usage is comparable. Without autocast the allocated memory on AMD is 0.4GB, with autocast the AMD uses 5.1GB. The nvidia gpu uses 0.2GB in both cases. If possible I would still like to use autocast on AMD as well.</p>
<pre><code class="lang-auto"># -*- coding: utf-8 -*-
import torch
from torch import nn
from torch.cuda.amp import autocast
import sys

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

rnn = nn.GRU(input_size=768, hidden_size=512, batch_first=True,
                                 bidirectional=True, num_layers=3, dropout=0.5).to(device)

inputs = torch.randn(10, 231, 768).to(device)
h0 = torch.randn(2, 3, 768).to(device)
print('Allocated after init:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')

with autocast():
    output, hn = rnn(inputs)
    print(output.dtype)
        
print('Allocated after rnn pass:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
print('Size of output:', sys.getsizeof(output.storage()))
print('Size of hn:', sys.getsizeof(hn.storage()))
</code></pre>
<p>Output on nvidia:</p>
<pre><code class="lang-auto">Allocated after init: 0.1 GB
torch.float32
Allocated after rnn pass: 0.2 GB
Size of output: 9461832
Size of hn: 122952
</code></pre>
<p>Output on AMD:</p>
<pre><code class="lang-auto">Allocated after init: 0.1 GB
torch.float16
Allocated after rnn pass: 5.1 GB
Size of output: 4730928
Size of hn: 61488
</code></pre>
Differences between vision transformers variants

binbbaz: <p>I have tried 4 different types of attacks on vision transformers (ViT small and tiny, DeiT small and tiny) but the attack successes on smaller versions are higher than the tiny versions. My understanding is that the smaller versions are stronger than the tiny ones. I expect the results to be higher on tiny than smaller versions. I am really confused. Is this normal? is my understanding correct?</p>
Mps does not work with projects using Torchtext

maqboolurrahim: <p>Hi Pytorch,</p>
<p>Great work everybody to bring <code>Pytorch</code> to Mac M1s! I am working on a text classification problem.<br>
I installed a nightly version of Pytorch to use <code>mps</code> backend. However, I can only install the version <code>0.12.0</code> of <code>torchtext</code> and which throws the error of</p>
<p><code>ImportError: Symbol not found</code></p>
<p>I believe it is because of the version mismatch. I noticed there are wheels available for torchtext version 0.13.0 but sadly not for M1 (arm chip).<br>
Should I wait for the next version of <code>torchtext</code> ?<br>
Or is it another problem?</p>
<p>Thanks in advance.</p>
<p>cc <a class="mention" href="/u/alband">@albanD</a></p>

albanD: <p>Hi,</p>
<p>Thanks for the report!<br>
This is indeed “expected” today: domain libraries do not provided nightly builds for arm. So you are getting the latest released version which is not binary compatible with a nightly version of torch.</p>
<ul>
<li>As of right now, you can build torchtext from source to solve this.</li>
<li>We are also working on making these builds available (you can track <a href="https://github.com/pytorch/vision/pull/5948" class="inline-onebox">Add TorchVision wheel build for M1 by malfet · Pull Request #5948 · pytorch/vision · GitHub</a> for torchvision and we’re working on similar PRs for text and audio).</li>
</ul>
<p>Hope this helps!</p>
Hello world not working with mps

pannous: <p>This simple pytorch hello world script (xor_torch.py) does not converge with mps:</p><aside class="onebox githubgist" data-onebox-src="https://gist.github.com/pannous/1b820d6dc4ba434910ac513609b024ac">
  <header class="source">

      <a href="https://gist.github.com/pannous/1b820d6dc4ba434910ac513609b024ac" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/pannous/1b820d6dc4ba434910ac513609b024ac" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/pannous/1b820d6dc4ba434910ac513609b024ac</a></h4>

  <h5>xor_torch.py</h5>
  <pre><code class="Python">#!/usr/bin/env python
import torch
import torch.nn as nn
import numpy as np

device = torch.device("mps") # factor 100 slower AND erroneous !?!?
device = torch.device("cpu") 

x0 = [[0, 0], [0, 1], [1, 0], [1, 1]]
y0 = [0, 1, 1, 0]</code></pre>
    This file has been truncated. <a href="https://gist.github.com/pannous/1b820d6dc4ba434910ac513609b024ac" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Any suggestions about the root cause?</p>
<p>PS: In general torch with mps works fine here:</p>
<pre><code class="lang-auto">import time
import torch
c = torch.rand((10000, 500)).to("mps")
d = torch.rand((500, 10000)).to("mps")
tic = time.time()
torch.matmul(c, d)
toc = time.time()
print(toc - tic)
print(c.device, "100 times faster than cpu, NICE!")
</code></pre>

albanD: <p>Hi!</p>
<p>I am afraid the speed difference will be true for any GPU-like device you use. There is a small overhead related to using such hardware and so if your model is very small, that overhead is going to make the whole program a lot slower.</p>
<p>There is indeed an issue in the initialization sorry about that. Opened an issue here: <a href="https://github.com/pytorch/pytorch/issues/77849" class="inline-onebox">Conversion from int to float dtype is not working on MPS device · Issue #77849 · pytorch/pytorch · GitHub</a></p>
<p>You can change your code to do the following to fix the issue (just create the Tensor on mps directly):</p>
<pre><code class="lang-auto">x = X_train = torch.tensor(x0, dtype=torch.float32, device=device)
</code></pre>
Introducing Accelerated PyTorch Training on Mac in v1.12

David_Laxer: <aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://pytorch.org//assets/images/METAPT-002-BarGraph-02-static.png" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/" target="_blank" rel="noopener nofollow ugc">Introducing Accelerated PyTorch Training on Mac</a></h3>

  <p>In collaboration with the Metal engineering team at Apple, we are excited to announce support for GPU-accelerated PyTorch training on Mac. Until now, PyTorch training on Mac only leveraged the CPU, but with the upcoming  PyTorch v1.12 release,...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Does this feature support AMD GPUs with Metal or only M1 support?<br>
Does v1.12 nightly build support the Apple Metal only with source?</p>
Runtime Error: Invalid buffer size when calculating cosine similarity

theyorubayesian: <p>I keep getting a RuntimeError: Invalid buffer size: 2894.04 GB when trying to calculate cosine similarity as follows</p>
<pre><code class="lang-auto">DEVICE = "mps"

# usage_data is a (244456, 13) shape dataframe 
usage_data_tensor = torch.from_numpy(usage_data.values).float().to(DEVICE)
user_similarity = F.cosine_similarity(usage_data_tensor[:,:,None], usage_data_tensor.t()[None,:,:])

</code></pre>

albanD: <p>Hi,</p>
<p>I think this is expected? The broadcasting of these Tensors would require almost 3TB of memory to store it.<br>
This also crash on CPU for me.</p>
Error when moving tensor to mps

ZviStein: <p>if torch.backends.mps.is_available():<br>
device = torch.device(‘mps’)<br>
else:<br>
device = torch.device(‘cpu’)</p>
<p>seq.shape<br>
torch.Size([40, 1, 360, 4])</p>
<p>seq<br>
tensor([[[[ 3.0769,  3.0000,  0.0667, -0.7650],<br>
[ 3.0769,  3.0000,  0.0667, -0.7650],<br>
[ 3.0769,  3.0000,  0.0667, -0.7650],<br>
…,<br>
[ 3.0769,  3.0000,  0.0667, -0.7650],<br>
[ 3.0769,  3.0000,  0.0667, -0.7650],<br>
[ 3.0769,  3.0000,  0.0667, -0.7650]]],<br>
[[[ 3.2903,  3.0000,  0.4254, -0.1959],<br>
[ 3.2903,  3.0000,  0.4254, -0.1959],<br>
[ 3.2903,  3.0000,  0.4254, -0.1959],<br>
…,<br>
[ 3.2903,  3.0000,  0.2461, -0.2670],<br>
[ 3.2903,  3.0000,  0.2461, -0.2670],<br>
[ 3.2903,  3.0000,  0.2461, -0.2670]]],<br>
[[[ 4.2152,  3.0000, -1.0094,  1.0848],<br>
[ 4.2152,  3.0000, -1.0094,  1.0848],<br>
[ 4.2152,  3.0000, -1.0094,  1.0848],<br>
…,<br>
[ 3.7883,  3.0000,  0.2461,  0.7290],<br>
[ 3.7883,  3.0000,  0.2461,  0.7290],<br>
[ 3.7883,  3.0000,  0.2461,  0.7290]]],<br>
…,<br>
[[[ 3.7883,  3.0000,  0.0667,  0.6579],<br>
[ 3.7883,  3.0000,  0.0667,  0.6579],<br>
[ 3.7883,  3.0000,  0.0667,  0.6579],<br>
…,<br>
[ 3.7883,  3.0000,  0.0667,  0.6579],<br>
[ 3.7883,  3.0000,  0.0667,  0.6579],<br>
[ 3.7883,  3.0000,  0.0667,  0.6579]]],<br>
[[[ 4.0729,  3.0000, -0.8300,  0.8713],<br>
[ 4.0729,  3.0000, -0.8300,  0.8713],<br>
[ 4.0729,  3.0000, -0.8300,  0.8713],<br>
…,<br>
[ 4.1441,  3.0000, -0.8300,  1.0136],<br>
[ 4.1441,  3.0000, -0.8300,  1.0136],<br>
[ 4.1441,  3.0000, -0.8300,  1.0136]]],<br>
[[[ 3.2903,  3.0000,  1.5015,  0.2310],<br>
[ 3.1480,  3.0000,  1.5015, -0.0536],<br>
[ 3.1480,  3.0000,  1.5015, -0.0536],<br>
…,<br>
[ 3.1480,  3.0000,  1.5015, -0.0536],<br>
[ 3.1480,  3.0000,  1.5015, -0.0536],<br>
[ 3.1480,  3.0000,  1.5015, -0.0536]]]], dtype=torch.float64)</p>
<p>device<br>
PyDev console: starting.<br>
device(type=‘mps’)</p>
<p>seq.to(device)</p>
<p>the masseg I get:<br>
python quit unexpectedly</p>
<p>with the following Error:</p>
<p>libc++abi: terminating with uncaught exception of type c10::TypeError: Trying to convert Double to the MPS backend but there is no mapping for it.<br>
Exception raised from getMPSDataType at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/mps/OperationUtils.mm:121 (most recent call first):<br>
frame <span class="hashtag">#0:</span> at::native::mps::getMPSDataType(c10::ScalarType) + 180 (0x293701318 in libtorch_cpu.dylib)</p>

psobolewskiPhD: <p>Metal GPU doesn’t support float64. You need to make sure your dtype is float32.</p>
How to delete gradients once used

malfonsoarquimea: <p>Hi! I am currently facing a problem where I do something like this:</p>
<pre><code class="lang-auto">for batch_of_inputs in dataset:
   batch_of_inputs .requires_grad_()
   batch_of_outputs=model(preprocess(batch_of_inputs ))
   # compute the gradient of the output with respect to the inputs
   gradients=torch.autograd.grad(outputs=batch_of_outputs,
                                                    inputs=batch_of_outputs,
                                                    grad_outputs=torch.ones_like(batch_of_outputs),
                                                    retain_graph=True)[0]
   save_to_disk(gradients)
</code></pre>
<p>And the thing is that GPU memory keeps increasing as the batches are processed until no more memory is available and the process crashes.<br>
As far as I understand, the problem is that all the gradients are being tracked and stored and the memory is not cleaned after each iteration. How should I delete de gradients so that they don’t use GPU memory once they are saved to disk?<br>
Thanks in advance</p>

tom: <p>The problem is likely <code>retain_graph=True</code>…<br>
My rule of thumb is to never use <code>retain_graph</code> unless I can explain, in my own words, why I need to keep the old autograd graph.<br>
Maybe if you can describe that, we could figure out a solution.</p>
<p>Best regards</p>
<p>Thomas</p>

malfonsoarquimea: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="152094">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>The problem is likely <code>retain_graph=True</code>…<br>
My rule of thumb is to never use <code>retain_graph</code> unless I can explain, in my own words, why I need to keep the old autograd graph.<br>
Maybe if you can describe that, we could figure out a solution.</p>
<p>Best regards</p>
</blockquote>
</aside>
<p>Yes, you were right. I shouldn’t be using <code>retain_graph=True</code>, that was causing the memory usage.<br>
Thanks!</p>
How to train GRU model with correct forward function?

yanzhou01: <p>there are N sequences of data with the shape of (N sequences, length of each sequence, features of each row). each row is corresponded to a class.<br>
I want to use the GRU model to predict the class for each row in each sequence (1, length, features).</p>
<p>But I am confused with updating parameters. Two ways thought:</p>
<ol>
<li>
<p>make predictions of the whole of a sequence (1, <em>length</em>, features). calculate the loss together (the length of the predictions = the <em>length</em> of a sequence). then update the parameters.</p>
</li>
<li>
<p>make prediction of each class of each row in a sequence one by one. calculate the loss for each row and update.</p>
</li>
</ol>
<p>I found that after training with (1), <strong>the model will makes the same predictions regardless of input data (even random data)</strong><br>
(2) is better, but still the loss decreased a little.</p>
<p><strong>what is the difference with the two methods?</strong></p>
<p>model code:</p>
<pre><code class="lang-auto">#...
self.rnn = nn.GRU(100, 200, 1)
self.fc = nn.Linear(1, 7)  # there are 7 classes
#...
</code></pre>
<p>for (1):</p>
<pre><code class="lang-auto">def forward(self, seqs):
     h_n = torch.zeros(1, 200, dtype=torch.float).cuda()
     outputs = []
     for seq in seqs:
          output, h_n = self.rnn(seq, h_n)
          output = self.fc(output)
          output = F.softmax(output)
          outputs += [output]
     return torch.cat(outputs, dim = 1)

#... training
outputs = model(seqs)
loss = loss_fn(outputs, targets) # loss_fn = nn.CrossEntropyLoss()
loss.backward()
optimizer.zero_grad()
optimizer.step()
</code></pre>
<p>for (2):</p>
<pre><code class="lang-auto">def forward(self, seq, h_n):
     output, h_n = self.rnn(seq, h_n)
     output = self.fc(output)
     return F.softmax(output), h_n

#... training
h_n = torch.zeros(1, 200, dtype=torch.float).cuda()
for seq in seqs:
     h_n = h_n.detach()
     output, h_n = model(seq, h_n)
     loss = loss_fn(output, target) # loss_fn = nn.CrossEntropyLoss()
     loss.backward()
     optimizer.zero_grad()
     optimizer.step()
</code></pre>
How to use nn.Linear to transform the specified dimension of a tensor?

Andrew_t_y_Wong: <p>Hi,</p>
<p>Usually, we use torch.nn.Linear to transform a tensor, for example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
x = torch.rand(16,32,64)
W = nn.Linear(64,128)
x = W(x) 
</code></pre>
<p>Then we can get <code>x</code> with shape <code>[16, 32, 128]</code></p>
<p>Now, the question is, what should I do if I want to transform the second dimension of <code>x</code>? That is, transform <code>x</code> from <code>[16,32,64]</code> to <code>[16,the_size_I_want,64]</code></p>
<p>Thanks!</p>

ID56: <p>You can simply swap the dimensions to send 32 to the end before the operation, and then swap it back to the original position after computing the linear projection.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

x = torch.rand(16,32,64)
W1 = nn.Linear(32,128)
y = W1(x.permute(0,2,1)).permute(0,2,1)
print(y.shape)
# (16, 128, 64)
</code></pre>
<p>Alternatively, you can do a trick. A 1d convolution is equivalent to nn.Linear on transposed input when the kernel size is 1. So you can also do:</p>
<pre><code class="lang-python">W2 = nn.Conv1d(32, 128, 1)
z = W2(x)
print(z.shape)
# (16, 128, 64)
</code></pre>
<p>The number of weight and bias parameters of W1 and W2 are also the same. You can check with W1.weight.data.shape and W1.bias.data.shape, and similarly for W2.</p>

Andrew_t_y_Wong: <p>Thank you! That’s cool!</p>
How is optimizer connected to model

sesale: <p>I am a little bit confused about the connection between the model and its optimizer. I am initializing the optimizer with the model’s parameters, e.g.:</p>
<pre><code class="lang-auto">optimizer = Adam(task_model.parameters(), lr=0.003)
</code></pre>
<p>When performing the optimizer step, how does the optimizer know which model to update?<br>
I always was of the opinion that through initialization, the optimizer is somehow connected to the model. S.t. when performing an optimizer step, it will update the model’s parameter, meaning when checking the values of the model’s parameters <code>(list(model.parameters()))</code>, the values should be different before and after performing the backward pass and optimizer step.<br>
Is this correct? This also means that an optimizer is always tied to a specific model, correct?</p>

ptrblck: <p>Yes, your description is mostly correct.<br>
You are explicitly passing the parameters via <code>task_model.parameters()</code> to the optimizer such that it would update these parameters using their <code>.grad</code> attribute (and potentially internal running stats depending on the optimizer) in its <code>.step()</code> call.</p>
<aside class="quote no-group" data-username="sesale" data-post="1" data-topic="151966">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/cdc98d/48.png" class="avatar"> sesale:</div>
<blockquote>
<p>This also means that an optimizer is always tied to a specific model, correct?</p>
</blockquote>
</aside>
<p>It’s not necessarily a model, as you can pass subsets of a model’s parameters or combinations of any parameter sets to the optimizer.</p>

sesale: <p>Thanks, now it’s clear to me! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Unable to run my Inference code for Image classification

Lohith_Kumaar: <p>I was trying to do Inference from my saved model i got this error can anyone help me with it.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torchvision.transforms import transforms
import numpy as np
from torch.autograd import Variable
import torch.functional as F
from io import open
import os
from PIL import Image
import pathlib
import glob
import cv2


train_path='D:/Anaconda Environment/Dataset/final/train'
pred_path='D:/Anaconda Environment/Dataset/Infence Images'


root=pathlib.Path(train_path)
classes=sorted([j.name.split('/')[-1] for j in root.iterdir()])

model=torch.load('Inference model/mobilenet_v2_pre(8).pth')
model.eval()

#Transforms
transformer=transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(),  #0-255 to 0-1, numpy to tensors
    transforms.Normalize([0.5,0.5,0.5], # 0-1 to [-1,1] , formula (x-mean)/std
                        [0.5,0.5,0.5])
])

#prediction function
def prediction(img_path,transformer):
    
    image=Image.open(img_path)
    
    image_tensor=transformer(image).float()
    
    
    image_tensor=image_tensor.unsqueeze_(0)
    
    if torch.cuda.is_available():
        image_tensor.cuda()
        
    input=Variable(image_tensor)
    
    
    output=model(input)
    
    index=output.data.numpy().argmax()
    
    pred=classes[index]
    
    return pred

images_path=glob.glob(pred_path+'/*.jpg')

pred_dict={}

for i in images_path:
    pred_dict[i[i.rfind('/')+1:]]=prediction(i,transformer)
</code></pre>
<blockquote>
<p>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor</p>
</blockquote>
<p>Can anyone suggest me a solution</p>

ptrblck: <p>This won’t work:</p>
<pre><code class="lang-python">    if torch.cuda.is_available():
        image_tensor.cuda()
</code></pre>
<p>as you need to re-assign the tensor:</p>
<pre><code class="lang-python">    if torch.cuda.is_available():
        image_tensor = image_tensor.cuda()
</code></pre>

Lohith_Kumaar: <p>Thank you man<br>
This is Working</p>
Split data by label in ImageFolder Dataset

kenkpix: <p>I have a one folder with a lot of images, and DataFrame that contains a labels to all images. I wrote a function to load my data into ImageFolder Dataset.<br>
And this works good, but i have no idea how to split my ImageFolder Dataset into train and validation. Is it possible to do? I have a 100 000 images in my folder, and like 90% of all data is belongs to class 0, and class 1 has only 10% of all data, so i need to split it equivalently. For example train dataset should contain 10 000 samples (5000 for 0 class and 5000 for 1), validation dataset - 2000 samples (1000 for 0, and 1000 for 1).</p>
<pre><code class="lang-auto">def load_data(data_path, df):
    train_data = torchvision.datasets.ImageFolder(
        root=data_path,
        transform=torchvision.transforms.ToTensor()
    )
    train_data.classes = ['FAKE', 'REAL']
    train_data.class_to_idx = {k: v for k, v in zip(train_data.classes, [0, 1])}
    
    img_folder = data_path + 'faces_224/'
    
    df = df.sort_values(by='image_name')
    labels_array = np.array(df.label.astype('category').cat.codes)
    files = [img_folder + f for f in os.listdir(img_folder) 
             if os.path.isfile(os.path.join(img_folder, f))]
    samples = np.column_stack((files, labels_array))
    train_data.samples = samples
       
    return train_data
</code></pre>

Flint: <p>Yes! here is a very helpful <a href="https://gist.github.com/kevinzakka/d33bf8d6c7f06a9d8c76d97a7879f5cb" rel="nofollow noopener">example</a> on how to do it</p>

kenkpix: <p>Sorry for not writing for a long time. But this code is only suitable for dividing data into training and validation. As I wrote in my question, i need to split my dataset equivalently. The number of samples for each class should be equal in both dataset (train and validation).</p>
<blockquote>
<p>For example train dataset should contain 10 000 samples (5000 for 0 class and 5000 for 1), validation dataset - 2000 samples (1000 for 0, and 1000 for 1).</p>
</blockquote>

Akshay_Gulabrao: <pre><code class="lang-auto">import numpy
# numpy.sort(train.data,key=train.labels)
sorted_by_value = [0]*10
for i in range(10):
  sorted_by_value[i] =(train.data[numpy.where(numpy.array(train.targets) == i)])
  numpy.random.shuffle(sorted_by_value[i])
</code></pre>
How to fix this warning about bool type mismatch?

phoenixbai: <p>I am getting tons of this warnings:</p>
<blockquote>
<p>[W TensorAdvancedIndexing.cpp:1391] Warning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (function masked_fill_impl_cpu)</p>
</blockquote>
<p>my mask is specified as tensor:kBool type, but it keeps complaining about it.<br>
how can I fix it ?  please help.</p>
<p>my code as below:</p>
<pre><code class="lang-auto">auto inputs = at::from_blob(feats.data(), {1, num}, at::kFloat).reshape({1, frame_count, dimension_size});
torch::Tensor length = torch::tensor({frame_count}, {torch::kInt64});

auto mask = torch::ones({1, frame_count}, torch::kBool);
//auto options = torch::TensorOptions().dtype(torch::kBool);
// auto mask = torch::ones({1, frame_count}, options);

std::vector&lt;torch::jit::IValue&gt; feat_inputs;
feat_inputs.emplace_back(inputs);
feat_inputs.emplace_back(length);
feat_inputs.emplace_back(mask);
torch::Tensor outputs = module.forward(feat_inputs).toTensor();
</code></pre>
<p>or how can i disable this warning? it complicates the log with useless messages.</p>
Error in Documentation

AlecPepe: <p>“Calling the model on the input returns a 10-dimensional tensor with raw predicted values for each class.” stated at <a href="https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html" rel="noopener nofollow ugc">this link</a>. In reality, a 2-dimension tensor is returned with dim=0 corresponding to each tensor and dim=1 corresponding to the 10 values in each tensor. There are 10 values in dim=1, not 10 dimensions. Is there anyone I can contact to help get this fixed?</p>

ptrblck: <p>Thanks for reporting this as the phrasing seems to be indeed confusing.</p>
<aside class="quote no-group" data-username="AlecPepe" data-post="1" data-topic="152018">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alecpepe/48/49749_2.png" class="avatar"> AlecPepe:</div>
<blockquote>
<p>Is there anyone I can contact to help get this fixed?</p>
</blockquote>
</aside>
<p>Yes, you could provide a fix by submitting a PR on <a href="https://github.com/pytorch/pytorch/pulls">GitHub</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

AlecPepe: <p>Thank you, first time trying to correct documentation for something like this</p>

ptrblck: <p>That sounds like a good start! As a general recommendation check the <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">Contribution Guide</a> and let me know if you need more information.</p>
CosineAnnealingLR's T_max

Z_Huang: <p>Hi community,</p>
<p>I am seeing to ways of using T_max in CosineAnnealingLR. One is to use the number of epochs and another is to use the number of iterations in the T_max argument.</p>
<blockquote>
<p>Example 1:<br>
torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_of_epoch, eta_min=0)</p>
</blockquote>
<blockquote>
<p>Example 2:<br>
torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_of_iteration, eta_min=0)</p>
</blockquote>
<p>I am not sure which method is correct. Any idea?</p>
Mps do not work as well

yinxiangshi: <p>I am trying using mps on the newest version of torch:1.12.0.dev20220518<br>
however, mps does not work.<br>
Here is my code:</p>
<pre><code class="lang-auto">a = torch.rand((10000, 1000))
b = torch.rand((1000, 10000))
tic = time.time()
torch.matmul(a, b)
toc = time.time()
print(toc - tic)
print(a.device)

a.to("mps")
b.to("mps")
tic = time.time()
torch.matmul(a, b)
toc = time.time()
print(toc - tic)
print(a.device)
</code></pre>
<p>and output:</p>
<pre><code class="lang-auto">0.2177259922027588
cpu
0.22797489166259766
cpu
</code></pre>

yinxiangshi: <p>Btw my device is M1 with macOS 12.3.</p>

nick_elsey: <p>You should assign the device tensor to a variable:<br>
a = a.to(“mps”)<br>
b = b.to(“mps”)</p>

yinxiangshi: <p>I upded my code, so here is the res:<br>
This actually works:</p>
<pre><code class="lang-auto">a = torch.rand((10000, 500))
b = torch.rand((500, 10000))
tic = time.time()
torch.matmul(a, b)
toc = time.time()
print(toc - tic)
print(a.device)

c = torch.rand((10000, 500)).to("mps")
d = torch.rand((500, 10000)).to("mps")
tic = time.time()
torch.matmul(c, d)
toc = time.time()
print(toc - tic)
print(c.device)

0.13344311714172363
cpu
0.0002028942108154297
mps:0
</code></pre>
How can I set `device` globally instead of specify this every time when I create tensors?

ivaquero: <p>GPU acceleration is great. However, when I want to use this feature, I have to specify this every time when I create tensors, like</p>
<pre><code class="lang-python">dev = torch.device("cuda")

torch.ones(2, 3, device=dev)
</code></pre>
<p>How can I set <code>device</code> globally to make it the default value, so that I don’t need to specify <code>device</code> in the following codes ?</p>

eqy: <p>This previous post might be relevant: <a href="https://discuss.pytorch.org/t/how-to-create-a-tensor-on-gpu-as-default/2128">How to create a tensor on GPU as default - PyTorch Forums</a></p>

ivaquero: <p>Thank you! It’s very helpful.</p>