akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why shuffling DataLoader changes model accuracy even though model is in eval mode?

The_Snail: <p>So I created the following simple function for evaluating the trained model:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import DataLoader
from torchmetrics import MeanAbsolutePercentageError, MeanSquaredError, MeanAbsoluteError, R2Score

def compute_metrics(y_pred, y_true, device):
    mape = MeanAbsolutePercentageError().to(device)(y_pred, y_true)
    mse = MeanSquaredError().to(device)(y_pred, y_true)
    mae = MeanAbsoluteError().to(device)(y_pred, y_true)
    r2 = R2Score().to(device)(y_pred, y_true)
    return mape, mse, mae, r2


def evaluate_model(model, dataset, device):
    dataloader = DataLoader(dataset, batch_size=32, shuffle=True, drop_last=True)
    with torch.no_grad():
        model.eval()
        ys = []
        y_preds = []
        for batch_idx, (X, y) in enumerate(dataloader):
            y_pred = model(X)
            ys.append(y)
            y_preds.append(y_pred)
        mape, mse, mae, r2 = compute_metrics(torch.cat(y_preds), torch.cat(ys), device)
        print(f'MAPE: {mape:.4f}, MSE: {mse:.4f}, MAE: {mae:.4f}, R2: {r2:.4f}')
</code></pre>
<p>The function takes in the trained model, the dataset and the device and is supposed to print the MAPE, MSE, MAE and R2 metrics.<br>
However I noticed that if I leave the <code>shuffle</code> argument in <code>DataLoader</code> as <code>true</code> it will give me different output each time on the same model and data. Why could that be? I am accumulating all the predictions and computing the metric on all of them so the order shouldn’t matter.<br>
Example output I am getting by calling the function few times with the same data:</p>
<pre><code class="lang-auto">MAPE: 1.0839, MSE: 0.4303, MAE: 0.4552, R2: 0.3663
MAPE: 1.0192, MSE: 0.4918, MAE: 0.4661, R2: 0.3544
MAPE: 0.9897, MSE: 0.4885, MAE: 0.4701, R2: 0.3661
</code></pre>
<p>Dataloader has 145 batches and even if I am dropping the last (not full one) it shouldn’t have such a strong effect I think. Is there some bug in this code?</p>

ptrblck: <aside class="quote no-group" data-username="The_Snail" data-post="1" data-topic="173831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_snail/48/44803_2.png" class="avatar"> The_Snail:</div>
<blockquote>
<p>Dataloader has 145 batches and even if I am dropping the last (not full one) it shouldn’t have such a strong effect I think.</p>
</blockquote>
</aside>
<p>Could you keep the last batch and check how large the difference would be?</p>

The_Snail: <p>Thanks, that helped, I guess the variation really was due to changing samples, and I didn’t realize that during inference it doesn’t make sense anymore to drop the last batch, so that solved the problem.</p>
Applying quantization for StreamingASR code in pytorch mobile

BillyGun27: <p>Does anyone ever try to apply quantization for StreamingASR in android demo before?<br>
Could someone provide a guidance on what to do?<br>
I am not sure why this error occured, is the model just not supported yet?</p>
<p>I am trying to apply dynamic quantization for Emformer-RNNT-based Model</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/android-demo-app/blob/master/StreamingASR/generate_ts.py">
  <header class="source">

      <a href="https://github.com/pytorch/android-demo-app/blob/master/StreamingASR/generate_ts.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/android-demo-app/blob/master/StreamingASR/generate_ts.py" target="_blank" rel="noopener nofollow ugc">pytorch/android-demo-app/blob/master/StreamingASR/generate_ts.py</a></h4>


      <pre><code class="lang-py">from typing import Dict, List, Optional, Tuple
import json
import math

from fairseq.data import Dictionary
import torch
import torchaudio
from torchaudio.pipelines import EMFORMER_RNNT_BASE_LIBRISPEECH
from torchaudio.models import Hypothesis


def get_hypo_tokens(hypo: Hypothesis) -&gt; List[int]:
    return hypo[0]


def get_hypo_score(hypo: Hypothesis) -&gt; float:
    return hypo[3]


def to_string(input: List[int], tgt_dict: List[str], bos_idx: int = 0, eos_idx: int = 2, separator: str = "",) -&gt; str:
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/android-demo-app/blob/master/StreamingASR/generate_ts.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>this is the quantization code that I use</p>
<pre><code class="lang-auto">model_pquant = torch.quantization.quantize_dynamic(
    wrapper,  # the original model
    {torch.nn.LSTM, torch.nn.Linear},  # a set of layers to dynamically quantize
    dtype=torch.qint8)  # the target dtype for quantized weights

torch.save(model_pquant,'quant_dynamic_Emformer-RNNT_pure')
</code></pre>
<p>during inference, it produce this error</p>
<pre><code class="lang-auto">RuntimeError: In ChooseQuantizationParams, min should be less than or equal to max
</code></pre>

BillyGun27: <p>after further analyzing, the error is caused because the input tensor is wrong. Since I was trying to replace pyaudio.PyAudio() streaming with audio file.</p>
<p>So my error is because the input.</p>
<p>I unable to replace streaming with audio file</p>
Apnea-ecg database process

goodboy-hub: <p>I met a problem when I started to process the apnea-ecg dataset released in 2000. code is shown as follows. If the utf-8 is wrong, how could i know what it is. I have tried utf-16,<br>
GBK, wrong either.</p>

goodboy-hub: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac6a6074b847b1e8eec329f2e70ac01bf3f03c29.png" data-download-href="https://discuss.pytorch.org/uploads/default/ac6a6074b847b1e8eec329f2e70ac01bf3f03c29" title="屏幕截图 2023-03-02 085619"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac6a6074b847b1e8eec329f2e70ac01bf3f03c29_2_690x269.png" alt="屏幕截图 2023-03-02 085619" data-base62-sha1="oBg2eQgIeQVVuVadXiN73qI1f97" width="690" height="269" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac6a6074b847b1e8eec329f2e70ac01bf3f03c29_2_690x269.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac6a6074b847b1e8eec329f2e70ac01bf3f03c29_2_1035x403.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac6a6074b847b1e8eec329f2e70ac01bf3f03c29.png 2x" data-dominant-color="0A0909"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">屏幕截图 2023-03-02 085619</span><span class="informations">1294×505 56.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

goodboy-hub: <p>i solved my problem. use wfdb package</p>
Why does torch bmm exist?

learning_experiments: <p>Given there is torch.matmul, which can also do a 3d matrix multiplication, why does torch.bmm exist?</p>

ptrblck: <p><code>torch..matmul</code> would be the more flexible op allowing for inputs using different shapes as described in the <a href="https://pytorch.org/docs/stable/generated/torch.matmul.html#torch.matmul">docs</a> and would broadcast the tensors if needed.<br>
<code>torch.bmm</code> expects two 3D tensors and will not broadcast them as noted in the <a href="https://pytorch.org/docs/stable/generated/torch.bmm.html#torch.bmm">docs</a>. I would claim standard use cases should call into <code>torch.matmul</code> and benefit from the flexibility.</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x20184 and 83544x85)

Zelreedy: <p>Hello. I am working on a project to develop a CNN model for image classification. It is required to flatten the image after the convolutional layers, in order to pass it to the fully connected layers. I am using the following formula to calculate the required input shape to the fully connected layers:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/1576a3eda8e0cfda926062be441213206d0698f4.png" alt="image" data-base62-sha1="33SdbpDTKKrfjWk3R3S7GQgYtog" width="208" height="74"></p>
<p>Where W is the width of the output image, F is the kernel/filter size, P represents the padding, and S is the stride. My original image size is 119, and my convolutional layers are as follows:</p>
<ol>
<li>Conv Layer (W=119, F=5, P=2, S=1) (Size = 119)</li>
<li>Max Pooling Layer(F=2) (Size = 59 - Correct me if I’m wrong here)</li>
<li>Conv Layer(W=59, F=5, P=2, S=1) (Size = 59)</li>
</ol>
<p>Going through the above equation, the size of the output image after going through all the layers should be 59*59. Also note that the output channels from the 2nd Conv Layer are 24, so the input vector to the fully connected layer should be 24x59x59 = 83544. However, the actual output, from the error, seems to be 24x29x29=20184. Attached are my convolutional layers:</p>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=5, stride=1, padding=2)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv2 = nn.Conv2d(in_channels=12, out_channels=24, kernel_size=5, stride=1, padding=2)
        self.drop1 = nn.Dropout(p=0.5)
        self.fc1 = nn.Linear(24*59*59, params['n_unit'])
        self.drop2 = nn.Dropout(p=0.1)
        self.fc2 = nn.Linear(params['n_unit'],num_classes)
</code></pre>
<p>Where did I go wrong in the calculations?</p>

ptrblck: <p>Did you apply the pooling layer twice and could you post the <code>forward</code> method, please?</p>

Zelreedy: <p>Thank you! I actually realized I was implementing pooling twice in the forward method, indeed.</p>
Why does `torch.nn.functional.cross_entropy` use the natural logarithm instead of base 2?

rohil: <p>Hi there,</p>
<p>There have been <a href="https://discuss.pytorch.org/t/manual-calculation-of-binary-cross-entropy-with-logits/61602/2">questions in the past</a> that reveal that under the hood, the <code>cross_entropy</code> calculation uses the natural log rather than <code>log_2</code>. I’m curious if anyone knows why that choice was made?</p>
<p>On the one hand, the loss distributions are still similarly shaped, although they are now scaled slightly differently. On the other hand, using base 2 fits more with the information-theoretic definition of cross-entropy.</p>
<ul>
<li>Was there a specific reason the natural log is used?</li>
<li>Is it computationally more efficient, or simpler?</li>
<li>Does it matter at all in anyone’s practical experience?</li>
</ul>

KFrank: <p>Hi Rohil!</p>
<aside class="quote no-group" data-username="rohil" data-post="1" data-topic="173753" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ecccb3/48.png" class="avatar"> rohil:</div>
<blockquote>
<p>they are now scaled slightly differently.</p>
</blockquote>
</aside>
<p>That is indeed the only difference.  To get the log-2 version of cross entropy<br>
from pytorch’s cross entropy, multiply by <code>1 / log (2)</code>.  This has almost no<br>
effect.  For example, with plain-vanilla <code>SGD</code> as your optimizer, this rescaling<br>
can be absorbed into the learning rate.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>Was there a specific reason the natural log is used?</li>
</ul>
</blockquote>
</aside>
<p>My guess would be because <code>log()</code> (i.e., “natural log”) is the most “standard”<br>
version of log.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>Is it computationally more efficient, or simpler?</li>
</ul>
</blockquote>
</aside>
<p>No.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>Does it matter at all in anyone’s practical experience?</li>
</ul>
</blockquote>
</aside>
<p>It really doesn’t matter in the least.  It’s just a modest rescaling by a factor<br>
that is pretty close to one.</p>
<p>Best.</p>
<p>K. Frank</p>
Batch-wise Matrix-Free LSQR

FrankiePoon: <p>I am trying to solve a least square problem for a linear system Ax - b, where x is a 1D vector , b is also a 1D vector that contains two sub-vectors b1 and b2, and A contains two operators: the identity map I and the differentiation operator D. This can be written as</p>
<pre><code class="lang-auto">A = torch.cat([I,  D]), b = torch.cat([b1, b2])
</code></pre>
<p>where</p>
<pre><code class="lang-auto">x = b1, D(x) = b2
</code></pre>
<p>The operator D is written as a function that takes the 1st order derivative of the signal x. The scipy function</p>
<pre><code class="lang-auto">scipy.sparse.linalg.lsqr
</code></pre>
<p>with first argument a function can solve the above problem matrix-free by the linear operator</p>
<pre><code class="lang-auto">scipy.sparse.linalg.LinearOperator
</code></pre>
<p>I am wondering whether there are similar features for</p>
<pre><code class="lang-auto">torch.linalg.lstsq
```.
Or is there any alternative way to achieve this using pytorch?</code></pre>
Statically linking custom ops in C++

Amit_Gupta: <p>Hi,<br>
I am using <a href="https://github.com/rusty1s/pytorch_scatter" rel="noopener nofollow ugc">torch_scatter</a> and <a href="https://github.com/rusty1s/pytorch_sparse" rel="noopener nofollow ugc">torch_sparse</a> in my project. Currently they are compiled as shared library and linked against my program for runtime custom ops.</p>
<p>I would like to include the torch_sparse and torch_scatter in my program to reduce runtime dependency. I managed to compile it, but at runtime I get an error:</p>
<blockquote>
<p>Could not find any similar ops to torch_sparse::ptr2ind. This op may not exist or may not be currently supported in TorchScript.</p>
</blockquote>
<p>See discussion <a href="https://github.com/pyg-team/pytorch_geometric/discussions/6811" rel="noopener nofollow ugc">here</a>.</p>
<p>How can I make torch read the custom ops from a static binary?</p>
Train EfficientNet-B1 with input size 512

Madh: <p>Hi,<br>
I am doing experiments with EfficientNet-B1 as backbone. Currently I have trained the model with input size of 240 × 240.</p>
<p>Now I want to try training with 512 × 512, can you please help me understand what will be effect of it on the model ?</p>

eqy: <p>You might want to check out popular repositories of models e.g., <a href="https://github.com/huggingface/pytorch-image-models" rel="noopener nofollow ugc">huggingface/pytorch-image-models: PyTorch image models, scripts, pretrained weights – ResNet, ResNeXT, EfficientNet, EfficientNetV2, NFNet, Vision Transformer, MixNet, MobileNet-V3/V2, RegNet, DPN, CSPNet, and more (github.com)</a> which would have example results on how models perform when scaled up in terms of resolution. It would ultimately depend on your specific task, but generally the model should have somewhat higher accuracy (provided that the test-time resolution is also scaled up appropriately <a href="https://arxiv.org/abs/1906.06423" rel="noopener nofollow ugc">[1906.06423] Fixing the train-test resolution discrepancy (arxiv.org)</a>), at roughly (512/240)^2 times the original compute cost.</p>
Model doesn't register parameters if done through dict

mkserge: <p>Hi,</p>
<p>I can’t figure out how to make the model register my params if they are initialized as values in a dict.</p>
<p>Example code,</p>
<pre><code class="lang-auto">import torch

class EmbeddingModel(torch.nn.Module):

    def __init__(self):
        super(EmbeddingModel, self).__init__()

        self.embeddings = {
            'source': None,
            'target': None
        }

        self.init_embedding("source", 10, 5)
        self.init_embedding("target", 10, 4)

    def init_embedding(self, emb_type, num_items, emb_dim):
        self.embeddings[emb_type] = torch.nn.Embedding(
            num_embeddings=num_items, embedding_dim=emb_dim)

    def forward(self, src_idx):
        pass


model = EmbeddingModel()

for param in model.parameters():
    print(param) # &lt;- No parameters in the model.
</code></pre>
<p>Is there a way to do this cleanly?</p>

ptrblck: <p>That’s expected and you would need to use <code>nn.ModuleDict</code> instead of a plain Python <code>dict</code>.</p>

mkserge: <p>I knew it’s gonna be you! <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>
<p>Didn’t know about <code>ModuleDict</code>. That looks like exactly what I need. Thanks Patrick!</p>
Windows c++ compile errors

Wesley_Neill: <p>I’m trying to debug some custom extensions.</p>
<p>when using cl to build in vscode, I get the following errors from <code>Optional.h</code>:</p>
<p>‘c10::constexpr_storage_t’: destructor was implicitly defined as deleted</p>
<p>and</p>
<p>‘c10::trivially_copyable_optimization_optional_base’: destructor was implicitly defined as deleted</p>
<p>Does anyone have any idea how to overcome this?</p>
Sampler with set indices

Roos_Kraaijveld: <p>I have a specific order that I would like to feed into my data loader. I already made a list of indices for each batch.</p>
<p>Which sampler on PyTorch can I use to do this?</p>

ptrblck: <p>I’m not sure if you plan to pass the indices of the entire batch to the <code>Dataset.__getitem__</code> and would like to load all samples for this batch already (instead of a single sample which would be the standard use case), but if so then creating a custom <code>Sampler</code> might work as seen here:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.arange(20).view(-1, 1).float()
        
    def __getitem__(self, index):
        print("received indices {}".format(index))
        x = self.data[index]
        return x
    
    def __len__(self):
        return len(self.data)
     

class MySampler(torch.utils.data.sampler.Sampler):
    def __init__(self, indices):
        self.indices = indices
        
    def __iter__(self):
        return iter(self.indices)


# create indices for each batch
indices = [torch.arange(5, 10),
           torch.arange(0, 5),
           torch.arange(15, 20),
           torch.arange(10, 15),
]

sampler = MySampler(indices)
dataset = MyDataset()   
loader = DataLoader(dataset, sampler=sampler)
for data in loader:
    print(data)
</code></pre>
<p>Output:</p>
<pre><code class="lang-python"># received indices tensor([5, 6, 7, 8, 9])
# tensor([[[5.],
#          [6.],
#          [7.],
#          [8.],
#          [9.]]])
# received indices tensor([0, 1, 2, 3, 4])
# tensor([[[0.],
#          [1.],
#          [2.],
#          [3.],
#          [4.]]])
# received indices tensor([15, 16, 17, 18, 19])
# tensor([[[15.],
#          [16.],
#          [17.],
#          [18.],
#          [19.]]])
# received indices tensor([10, 11, 12, 13, 14])
# tensor([[[10.],
#          [11.],
#          [12.],
#          [13.],
#          [14.]]])
</code></pre>
<p>Is this what you are looking for?</p>
Orthogonal Parametrizations Error

Qingyu_Song: <p>Hi,<br>
I’m trying to use <code>torch.nn.utils.parametrizations.orthogonal</code> to achieve unitary weights in my self-defined module on PyTorch version 1.10.2. But it always put out an error that:</p>
<pre><code class="lang-auto">Module 'DimTransModel(
  (increase_mlp): Linear(in_features=8, out_features=16, bias=False)
  (decrease_mlp): Linear(in_features=8, out_features=16, bias=False)
)' does not have a parameter, a buffer, or a parametrized element with name 'weight'
</code></pre>
<p>I’m not sure whether it is a bug of PyTorch itself.<br>
This is my implementation of the module:</p>
<pre><code class="lang-python">class DimTransModel(nn.Module):
    weight = torch.Tensor

    def __init__(self, num_antenna, num_output):
        super(DimTransModel, self).__init__()
        self.increase_mlp = nn.Linear(num_antenna, num_output,
                                      bias=False).to(torch.complex64)
        self.decrease_mlp = nn.Linear(num_antenna, num_output,
                                      bias=False).to(torch.complex64)
        self.weight = nn.Parameter(torch.Tensor(num_antenna, num_output)).to(
            torch.complex64)

    def increase_dim(self, data):
        self.increase_mlp.weight = self.weight
        return self.increase_mlp(data)

    def decrease_dim(self, data):
        self.decrease_mlp.weight = self.weight.conj().transpose()
        return self.decrease_mlp(data)
</code></pre>
<p>And this is the function call:</p>
<pre><code class="lang-python">self.dim_trans_model = nn.utils.parametrizations.orthogonal(
            DimTransModel(num_antenna, num_antenna * 2), "weight")
</code></pre>

Qingyu_Song: <p>Sorry. I found that there is a typo in the property definition in class. It should be <code>:</code> but not <code>=</code>. Fixed and no error.</p>
How to effectively construct a mask tensor?

ruka: <p>Suppose I have a tensor indicating which column should be 1 for each row, for example,</p>
<blockquote>
<p>index = torch.tensor([3,1,0,0,2])</p>
</blockquote>
<p>and I would like to construct a mask tensor from above one and get this result:</p>
<blockquote>
<p>mask = torch.tensor([[0,0,0,1,0],<br>
[0,1,0,0,0],<br>
[1,0,0,0,0],<br>
[1,0,0,0,0],<br>
[0,0,1,0,0]])</p>
</blockquote>
<p>Is there any pytorch api can help me finish this job? Or maybe with several apis?</p>

ptrblck: <p><code>scatter_</code> should work:</p>
<pre><code class="lang-python">index = torch.tensor([3,1,0,0,2])
torch.zeros(index.size(0), 5).scatter_(1, index.unsqueeze(1), 1.)
# tensor([[0., 0., 0., 1., 0.],
#         [0., 1., 0., 0., 0.],
#         [1., 0., 0., 0., 0.],
#         [1., 0., 0., 0., 0.],
#         [0., 0., 1., 0., 0.]])
</code></pre>
<p>I don’t know how the mask shape was defined in <code>dim1</code> so I just used <code>5</code> as seen in your example.</p>

ruka: <p>Wow, this works perfectly! Thank you!</p>
Why does `torch.Tensor` not have a `device` argument

Kallinteris-Andreas: <p>This fails:</p>
<pre><code class="lang-python">&gt;&gt;&gt; torch.Tensor(2,3, device="cuda")
RuntimeError: legacy constructor expects device type: cpu but device type: cuda was passed
</code></pre>
<p>A solution, I can find, but it is not elegant and creates the tensor on CPU then moves it to GPU</p>
<pre><code class="lang-python">&gt;&gt;&gt; torch.tensor(torch.Tensor(2,3), device="cuda")
tensor([[-7.6854e-18,  3.0808e-41, -1.5538e-01],
        [ 4.5572e-41,  8.9683e-44,  0.0000e+00]], device='cuda:0')
</code></pre>
<p>Is there a way to create a tensor on cuda inplace?</p>
<p>Thanks!</p>

eqy: <p>It looks like you are specifying a shape and not providing data for initialization, so what you are doing would be equivalent to</p>
<pre><code class="lang-auto">torch.empty(2, 3, device='cuda')
</code></pre>
Copy tensor from cuda to cpu used long time

chao_zhao: <p>When i test model infer by libtorch. I found cp tensor(size is [600]) from cuda to cpu always used long time. Like codes:</p>
<pre><code class="lang-auto">int textcnn_simcse_test() {
  // load model
  torch::jit::Module model = torch::jit::load(
      "../textcnn_simcse_qqsearch_jit.pt", torch::Device({torch::kCUDA, 0}));
  model.eval();
  
  while (true) {
    torch::Tensor tensor = torch::rand({601, 50});
    tensor = tensor.to(torch::kInt64).to(torch::kCUDA);
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.emplace_back(tensor);
    torch::Tensor out = model(inputs).toTensor();
    
    auto begin = std::chrono::duration_cast&lt;std::chrono::microseconds&gt;(
                     std::chrono::system_clock::now().time_since_epoch())
                     .count();
    torch::Tensor query = out.index({torch::indexing::Slice(0, 1)});
    torch::Tensor titles =
        out.index({torch::indexing::Slice(1, torch::indexing::None)});
    torch::Tensor result = torch::sum(query * titles, 1);
    auto calc_end = std::chrono::duration_cast&lt;std::chrono::microseconds&gt;(
                        std::chrono::system_clock::now().time_since_epoch())
                        .count();
    
    // Copy tensor from cuda to cpu.
    result = result.to(torch::kCPU);
    auto cp_end = std::chrono::duration_cast&lt;std::chrono::microseconds&gt;(
                      std::chrono::system_clock::now().time_since_epoch())
                      .count();
    
    std::cout &lt;&lt; "calc: " &lt;&lt; calc_end - begin
              &lt;&lt; " us, cp: " &lt;&lt; cp_end - calc_end &lt;&lt; " us" &lt;&lt; std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
  }
  return 0;
}
</code></pre>
<p>Log like:</p>
<pre><code class="lang-auto">calc: 84 us, cp: 5616 us
calc: 83 us, cp: 5600 us
calc: 82 us, cp: 5587 us
calc: 82 us, cp: 5606 us
</code></pre>
<p>I don’t know why. Is there some wrong in my codes?</p>

ptrblck: <p>CUDA operations are executed asynchronously and you would need to synchronize the code before starting and stopping host timers. Otherwise you would profile the dispatching, kernel launches etc. unless you are implicitly synchronizing the code e.g. by moving data to the host.</p>
Pytorch is unable to access GPU in anaconda

hahaznewbie: <p>These are the steps I have done on my windows 10 desktop<br>
I installed CUDA 11.3 from <a href="https://developer.nvidia.com/cuda-11.3.0-download-archive?target_os=Windows&amp;target_arch=x86_64&amp;target_version=10&amp;target_type=exe_network" rel="noopener nofollow ugc">here</a></p>
<p>I have also installed cuDNN v8.3.2 from <a href="https://developer.nvidia.com/rdp/cudnn-archive" rel="noopener nofollow ugc">here</a><br>
To be more precise, I downloaded a zipped file and copy pasted the contents to my \CUDA\v11.3\bin , \CUDA\v11.3\include and \CUDA\v11.3\lib folders respectively as per the installation documentations</p>
<p>I have created an environment in anaconda have run the command that can be found<br>
<a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">INSTALLING PREVIOUS VERSIONS OF PYTORCH</a></p>
<pre><code class="lang-auto">conda install pytorch==1.9.0 torchvision==0.10.0 torchaudio==0.9.0 cudatoolkit=11.3 -c pytorch -c conda-forge
</code></pre>
<p>This is a screenshot containing the list of packages after running conda list command.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/227e2394e9e803650849f71417a473c5b33c1046.png" alt="image" data-base62-sha1="4V8u6p54MHSdyuxpbNUBCIgkrci" width="639" height="259"></p>
<p>It shows that the pytorch installed is “cpuonly”. What are commands or steps am I missing to enable the gpu when using pytorch ? I wish to utilise pytorch v1.9.0 and torchvision v0.10.0 with GPU.</p>
<p>I am not looking to install the bleeding edge PyTorch code but rather a workable version used for introductory deep learning.</p>
<p>Thank you in advance!</p>
Can I use hooks to check how tensors move between layers

vdw: <p>I’ve tried my hands on hooks, and get the basic idea how to add hooks (I’m using forward hooks) to each layer/module. This allows me, e.g., to print the input and output shapes of the tensor coming and and going out of the layer.</p>
<p>Is there also a way to tell from which module a subsequent module got its input?</p>

ptrblck: <p>I’m not sure if there would be any proper way to determine the previously used module using hooks as even checking the <code>.grad_fn</code> wouldn’t be sufficient as some layers would return the “lower level” operation and not their layer name:</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(3)
x = torch.randn(1, 3, 24, 24)
out = bn(x)
print(out.grad_fn)
# &lt;NativeBatchNormBackward0 object at 0x7fa1d8612730&gt;

lin = nn.Linear(10, 10)
x = torch.randn(1, 10)
out = lin(x)
print(out.grad_fn)
# &lt;AddmmBackward0 object at 0x7fa1d86d6a30&gt;

lin = nn.Linear(10, 10, bias=False)
x = torch.randn(1, 10)
out = lin(x)
print(out.grad_fn)
# &lt;MmBackward0 object at 0x7fa1d8667610&gt;
</code></pre>
<p>Your use case comes close to tracing the actual operations and <a class="mention" href="/u/alband">@albanD</a> shared a few code snippets using the <code>__torch_dispatch__</code> mechanism <a href="https://github.com/albanD/subclass_zoo">here</a>.<br>
<a href="https://github.com/albanD/subclass_zoo/blob/9a699abfece0bd90037558b10049f7ac89e37582/tracer_tensor.py">This tracing tensor</a> code would still record the actual ops, but maybe could be manipulated to record module types instead.<br>
Let me know if this approach would even work or if you depend on hooks.</p>

vdw: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks! I thought this would be relatively easy to do since I stumbled upon various <a href="https://stackoverflow.com/questions/52468956/how-do-i-visualize-a-net-in-pytorch" rel="noopener nofollow ugc">visualization methods</a>. I assume they need to know which module feeds into which to draw the edges between modules.</p>
<p>Of course, I don’t even know if those visualization packages utilize hooks or do something else. Hooks where just my first guess. I will need to check the links you posted.</p>

ptrblck: <p>I believe libraries visualizing the computation graph would also trace or script the model internally (but I might also be wrong).<br>
<a class="mention" href="/u/tom">@tom</a> created <a href="https://github.com/t-vi/pytorch-tvmisc/blob/master/visualize/visualize-jit-models-interactive.ipynb">this great notebook</a> a while ago which does exactly this.<br>
Based in the code the model is also traced and I assume this is a requirement.</p>
Runtime Error related to shared memory

yushu: <p>Hi,</p>
<p>i got an error might relate to dataloader and shared memory, the error below would pop up randomly.</p>
<p><code>RuntimeError: falseINTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/MapAllocator.cpp:263, please report a bug to PyTorch. unable to open shared memory object &lt;/torch_43891_16&gt; in read-write mode</code></p>
<p>I am using DDP <code>torch.distributed.run</code> to train a model on imagenet dataset with a multi-node multi-gpu setup. i’ve tried many possible solution such as setting <code>ulimit -n &lt;max value&gt;</code>, or using the command <code>torch.multiprocessing.set_sharing_strategy('file_system')</code>, but neither doesn’t work.</p>
<p>Each node has <code>AMD CPU with 64 processors</code> + <code>2 A100 GPUs</code> + <code>250G RAM</code></p>
<p>I previously set <code>num_workers=24</code> in data loader since the node has 64 processors in total but still failed</p>
<p>In the latest attemption i set <code>num_workers=12</code> and <code>ulimit -n &lt;max value&gt;</code>, it works for now but the code is still running and i cannot make sure the error won’t pop up later.</p>
<p>May i ask would the <code>num_workers</code> related to this error? and also what would be a proper way to choose the <code>num_workers</code> if it is the case.</p>
<p>Thanks!</p>

yushu: <p>The code worked only once when setting <code>export OMP_NUM_THREADS=32</code> and <code>num_workers=12</code> in two nodes training. If I increase the nodes to 4 it fails again.</p>

wanchaol: <p>Thanks for posting the question <a class="mention" href="/u/yushu">@yushu</a>. There’re someone else who posted similar problem before, do you think this might help you resolve the issue? <a href="https://discuss.pytorch.org/t/how-to-cache-an-entire-dataset-in-multiprocessing/133842" class="inline-onebox">How to cache an entire dataset in multiprocessing?</a></p>

yushu: <p>Thanks for your advice. I’ve looked the link you refer, unfortunately it seems not related to my issue.<br>
Current i assume it is related to the PyTorch version. i previously worked on PyTorch 1.10+cuda11.3 on miniconda. Now I switch back to PyTorch 1.9+cuda11, it seems no such error. It is a work around but still can debug why it happens before.</p>

Benjamin_Therien: <pre><code class="lang-auto">RuntimeError: falseINTERNAL ASSERT FAILED at 
"/opt/conda/conda-bld/pytorch_1639180588308/work
/aten/src/ATen/MapAllocator.cpp":323, │please report 
a bug to PyTorch. unable to mmap 8 bytes from file 
&lt;filename not specified&gt;: Cannot allocate memory (12)
</code></pre>
<p>I get a similar error to you with torch 1.10.1 and cuda 11.3. I’m using torch.distributed.dataparallel to train on a 2 GPU node. The bug only occurs when setting <code>pin_memory=True</code> in the dataloader. However, setting it to false is not a great solution as my gpus are under-utilized without <code>pin_memory=True</code>.</p>
What version of cuDNN is supported by pytorch 1.9.0

hahaznewbie: <p>I wish to work with pytorch v1.9.0. I have already downloaded CUDA 11.3 as well as visual studios 2019. I am unsure of which version of cuDNN would allow me to utilise pytorch in anaconda.</p>

ptrblck: <p>I assume you are interested in installing a binary for this old PyTorch release? If so, then note that the PyTorch <code>1.9</code> binaries were built with CUDA 10.2, which shipped with cuDNN 7.6.5, and CUDA 11.3, which used cuDNN 8.3.2 and you can install this binary using the supported commands from <a href="https://pytorch.org/get-started/previous-versions/">here</a>.<br>
In case you want to build PyTorch from source with your local CUDA toolkit and cuDNN, <code>1.9.0</code> should have supported CUDA 11.4 in source builds as it was released in Sept. 2021 while CUDA 11.4 was published in July 2021.</p>

hahaznewbie: <p>Thank you for your reply. Based on this website</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://developer.nvidia.com/rdp/cudnn-archive">
  <header class="source">
      <img src="https://developer.nvidia.com/sites/all/themes/devzone_base/favicon.ico" class="site-icon" width="48" height="48">

      <a href="https://developer.nvidia.com/rdp/cudnn-archive" target="_blank" rel="noopener nofollow ugc" title="12:35PM - 12 March 2015">NVIDIA Developer – 12 Mar 15</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://developer-blogs.nvidia.com/wp-content/uploads/2022/04/cuda-image-16x9-1.jpg" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://developer.nvidia.com/rdp/cudnn-archive" target="_blank" rel="noopener nofollow ugc">cuDNN Archive</a></h3>

  <p>Download releases from the GPU-accelerated primitive library for deep neural networks.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>, cuDNN v8.3.2 is for CUDA 11.5. Does that mean cuDNN v8.3.2 works for CUDA 11.3 as well?<br>
Hope you can clarify my misunderstanding</p>

ptrblck: <p>Yes, cuDNN would ship with their own CUDA runtime and should be compatible between minor CUDA 11.x versions.</p>

hahaznewbie: <p>Thank you for your help.</p>
RuntimeError: expected scalar type Double but found Float even though I convert to double

Ryan_Marr: <pre><code class="lang-auto">def encoder_block(inp, max_pool, in_channels):
    conv = torch.nn.Conv2d(in_channels=in_channels, out_channels=64, kernel_size=3, padding='same')(inp.double())
    relu = torch.nn.ReLU()(conv)
    conv = torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding='same')(relu)
    relu = torch.nn.ReLU()(conv)
    if max_pool:
        return torch.nn.MaxPool2d(2,2)(relu)
    return relu

test_load = nib.load(fpath).get_fdata()
test_numpy = test_load[:,:,0].reshape(1,1,256,256).astype('float64')
tens = torch.DoubleTensor(test_numpy)
out = encoder_block(tens, True, 1)
print(out)
</code></pre>
<p>This code should take a nifti file from my local storage, convert it to a numpy array then perform some convolutions on the 2d image as a basic test just for now.</p>
<p>The error happens on the first conv2d and says <code>RuntimeError: expected scalar type Double but found Float</code>. I’m not sure what else I can do to convert my data to float.</p>

ptrblck: <p>Your model is using <code>float32</code> parameters and buffers by default while you are explicitly casting the input to <code>float64</code>. Either cast the inputs to <code>float32</code> or the model to <code>float64</code>, if you really need this high precision and are fine with the performance penalty (especially on GPU).</p>
Pytorch does not support python 3.11

BlackEagle1983: <p>Hello . I must thank your team for developing and extending this package.<br>
but we have a problem in windows . pytorch does not support python 3.11 in windows.<br>
when will you publish python 3.11 supported version ?<br>
and why do you much delay after every python’s major release ?<br>
sorry if my questions annoyed your team .<br>
Thank you .<br>
Have a good day .</p>

ptrblck: <p>The nightly packages for PyTorch already support Python 3.11 as seen e.g. in the last build:<br>
<a href="https://download.pytorch.org/whl/nightly/cu118/torch-2.0.0.dev20230228%2Bcu118-cp311-cp311-win_amd64.whl">torch-2.0.0.dev20230228-cu118-cp311-cp311-win_amd64.whl</a></p>
<aside class="quote no-group" data-username="BlackEagle1983" data-post="1" data-topic="173747">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blackeagle1983/48/57829_2.png" class="avatar"> BlackEagle1983:</div>
<blockquote>
<p>and why do you much delay after every python’s major release ?</p>
</blockquote>
</aside>
<p>I don’t think the support is delayed, but higher priority tasks were done instead.<br>
If you want a faster support your contributions are more than welcome. <a href="https://github.com/pytorch/pytorch/issues/86566">This issue</a> tracks the status so please let me know if you are interested in working on any open issue.</p>
What's the meaning of size_ in torch/csrc/distributed/c10d/ProcessGroup.hpp?

big_old: <p>Hi, I am trying to use pytorch/tests/cpp/c10d/ProcessGroupNCCLTest.cpp in a 1 CPU , 8 GPUs linux machine for some tests. I found that in the Allgahter test, the size_ in ProcessGroup.hpp has a value of 1. As a rule of thumb, isn’t size_ the number of cuda devices 8? Why is it 1?</p>

ptrblck: <p><code>pg-&gt;getSize()</code> should return the world size, but I’m currently unsure how you are executing the NCCL tests and which part of the code you are checking.</p>

big_old: <p>I recompiled pytorch in the NGC image and commented out the other tests except for Allgather in the file pytorch/test/cpp/c10d/ProcessGroupNCCLTest.cpp. Also, I print size_ in ProcessGroupNCCL.cpp. Finally, I ran ProcessGroupNCCLTest, the compiled executable, and got size_ to be 1. I’m guessing that ProcessGroupNCCLTest generated the data on the CPU and then copied it to the GPU, so size_ prints out the number of cpus.</p>
Should parameters within optimizer groups be disjoint?

ataxias: <p>In all examples with optimizer parameter groups I have seen, people split parameters in disjoint groups (groups with no shared members), like this:</p>
<pre><code class="lang-auto">optim.SGD([
        {'params': model.base.parameters()},
        {'params': model.classifier.parameters(), 'lr': 1e-3}
          ], lr=1e-2, momentum=0.9)
</code></pre>
<p>However, one may consider additional groups, which dissect the set of all parameters with respect to some property, and overlap with other groups. For example, one may additionally wish to have a default weight decay value, and 0 weight decay for all biases and <code>LayerNorm</code> parameters. Would the following example, where the 3rd group contains parameters also contained in the 1st and 2nd groups, be valid?</p>
<pre><code class="lang-auto">optim.SGD([
        {'params': model.base.parameters()},
        {'params': model.classifier.parameters(), 'lr': 1e-3}
        {'params': [param for name, param in model.named_parameters() \
                         if any(nd in name for nd in ['bias', 'LayerNorm.weight'])], 'weight_decay': 0.0},
          ], lr=1e-2, momentum=0.9, weight_decay=0.01)
</code></pre>
<p>If so, does the order in which the groups (<code>dict</code> objects) appear in the list determine the final optimizer hyperparameters which will be used for the parameters shared among different groups?</p>

ptrblck: <p>No, that should not be possible and you would get an error:</p>
<pre><code class="lang-python">ValueError: some parameters appear in more than one parameter group
</code></pre>

ataxias: <p>I see, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> ! So the only way to implement this then would be to define 4 disjoint groups: 1) biases/LayerNorm in <code>model.base</code>, 2) non biases/LayerNorm in <code>model.base</code>, 3) biases/LayerNorm in <code>model.classifier</code>, 4) non biases/LayerNorm in <code>model.classifier</code>.</p>
<p>Any suggestions on how to use a scheduler to change the learning rate for groups 1-2 together, and 3-4 together?</p>

Rahul_S: <p>Can someone answer the above question please.</p>
<p>I have same situation, I have optimizer with different parameter group with different learning rate. Now can I know how to use the scheduler for different parameter group?</p>
<p>Thanks</p>

ptrblck: <p>The learning rate scheduler will apply its update rule to each parameter group as seen <a href="https://github.com/pytorch/pytorch/blob/ed1957dc1989417cb978d3070a4e3d20520674b4/torch/optim/lr_scheduler.py#L159-L164">here</a>.<br>
In case you want to use different schedulers for different parameter groups you might need to create separate optimizers and also separate schedulers instead.</p>
RandomSampler uses excessive amounts of memory for indices list when shuffling

leokraft: <p>Hello,</p>
<p>we’re currently trying to train a model on a large dataset (&gt;60m samples).<br>
However, when training we notice that our CPU memory is steadily increasing.</p>
<p>This also happens when we’re using a dummy Dataset i.e.</p>
<pre><code class="lang-auto">class DummyDataset(Dataset):

    def __init__(self):
        super().__init__()

    def __len__(self):
        return 60_000_000

    def __getitem__(self, idx: int):

        return torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0, 6.0]), torch.tensor([7.0])
</code></pre>
<p>We’re using <code>num_workers &gt; 0</code> and for each worker there seems to be an additional ~1.5 GB of RAM consumption.</p>
<p>Our assumption is, that this happens due to <code>shuffle=True</code> in our DataLoader.<br>
The RandomSampler in the DataLoader permutates the indices for sampling and converts them into a list of <code>int</code>s. (<a href="https://github.com/pytorch/pytorch/blob/71ad1005f66c9a53a2fe28d24b95c4e828aa944e/torch/utils/data/sampler.py#L132" rel="noopener nofollow ugc">here</a>)</p>
<p>For our example and on our machine (where <code>sys.getsizeof(some_int)==28</code>) the size of this list would then be around:</p>
<p><code>28B * 60 000 000 = 1.68GB</code></p>
<p>Which would be copied for each worker and explain the memory consumption.</p>
<p>Is there a solution or alternative to this that still enables shuffling of the dataset after each epoch?<br>
Furthermore, are there any best practices around shuffling of large datasets, that eventually also don’t fit into memory anymore?</p>
<p>Thank you for your help!</p>
<p>The same/similar problem was also mentioned in <a href="https://github.com/pytorch/pytorch/issues/50201" rel="noopener nofollow ugc">this issue</a>.</p>

ptrblck: <p>Could you add a comment in the linked issue with your use case so that the code owners could take another look at it, please?<br>
<a href="https://github.com/pytorch/pytorch/pull/50202">This PR</a> wanted to implement a fix but it seems it was abandoned and closed.</p>

leokraft: <p>Thank you for the fast reaction.<br>
I’ve commented on the issue.<br>
In the meanwhile, is there any workaround for shuffling large datasets?</p>

ptrblck: <p>You could of course revisit (or revive) the stale PR to check which approach was being worked on, but as a workaround maybe <code>random.sample</code> could be helpful as it should work on <code>range(N)</code> without actually creating the population. Note however, that duplicates could be drawn so I’m unsure if this would fit your use case.</p>
Load and use only part of model

ays: <p>I have a pretrained model (I’ll use a sample of an MLP shown below). I’m trying to load this pretrained model but use only layers 2 to 6 (i.e from max_pool to fc3). how can I go about that please?</p>
<pre><code class="lang-auto">pretrained_mlp = mlp.load_state_dict(torch.load(model_path))
new_model = #only layers 2 to 6 go pretrained_mlp
</code></pre>
<pre><code class="lang-auto">MLP(
  (conv2d): Conv2d(256, 16, kernel_size=(3, 3), stride=(1, 1))
  (max_pool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (fc1): Linear(in_features=29344, out_features=64, bias=True)
  (relu): ReLU()
  (fc2): Linear(in_features=64, out_features=64, bias=True)
  (fc3): Linear(in_features=64, out_features=2, bias=True)
  (mapping): Sequential(
    (0): Linear(in_features=3, out_features=16, bias=True)
    (1): ReLU()
    (2): Linear(in_features=16, out_features=16, bias=True)
    (3): ReLU()
    (4): Linear(in_features=16, out_features=16, bias=True)
    (5): ReLU()
    (6): Linear(in_features=16, out_features=16, bias=True)
  )
)

</code></pre>

ptrblck: <p>Assuming you want to use these layers from the pre-trained model:</p>
<pre><code class="lang-python">  (max_pool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (fc1): Linear(in_features=29344, out_features=64, bias=True)
  (relu): ReLU()
  (fc2): Linear(in_features=64, out_features=64, bias=True)
  (fc3): Linear(in_features=64, out_features=2, bias=True)
</code></pre>
<p>I think the cleanest approach would be to write a custom <code>nn.Module</code> and pass these layers to this model or to re-wrap them into e.g. a new <code>nn.Sequential</code> container. Note that the latter approach would need some additional changes as the functional API calls from the original model’s <code>forward</code> method would be missing. In particular I expect a flattening operation in the <code>forward</code> was used (e.g. via <code>x = x.view(x.size(0), -1)</code>) between the <code>MaxPool2d</code> layer and the first <code>Linear</code> layer.</p>

ays: <p>How would I go about wrapping it into a new nn.sequential container? Because I tried doing below but I get an error</p>
<pre><code class="lang-auto">new_model = torch.nn.Sequential(*(list(pretrained_mlp.children()))[2:7])
</code></pre>

ptrblck: <p>You would need to check the <code>forward</code> implementation and could then explicitly create the <code>nn.Sequential</code> container e.g. via:</p>
<pre><code class="lang-python">model = nn.Sequential(
    model.max_pool,
    nn.Flatten(),
    model.fc1,
    model.relu,
    model.fc2,
    model.relu,
    model.fc3
)
</code></pre>
<p>Your approach should also work assuming that <code>children()</code> returns all modules, which are indeed applied sequentially in your original <code>forward</code> method.<br>
As previously mentioned, at least the flattening operation seems to be missing, so I guess this is what might be causing the error you are seeing.</p>