akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Histogram Observer Implementation Error

diviramon: <p>It seems to me that the <code>HistogramObserver</code> implementation has a bug. In <a href="https://github.com/pytorch/pytorch/blob/e85dbcc9b075961ab082975348c5cf1d99b7da76/torch/ao/quantization/observer.py#L1181" rel="noopener nofollow ugc">L 1181</a> the forward pass is as follows:</p>
<pre><code class="lang-auto">combined_histogram = self._combine_histograms(
    combined_histogram,
    self.histogram,
    self.upsample_rate,
    downsample_rate,
    start_idx,
    self.bins,
)
</code></pre>
<p>Where <code>_combine_histograms</code> is defined as:</p>
<pre><code class="lang-auto">def _combine_histograms(
    self,
    orig_hist: torch.Tensor,
    new_hist: torch.Tensor,
    upsample_rate: int,
    downsample_rate: int,
    start_idx: int,
    Nbins: int,
) -&gt; torch.Tensor:
</code></pre>
<p>Therefore, <code> combined_histogram</code> gets mapped to <code>orig_hist</code> while <code>self.histogram</code> gets mapped to <code>new_hist</code>. It seems to me like it should be the other way around given the variable names.</p>
<p>Can someone clarify?</p>
<p>Thank you!!</p>

ddang: <p>will address your question in your github issue: <a href="https://github.com/pytorch/pytorch/issues/87126" class="inline-onebox" rel="noopener nofollow ugc">Bug in Histogram Observer Implementation · Issue #87126 · pytorch/pytorch · GitHub</a></p>
Pca_lowrank raises data type error even when converting argument to right data type

Mgeyer: <p>Hi,</p>
<p>I’m using torch.pca_lowrank, and am getting this error</p>
<pre><code class="lang-auto">*Traceback (most recent call last):*
*File “&lt;string&gt;“, line 1, in &lt;module&gt;*
*File “* /.conda/envs/ldm_clone/lib/python3.8/site-packages/torch/_lowrank.py*”, line 281, in pca_lowrank*
*return _svd_lowrank(A - C, q, niter=niter, M=None)*
*File “* /.conda/envs/ldm_clone/lib/python3.8/site-packages/torch/_lowrank.py*”, line 163, in _svd_lowrank*
*Q = get_approximate_basis(A, q, niter=niter, M=M)*
*File “* /.conda/envs/ldm_clone/lib/python3.8/site-packages/torch/_lowrank.py*”, line 71, in get_approximate_basis*
*Q = torch.linalg.qr(matmul(A, R)).Q*
*RuntimeError: “geqrf_cuda” not implemented for ‘Half’*
</code></pre>
<p>I tried passing the data matrix argument as</p>
<pre><code class="lang-auto">torch.pca_lowrank(A.float(), q=None, center=True, niter=2)
</code></pre>
<p>also tried doing it outside the arguments A=A.float()</p>
<p>can’t understand why this is happening…</p>
Wrong number of frames returned by VideoClip

mllera: <p>I’ve been trying to replicate the results from <a href="https://arxiv.org/pdf/2103.15691.pdf" rel="noopener nofollow ugc">ViViT</a> in which you are required to sample 32 frames from videos in the Kinetics dataset. However I have been struggling with a weird issue. Basically, the VideoClip class returns one extra frame that what I require. A similar issue was referenced in <a href="https://github.com/pytorch/vision/issues/1884" rel="noopener nofollow ugc">issue #1884</a>, but there it returned less than expected.</p>
<p>I have checked and the videos don’t seem to be corrupted. Also it seems to happen to every video, thought I haven’t checked all of them (there’s quite a few).</p>
<p>I know this is probably an issue for Github and I have commented there, but it seems like the video api is not as widely maintained as other parts of the ecosystem and I figured people here are more active.</p>
Get layers in order of the data flow from input to output

rahan: <p>Hi -</p>
<p>Is there a way to get the layers in the order of the data flow? I need to change the size of the Conv2d input and output channels one by one, which means the number of input channels should be set to the number of the output channels from the previous convolution. I tried model.children() but it doesn’t work.</p>

sio277: <p>You can use forward hook in this way:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class Foo(nn.Module):
    def __init__(self):
        super(Foo, self).__init__()
        self.m1 = nn.Conv2d(1, 2, 3)
        self.m2 = nn.BatchNorm2d(2)
        self.m3 = nn.ReLU()
        self.m4 = nn.Conv2d(2, 3, 3)
        
    def forward(self, x):
        x = self.m1(x)
        x = self.m2(x)
        x = self.m3(x)
        x = self.m4(x)
        
        return x


modules = []
def add_hook(m):
    def forward_hook(module, input, output):
        modules.append(module)
    m.register_forward_hook(forward_hook)


foo = Foo()
foo.apply(add_hook)  # function `add_hook` is applied to the every submodule including self.

input = torch.rand(1, 1, 10, 10)
foo(input)  # hooks are fired sequentially from model input to the output
print(modules)
</code></pre>
<p>which prints out:</p>
<pre><code class="lang-auto">[Conv2d(1, 2, kernel_size=(3, 3), stride=(1, 1)), 
 BatchNorm2d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True), 
 ReLU(), 
 Conv2d(2, 3, kernel_size=(3, 3), stride=(1, 1)), 
 Foo(
  (m1): Conv2d(1, 2, kernel_size=(3, 3), stride=(1, 1))
  (m2): BatchNorm2d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (m3): ReLU()
  (m4): Conv2d(2, 3, kernel_size=(3, 3), stride=(1, 1))
)]
</code></pre>

sio277: <p>Also note that <code>model.modules()</code> or <code>model.children()</code> do not guarantee that we can get resident modules sequentially from model input to the output - they print out modules in an order of registration.</p>

rahan: <p>Thanks, I’m also looking at torch.nn.Sequential(*list(model.children())) but I’m not sure if it works with residual connections.</p>

sio277: <p>For a residual connection, I’d like to recommend writing a customized class. See official torch code here: <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py" class="inline-onebox" rel="noopener nofollow ugc">vision/resnet.py at master · pytorch/vision · GitHub</a>.</p>

LucasRr: <p>This solution doesn’t really print layers in order of the data flow, but more in the order in which the layers were initialized. For example if you invert the order of <code>self.m3</code> and <code>self.m4</code> in in <code>__init__</code> (but keep the forward pass unchanged), it will change the order of <code>modules</code> as well.</p>
torch/csrc/utils/python_arg_parser.h:424:94: error: format ‘%ld’ expects argument of type ‘long int’, but argument 7 has type ‘int’

tbyte: <p>I’m trying to compile the latest but it failed, then I tried to <code>git checkout v1.12.1</code> but it failed with the same error. The gcc setting for the CPU are <code>-march=armv7ve+simd -mfpu=neon-vfpv4</code> and it fails in the same way on the original hardware and while compiling it with <code>qemu-arm-static</code> on a different machine. There are a lot of errors like this:</p>
<pre><code class="lang-auto">/rstorage/drone/pytorch_install/pytorch/torch/csrc/utils/python_arg_parser.h: In member function ‘std::vector&lt;double, std::allocator&lt;double&gt; &gt; torch::PythonArgs::getDoublelist(int)’:
/rstorage/drone/pytorch_install/pytorch/torch/csrc/utils/python_arg_parser.h:751:82: error: format ‘%ld’ expects argument of type ‘long int’, but argument 7 has type ‘int’ [-Werror=format=]
  751 |           "%s(): argument '%s' must be %s, but found element of type %s at pos %ld",
      |                                                                                ~~^
      |                                                                                  |
      |                                                                                  long int
      |                                                                                %d
......
  756 |           idx + 1);
      |           ~~~~~~~
      |               |
      |               int
</code></pre>
<p>But even if i “fix” that by changing %ld to %d then i get this :</p>
<pre><code class="lang-auto">In file included from /rstorage/drone/pytorch_install/pytorch/torch/csrc/jit/python/python_list.cpp:3:
/rstorage/drone/pytorch_install/pytorch/third_party/pybind11/include/pybind11/detail/common.h: In instantiation of ‘pybind11::ssize_t pybind11::ssize_t_cast(const IntType&amp;) [with IntType = long long int; pybind11::ssize_t = int]’:
/rstorage/drone/pytorch_install/pytorch/third_party/pybind11/include/pybind11/pytypes.h:1985:68:   required from ‘pybind11::list::list(SzType) [with SzType = long long int; typename std::enable_if&lt;std::is_integral&lt;_Tp&gt;::value, int&gt;::type &lt;anonymous&gt; = 0]’
/rstorage/drone/pytorch_install/pytorch/torch/csrc/jit/python/python_list.cpp:32:25:   required from here
/rstorage/drone/pytorch_install/pytorch/third_party/pybind11/include/pybind11/detail/common.h:410:35: error: static assertion failed: Implicit narrowing is not permitted.
  410 |     static_assert(sizeof(IntType) &lt;= sizeof(ssize_t), "Implicit narrowing is not permitted.");
      |                   ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~
/rstorage/drone/pytorch_install/pytorch/third_party/pybind11/include/pybind11/detail/common.h:410:35: note: ‘(sizeof (long long int) &lt;= sizeof (pybind11::ssize_t))’ evaluates to false
</code></pre>
<p>It seem like some kind of an integer auto-sizing error somewhere.</p>
Gradients vanishing or exploding?

Saswat: <p>I am dealing with a classification problem (5000 classes).</p>
<p>My network has 3 MLP layers followed by 1 LSTM block.</p>
<pre><code class="lang-auto">class Model(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, output_size, num_selected, layer_size=1):
        super(Model, self).__init__()
        self.layer_size = layer_size
        self.hidden_size = hidden_dim

        self.fc = nn.Sequential(
            nn.Linear(4000, 1024),
            nn.ReLU(),
            nn.Linear(1024, 512),
            nn.ReLU(),
            nn.Linear(512, embedding_dim),
            nn.ReLU(),
        )

        self.lstm = nn.LSTM(embedding_dim, hidden_dim, bidirectional=True, num_layers=layer_size)
        self.hidden2tag = nn.Linear(2*hidden_dim, output_size)

    def forward(self, input_vector):
        embeddings = self.fc(input_vector)
        lstm_output, _ = self.lstm(embeddings)
        logits = self.hidden2tag(lstm_output)
        return logits 
</code></pre>
<p>This is the gradient flow observed. Are my gradients exploding in the Linear layers and vanishing in the LSTM? How do I bring uniformity to this flow?</p>
<p>Performance is also affected as the network always predicts a single class as output</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88d417779ae8dfbc91f849a16dd981a1d5a888c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/88d417779ae8dfbc91f849a16dd981a1d5a888c5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88d417779ae8dfbc91f849a16dd981a1d5a888c5_2_618x500.png" alt="image" data-base62-sha1="jwrl63Cragnb7zbJxCsnTIXmsYd" width="618" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88d417779ae8dfbc91f849a16dd981a1d5a888c5_2_618x500.png, https://discuss.pytorch.org/uploads/default/original/3X/8/8/88d417779ae8dfbc91f849a16dd981a1d5a888c5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/8/88d417779ae8dfbc91f849a16dd981a1d5a888c5.png 2x" data-dominant-color="E9F2F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">900×728 18.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Speeding up dataloaders on google colab

alireza_khatami: <p>Hello good people;  So I have read about questions similar to this one but didn’t get any results by implementing the changes mentioned(like increasing the number of the workers-the model is fast , thus doing so won’t be necessary, or pin = true -which actually helped a little bit. But still, there is a way to go . so my dataset contains images of size 384*512, in .bmp format (3000 of them ). and here is the code for the dataset :</p>
<pre data-code-wrap="lass"><code class="lang-nohighlight">    def __init__(self, imagenames , refDir , ssimDir, transform=None):
        self.refDir = refDir
        self.ssimDir=ssimDir
        self.imagesName = imagenames
        self.transform = transform
        
    def __getitem__(self, index):
      x = Image.fromarray(io.imread(os.path.join(self.refDir , self.imagesName [index])))
      y = Image.fromarray(io.imread(os.path.join(self.ssimDir , self.imagesName [index])))
      if self.transform:
          x = self.transform(x)
          y = self.transform(y)
      return x, y
    def __len__(self):
        return len(self.imagesName)
transform=transforms.Compose([
                            transforms.ToTensor(),
                            transforms.Normalize((0.5,0.5,0.5), (1.0,1.0,1.0))
                        ])
disortedDir = r'/content/gdrive/MyDrive/tid2013/distorted_images'
ssimMapsDir = r'/content/gdrive/MyDrive/tid2013/ssimMaps'
training_data = MyDataset(X_train , disortedDir , ssimMapsDir , transform)
</code></pre>
<p>And here is the code for dataloader :</p>
<pre><code class="lang-auto">training_loader = DataLoader(training_data, 
                             batch_size=batch_size, 
                             shuffle=True,
                             pin_memory=True)```
batchize = 32 when training the model and  here is how I train it : 
```for i in xrange(num_training_updates):
    (data, ssim) = next(iter(training_loader))
    data = data.to(device)
    ssim = ssim.to(device)
    optimizer.zero_grad()

    vq_loss, data_recon, perplexity = model(data)
    recon_error = F.mse_loss(data_recon, ssim) / data_variance
    loss = recon_error + vq_loss
    loss.backward()

    optimizer.step()  # based on the code from  Aäron van den Oord on google colab _ pytorch
</code></pre>
<p>In the original code when they used pytorch MNIST dataset as follows:</p>
<pre><code class="lang-auto">#                                   transform=transforms.Compose([
#                                       transforms.ToTensor(),
#                                       transforms.Normalize((0.5,0.5,0.5), (1.0,1.0,1.0))
#                                   ]))
</code></pre>
<p>It takes about seconds but when I use my own costume dataset it takes several minutes. Most of the time when training is spent on fetching the dataset. How can I improve it ?? Is this even normal ( sinceimages of my dataset are much bigger in size )?</p>
"at" has no member "cudnn_convolution_xxx"

FateScript: <p>Hello there,</p>
<p>I’m using pytorch 1.12 to compile a custom operator and I found that<br>
<code>at::cudnn_convolution_backward_weight</code> is missing which is supposed to be in <code>include/ATen/NativeFunctions.h</code>.<br>
So I got the following error:</p>
<pre><code class="lang-auto">namespace "at" has no member "cudnn_convolution_backward_weight"
</code></pre>
<p>I search the include directory and only found a function named <code>at::convolution_backward</code>.</p>
<p>Am I suppose to use this function instead of <code>at::cudnn_convolution_backward_weight</code> ? Is there any documentation for me? I couldn’t find any clue in release note.</p>
<p>Thanks!</p>

FateScript: <p>similar issue could be found <a href="https://discuss.pytorch.org/t/error-with-at-cudnn-convolution-backward-input/148528">here</a></p>

ptrblck: <p>The <code>cudnn</code> calls are not exposed anymore if I’m not mistaken, so use the plain <code>convolution</code> functions instead.</p>

FateScript: <p>Thanks! I refer to code <a href="https://github.com/pytorch/pytorch/blob/fe190078aa78dad94297aece4d8322e5f4262558/aten/src/ATen/native/Convolution.cpp#L1784-L1804" rel="noopener nofollow ugc">here</a> to solve my problem.</p>
<p>For those people who are stucked by this problem, plz use the <code>convolution_backward</code> instead of <code>cudnn_convolution_xxx</code>. Or you could use <code>torch &lt; 1.11.0</code> to compile your operator.</p>
Unrolling a single forward-backward pass

superock: <p>I am trying to replicate something that the authors in the paper, <a href="https://arxiv.org/pdf/1910.07169.pdf" rel="noopener nofollow ugc">Generative Modeling for Small-Data Object Detection</a>, have proposed in the <em>Section 3.2. Train Generator with Detection Losses</em> which says:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5ac2d47fe3178f2b06e0b04e776ad9375421ab61.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5ac2d47fe3178f2b06e0b04e776ad9375421ab61" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5ac2d47fe3178f2b06e0b04e776ad9375421ab61_2_340x500.jpeg" alt="image" data-base62-sha1="cWUopYqIMOEHmcyP1bNjXSycKml" width="340" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5ac2d47fe3178f2b06e0b04e776ad9375421ab61_2_340x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/a/5ac2d47fe3178f2b06e0b04e776ad9375421ab61.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/a/5ac2d47fe3178f2b06e0b04e776ad9375421ab61.jpeg 2x" data-dominant-color="E6E7E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">440×646 169 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I understand the theoretical aspect of the proposed solution, but I can’t figure out a way to code in PyTorch this unrolling procedure. I tried in several ways but I always get something related to the fact that the graph is disconnected.</p>
<p>This is a sketch of a simple implementation with VAE instead of GAN: <a href="https://colab.research.google.com/drive/1SwesVOwGBHd5bXvJ-ZRxXJkHsOw6Eaqq?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
Qunatizarion model for rasberry

AlOa: <p>Hi,<br>
I quantizied my model using fbgemm backend for my x64 cpu, now I want to try it on a rasberry 4 where I compiled libtorch. I switched to the qnnpack backend and I quantizied model on mi cpu, ijted it, but when I try to load it, I get an error</p>
<p>RuntimeError: Didn’t find engine for operation quantized::conv2d_prepack NoQEngine</p>
<p>The libtorch is build with qnnpack. I must do the quantization on rasberry?<br>
Thanks</p>
<pre><code>           Alberto
</code></pre>

Zafar: <p>can you provide more information? How do you compile the libtorch? When you perform the quantization, how do you set the qengine? Maybe share the minimum reproducible script?</p>

AlOa: <p>Hi,<br>
I use FX for quantization:</p>
<pre><code class="lang-auto">#backend = 'fbgemm''
backend = 'qnnpack'

model_q  = copy.deepcopy(model)
model_q.eval()
torch.backends.quantized.engine = backend
qconfig_dict = {"": torch.quantization.get_default_qconfig(backend)}
model_prepared = quantize_fx.prepare_fx(model_q, qconfig_dict)
model_prepared.eval()
print("Load dataset")
data = ImageFolder("") //Load dataset
datasets = train_val_dataset(data)
print("Validation dataset size {}".format(len(datasets['val'])))
dataLoaderVal = DataLoader(datasets['val'] , batch_size=64,
                           num_workers=4, pin_memory=True)

print("Calibration")
n = 0
with torch.inference_mode():
    for b in dataLoaderVal:
        n = n + 1
        model_prepared(b[0])
        if n &gt; 100:
            break
print("End calibration")

model_quantized = quantize_fx.convert_fx(model_prepared)


example = torch.rand(1, 3, 244, 244)
traced_script_module = torch.jit.trace(model_quantized, example)
traced_script_module.save(name+"_q.pt")
</code></pre>
<p>Then I load the model in my c++ with module = torch::jit::load(model_q).<br>
With fbgemm backen on x64 desktop it works without problem and also with qnnpack (it slower as aspected ).</p>
<p>On my rasberry I compiled litorch from sources, with</p>
<p><code>cmake  -DBUILD_PYTHON=OFF -DUSE_CUDA=OFF -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release</code></p>
<p>and in the lin direcotiry there is libnnpack.a.</p>
<pre><code>                  Alberto
</code></pre>
GPU utilization low until validation

m_mandel: <p>Hi,<br>
I am running a DDP run with 2 GPUs, and at every 10 epochs I evaluate my test data.</p>
<p>For some reason, the first 10 epochs run very slow - until the evaluation process occurs. Can anyone point me to the reason for this?</p>
<p>Attaching visualization of GPU utilization:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d36e810a1a265aab912a38dfe3eb85e18aa23803.png" data-download-href="https://discuss.pytorch.org/uploads/default/d36e810a1a265aab912a38dfe3eb85e18aa23803" title="gpu_util_crop"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d36e810a1a265aab912a38dfe3eb85e18aa23803_2_690x269.png" alt="gpu_util_crop" data-base62-sha1="uapv0t8rGuCGmC9yIk92IKW9C7x" width="690" height="269" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d36e810a1a265aab912a38dfe3eb85e18aa23803_2_690x269.png, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d36e810a1a265aab912a38dfe3eb85e18aa23803.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d36e810a1a265aab912a38dfe3eb85e18aa23803.png 2x" data-dominant-color="F2E1DA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpu_util_crop</span><span class="informations">924×361 56.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My train/test dataloaders use utils.data.distributed.DistributedSampler as samplers</p>

AlphaBetaGamma96: <p>Can you share a minimal reproducible example?</p>

m_mandel: <p>At the moment, my code is a bit too complex to share.<br>
I was hoping maybe someone might be able to point in the right direction.</p>

m_mandel: <p>Hi,</p>
<p>Solved! sort of…</p>
<p>Apparently calling <code>torch.set_num_threads(1)</code> before starting to train solves the problem. Though I have no idea why.</p>
<p>If anyone has any light to shed on this matter, it would be greatly appreciated!</p>
<p>Thanks</p>

mrshenli: <p>This might be relevant to OpenMP multithreading: <a href="https://github.com/pytorch/pytorch/issues/16894" class="inline-onebox" rel="noopener nofollow ugc">Number of CPU threads for the python process · Issue #16894 · pytorch/pytorch · GitHub</a></p>
<p>Distributed launcher also does the same thing: <a href="https://github.com/zhaojuanmao/pytorch/blob/12f0052eee599ae85c78ccb22c17ae41cc221ff2/torch/distributed/launch.py#L217-L224" class="inline-onebox" rel="noopener nofollow ugc">pytorch/launch.py at 12f0052eee599ae85c78ccb22c17ae41cc221ff2 · zhaojuanmao/pytorch · GitHub</a></p>

m_mandel: <p>Thank you! I will look into this.</p>
What pytorch loss function should I use for 1D sequence per-element classification?

JimW: <p>I have a network that can generate predictions with shape [8, 32, 400]. The first number “8” is simply the batch size, and the 2nd number “32” is the length of my sequence, and the 3rd number “400” is the total number of classes for each element in my sequence.</p>
<p>Basically, I want to perform per-element classification for each sequence (like a simplified version of semantic segmentation). So I define my labels to have shape [8， 32]， here, 8 is the batch size and 32 is the sequence length (or total number of elements in this sequence)</p>
<p>What pytorch loss function should I use here?</p>

ptrblck: <p>You could use <code>nn.CrossEntropyLoss</code> and pass the model outputs as <code>[batch_size, nb_classes, seq_len]</code> to this loss function while the targets are fine in<code>[batch_size, seq_len]</code>. Make sure to <code>.permute</code> the model output to create the desired shape.</p>
Is it possible to send a Dataset CPU multiple indices at once?

Sam_Lerman: <p>Right now, <code>__getitem__</code> implements 1 index at a time. Is it possible to have the <code>DataLoader</code> feed it multiple indices for retrieval at a time?</p>

ptrblck: <p>You could use a <code>BatchSampler</code> as described <a href="https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4">here</a>.</p>
How interpret different Pytorch nightly links?

Vedant_Roy: <p>I’m trying to install Pytorch nightly with CUDA 11.6.<br>
The documentation says to use the following URL: <a href="https://download.pytorch.org/whl/nightly/cu116" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu116</a> as an argument to pip.</p>
<p>When I go to this URL, I see 2 different links: <code>torch</code> and <code>torch_nightly</code>. What is the difference / why are they both under the <code>whl/nightly</code> path?</p>

ptrblck: <p>I would recommend to use the created install command from <a href="http://pytorch.org">pytorch.org</a> and allow <code>pip</code> to grab the right binary for your setup.</p>
Efficiency of DataParallel(DP), ModelParallel and Distributed DP

rabot: <p>Hi,<br>
I was playing around with the PyTorch distributed methodologies, and am wondering whether there’s any comparison documentation/tutorial available?</p>
<p>It appears that using DP actually takes 2x time on FashionMNIST classification task, than not using it.</p>
<p>For e.g., in the reference code below, net_dp took 367.25 s to train compared to 149.9 s using a single NVIDIA RTX a6000 GPU.</p>
<p>Wondering why is this or am I doing it wrong?</p>
<p>Ref:</p>
<ol>
<li>Sample code</li>
</ol>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 4 * 4, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 4 * 4)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

def train(net: nn.Module, is_dp=False): 
    ...

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

net1 = nn.DataPrallel(Net())
net.to(device)
train(net1, True)

"""
DataParallel(
  (module): Net(
    (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))
    (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
    (fc1): Linear(in_features=256, out_features=120, bias=True)
    (fc2): Linear(in_features=120, out_features=84, bias=True)
    (fc3): Linear(in_features=84, out_features=10, bias=True)
  )
)
"""
net2_single-gpu = Net().to(device)
train(net2_single-gpu, False)
</code></pre>

mrshenli: <p>Hey <a class="mention" href="/u/rabot">@rabot</a>, DP broadcasts model to all devices in every iteration. So if the parallelism on compute does not lead to sufficient speed up, the overhead of broadcasting model and gathering gradients make it slower than local training.</p>
<p>Usually,  DDP/DP is used when the global batch size large enough to make full use of the parallelism. For comparisons, please see this post: <a href="https://medium.com/pytorch/pytorch-data-parallel-best-practices-on-google-cloud-6c8da2be180d" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Data Parallel Best Practices on Google Cloud | by PyTorch | PyTorch | Medium</a></p>
Pytorch Profiler with DDP

kristianeschenburg: <p>Hi</p>
<p>I’m trying to use the <code>torch.profiler</code> tool but am running into issues using concurrently with <code>torch.distributed</code>.  My training script is organized like this:</p>
<p>python</p>
<pre><code class="lang-auto">def train(args.local_world_size, args.local_rank, args):
   # load data
   # set up model
   model = DDP(model)
   ...
   # train
   # evaluate

def main(local_world_size, local_rank, args):
    # These are the parameters used to initialize the process group
    home = os.path.expanduser("~")
    init_file = f"{home}/shared_init_file"

    env_dict = {
        key: os.environ[key]
        for key in ("MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE")
    }

    print(f"[{os.getpid()}] Initializing process group with: {env_dict}")

    dist.init_process_group(backend="nccl",
                            rank=local_rank,
                            world_size=local_world_size,
                            init_method=f'file://{init_file}')

    print(
        f"[{os.getpid()}] world_size = {dist.get_world_size()}, "
        + f"rank = {dist.get_rank()}, backend={dist.get_backend()}"
    )

    # train model
    train(args.local_world_size, args.local_rank, args)

    # Tear down the process group
    dist.barrier() # first synchronize devices
    dist.destroy_process_group() # then destroy

    # removed shared process file
    if os.path.exists(init_file):
        os.remove(init_file)


if __name__ == "__main__":
   # get CLI arguments
   parser = argparse.ArgumentParser()
   ...
   ...
   args = parser.parse_args()
   main(args.local_world_size, args.local_rank, args)

</code></pre>
<p>Irrespective if I put the profiler in <code>main()</code> or <code>train()</code>, the script hangs at the <code>dist.init_process_group</code> step of <code>main()</code>.  The script runs correctly when removing all lines associated with the profiler.  What is the correct way to utilize the profiler when using <code>torch.distributed</code>?</p>
<p>Thanks.</p>
<p>Kristian</p>

mrshenli: <aside class="quote no-group" data-username="kristianeschenburg" data-post="1" data-topic="163508">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kristianeschenburg/48/32129_2.png" class="avatar"> kristianeschenburg:</div>
<blockquote>
<p>Irrespective if I put the profiler in <code>main()</code> or <code>train()</code>, the script hangs at the <code>dist.init_process_group</code> step of <code>main()</code>.</p>
</blockquote>
</aside>
<p>This means even if it hasn’t started profiler context, it still hangs at <code>init_process_group</code>? Which PyTorch version are you using?</p>
<p>For sample code, this worked for me: <a href="https://github.com/mrshenli/ptd_benchmark/blob/62f8c89bab7d61607dc101abfb37480e27ee4616/trainer.py#L375-L384" rel="noopener nofollow ugc">https://github.com/mrshenli/ptd_benchmark/blob/62f8c89bab7d61607dc101abfb37480e27ee4616/trainer.py#L375-L384</a></p>
RuntimeError: assigned grad has data of a different type

Siladittya_Manna: <p>I am trying to assign a ndarray of zeros to the grad of the model parameters, but I am getting the error in the title.</p>
<p>I was just trying to see if I can assign a numpy array to the <code>param.grad</code>. But in my project I won’t be assigning zeros actually.</p>
<pre><code class="lang-auto">for param in model.parameters():
            #print(param.grad)
            param.grad = torch.from_numpy(np.zeros(param.grad.shape)).type(param.grad.dtype)
</code></pre>
<p>I am trying to do this step after <code>loss.backward()</code> step.<br>
I want to know what is causing this error and how to solve it.<br>
Kindly help!</p>

Siladittya_Manna: <p>Adding <code>model.to("cpu")</code> before the <code>for</code> loop worked.</p>
<p>Thanks!!</p>
<p>(Also need to move the model back to the device before passing an input through the model)</p>

albanD: <p>Note that this comes from the fact that the Tensor from numpy was on the CPU, while the param was on the GPU.<br>
Another way to solve this (without having to send the whole model back to the CPU) would be to send the new grad to the GPU as well with <code>.to(param.device)</code>.</p>

Jianquan_Zhao: <p>I met this error again, and I found  the true reason is your param.grad and new grad aren’t in the same device.<br>
so the best method to solve this error is</p>
<pre><code class="lang-auto">param.grad = torch.from_numpy(np.zeros(param.grad.shape)).type(param.grad.dtype).to(param.device)
</code></pre>

Minh_Vu: <p>to(param.device) is trully a solution !!!</p>
Circular shift without generating copy

pepper8362: <p>Hi,</p>
<p>I note I can use <code>torch.roll</code> to do circular shift for a tensor, but this function seems to generate a copy of the original tensor, which can be expensive if the original tensor is large. Is there a way to do circular shift without generating a copy of the original tensor (the underlying mechanism may be something like moving a head pointer in a circular array)?</p>
<p>Thanks!</p>

KFrank: <p>Hi Pepper!</p>
<aside class="quote no-group" data-username="pepper8362" data-post="1" data-topic="163696" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/9e8a1a/48.png" class="avatar"> pepper8362:</div>
<blockquote>
<p>Is there a way to do circular shift without generating a copy of the original tensor</p>
</blockquote>
</aside>
<p>I do not believe that pytorch offers a view (or an inplace) version of <code>roll()</code><br>
(although hypothetically it could).</p>
<aside class="quote no-group">
<blockquote>
<p>(the underlying mechanism may be something like moving a head pointer in a circular array)?</p>
</blockquote>
</aside>
<p>This won’t work with the way pytorch stores tensors.  Pytorch stores<br>
tensors (ignoring <em>strides</em>) in row-major form, and to preserve this storage<br>
format, <code>roll()</code> has to reorder the tensor’s elements in memory.  We can<br>
use <code>.storage()</code> to probe how a tensor is actually stored:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; t = torch.tensor ([[1, 2, 3], [4, 5, 6]])
&gt;&gt;&gt; t.storage()
 1
 2
 3
 4
 5
 6
[torch.storage._TypedStorage(dtype=torch.int64, device=cpu) of size 6]
&gt;&gt;&gt; t_roll = t.roll (1, dims = 1)
&gt;&gt;&gt; t_roll
tensor([[3, 1, 2],
        [6, 4, 5]])
&gt;&gt;&gt; t_roll.storage()
 3
 1
 2
 6
 4
 5
[torch.storage._TypedStorage(dtype=torch.int64, device=cpu) of size 6]
</code></pre>
<p>One could imagine implementing a “head pointer” as a more complicated<br>
data structure.  Note, such a head pointer couldn’t simply index into a<br>
circular array.  (Consider what would happen if you rolled, say, a 5d tensor<br>
along its third dimension.)  But implementing such a scheme would require<br>
having many of pytorch’s tensor operations, e.g., <code>matmul()</code>, <code>cumsum()</code>,<br>
tensor indexing, and so on, understand this head-pointer data structure at<br>
some cost in efficiency.</p>
<p>Best.</p>
<p>K. Frank</p>

pepper8362: <p>I see. Thank you very much for your answer!</p>
Convert DeepFillv2 to Onnx

npodsevatkin: <p>Hello! I’m trying to convert DeepFillv2 model to Onnx, but I have this error: Unsupported: ONNX export of convolution for kernel of unknown shape.</p>
<p>Do you have any ideas why am a getting this error?</p>
<p>Code for converting: `from mmcv.onnx import register_extra_symbolics</p>
<p>def pytorch2onnx(model,<br>
opset_version=11,<br>
show=False,<br>
output_file=‘tmp.onnx’,<br>
verify=False,<br>
dynamic_export=True):<br>
“”“Export Pytorch model to ONNX model and verify the outputs are same<br>
between Pytorch and ONNX.<br>
Args:<br>
model (nn.Module): Pytorch model we want to export.<br>
input (dict): We need to use this input to execute the model.<br>
opset_version (int): The onnx op version. Default: 11.<br>
show (bool): Whether print the computation graph. Default: False.<br>
output_file (string): The path to where we store the output ONNX model.<br>
Default: <code>tmp.onnx</code>.<br>
verify (bool): Whether compare the outputs between Pytorch and ONNX.<br>
Default: False.<br>
“””<br>
model.cpu().eval()</p>
<pre><code>img = torch.rand([3, 256, 256]).unsqueeze(0).cpu()
mask = torch.empty(img.shape[0], 1, img.shape[2], img.shape[3]).cpu()
data = (img, mask)

register_extra_symbolics(opset_version)
dynamic_axes = None
if dynamic_export:
    dynamic_axes = {
        'input': {
            0: 'batch',
            2: 'height',
            3: 'width'
        },
        'output': {
            0: 'batch',
            2: 'height',
            3: 'width'
        }
    }
with torch.no_grad():
    torch.onnx.export(
        model,
        data,
        output_file,
        input_names=['input'],
        output_names=['output'],
        export_params=True,
        keep_initializers_as_inputs=False,
        verbose=show,
        opset_version=opset_version,
        dynamic_axes=dynamic_axes)
print(f'Successfully exported ONNX model: {output_file}')
</code></pre>
<p>Traceback:<br>
~/anaconda3/lib/python3.9/site-packages/torch/onnx/symbolic_opset9.py in _convolution(g, input, weight, bias, stride, padding, dilation, transposed, output_padding, groups, benchmark, deterministic, cudnn_enabled, allow_tf32)<br>
1298<br>
1299     if kernel_shape is None or any([i is None for i in kernel_shape]):<br>
 → 1300         raise RuntimeError("Unsupported: ONNX export of convolution for kernel "<br>
1301                            “of unknown shape.”)<br>
1302</p>
<p>RuntimeError: Unsupported: ONNX export of convolution for kernel of unknown shape.</p>

LoveU3tHousand2: <p>I met same issue but i fixed by not using reshape(-1, a, b, c)    ‘-1’  not supported by onnx in contextual_attention.py , it must be calculated by yourself.</p>
How can I get the object feature vectors from Pytorch Faster-RCNN

rma: <p>Hi, I am wondering how can I get the detected object features from the model Faster-RCNN?<br>
For example, for example, in an image that detects 48 boxes, 1000 feature vectors are extracted from the “fc6” layer for all proposal regions. How can I get the vector corresponding to the detected objects?<br>
Thanks and Regards</p>
Does torchvision.transforms.ColorJitter apply the same transform to each batch element

JamesDickens: <p>I’m thinking of applying the transform <code>torchvision.transforms.ColorJitter</code> to a video,<br>
but I need to make sure the same transform is applied to each frame.</p>
<p>I have a function like:</p>
<pre><code class="lang-auto">#vid_t of shape [batch_size, num_channels, num_frames, height, width]
def rgb_vid_color_jitter(vid_t, brightness=0.4, contrast=0.4, saturation=0.4, hue=0.2):
    # note this transform expects the video/image to be in range [0,1], not [0, 255.0]
    transform = torchvision.transforms.ColorJitter(brightness=brightness, contrast=contrast,
                                                   saturation=saturation, hue=hue)
    vid_t = torch.permute(vid_t, (0, 2, 1, 3, 4))
    vid_t = transform(vid_t)
    vid_t = torch.permute(vid_t, (0, 2, 1, 3, 4))
    return vid_t
</code></pre>
<p>Any insights appreciated.</p>

ptrblck: <p>The transformations should be applied on each image separately and you could check it quickly by applying your code onto a static input tensor (e.g. just <code>torch.ones</code>). If you need to apply the same “random” transformation on multiple inputs you could use the functional API and create the random parameters once.</p>

JamesDickens: <p>I ended up with some code like:</p>
<pre><code class="lang-auto">#vid of shape [batch_size, num_channels, num_frames, height, width]
def random_color_jitter(vid, brightness, contrast, saturation, hue):
    batch_size = vid.shape[0]
    for i in range(batch_size): # a different transform per video
        if brightness &gt; 0:
            brightness_factor = random.uniform(
                max(0, 1 - brightness), 1 + brightness)
        else:
            brightness_factor = None
        if contrast &gt; 0:
            contrast_factor = random.uniform(
                max(0, 1 - contrast), 1 + contrast)
        else:
            contrast_factor = None
        if saturation &gt; 0:
            saturation_factor = random.uniform(
                max(0, 1 - saturation), 1 + saturation)
        else:
            saturation_factor = None
        if hue &gt; 0:
            hue_factor = random.uniform(-hue, hue)
        else:
            hue_factor = None
        vid_transforms = []
        if brightness is not None:
            vid_transforms.append(lambda img: torchvision.transforms.functional.adjust_brightness(img, brightness_factor))
        if saturation is not None:
            vid_transforms.append(lambda img: torchvision.transforms.functional.adjust_saturation(img, saturation_factor))
        if hue is not None:
            vid_transforms.append(lambda img: torchvision.transforms.functional.adjust_hue(img, hue_factor))
        if contrast is not None:
            vid_transforms.append(lambda img: torchvision.transforms.functional.adjust_contrast(img, contrast_factor))
        random.shuffle(vid_transforms)
        v = vid[i]
        v = torch.permute(v, (1, 0, 2, 3))
        for transform in vid_transforms:
            v = transform(v)
        v = torch.permute(v, (1, 0, 2, 3))
        vid[i] = v
    return vid
</code></pre>
Nightly build installs stable version

cruvy: <p>I’m trying to install the nightly build for linux using the command I get from <a href="https://pytorch.org/get-started/locally/:" rel="noopener nofollow ugc">https://pytorch.org/get-started/locally/:</a><br>
pip3 install --pre torch torchvision torchaudio --extra-index-url <a href="https://download.pytorch.org/whl/nightly/cpu" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cpu</a></p>
<p>It seems to install torch 1.11.0… When I do the same on mac, I get version 1.12.0.dev20220417.</p>
<p>Any idea how I can get the latest version on linux?</p>

ptrblck: <p>I cannot reproduce the issue on Linux and get the correct nightly binary:</p>
<pre><code class="lang-python">pip install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/nightly/cpu
Collecting torch
  Downloading https://download.pytorch.org/whl/nightly/cpu/torch-1.12.0.dev20220417%2Bcpu-cp38-cp38-linux_x86_64.whl (173.1 MB)
...
</code></pre>
<p>I’m on Ubuntu 20.04 with Python 3.8 in case that matters.</p>

cruvy: <p>I’m trying to get the aarch64 version, is that the reason?</p>

ptrblck: <p>It might be related to the architecture, yes.<br>
Assuming you are using a Mac, you could try to download the nightly binaries directly from e.g.<br>
<a href="https://download.pytorch.org/whl/nightly/cpu/torch-1.12.0.dev20220418-cp39-none-macosx_11_0_arm64.whl">here</a>.</p>

cruvy: <p>Any idea how to find the nightly binary for linux aarch64? Can’t locate it under <a href="https://download.pytorch.org/whl/nightly/cpu/torch_nightly/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cpu/torch_nightly/</a></p>

_kel: <p>Do they have any nightly binary for arm64 M1 (Monterey) for python 3.10.6?</p>
Fine Tuning: different learning rates for different parameters

atpudludrim: <p>I have created custom layers nested within one another, the first of which uses an Embedding layer. I’m loading pretrained embeddings in it with freeze=False and want to train it with the rest of the model, but at a slower learning rate. When creating optimizer, I can propagate parameters of this layer through the nested layers but it’s ugly and hacky. And it is also already contained in model.parameters() which causes conflicts when providing per parameter options. Is there a cleaner way to do this? See the code below for how I’m doing it right now.</p>
<pre><code class="lang-auto">class layer1(nn.Module):
    def __init__(self, emb_size, pretrained):
        self.embedding_layer = nn.Embedding.from_pretrained(pretrained, freeze=False)
        self.fine_tune = self.embedding_layer.parameters()
        #other parameters/layers/operations
    ...

class layer2(nn.Module):
    def __init__(self, emb_size, pretrained, proj_size):
        self.layer1 = layer1(emb_size, pretrained)
        self.fine_tune = self.layer1.fine_tune
        #other parameters/layers/operations
    ...

class myModel(nn.Module):
    def __init__(self, emb_size, pretrained, proj_size, attention_dim):
        self.layer2 = layer2(emb_size, pretrained, proj_size)
        self.fine_tune = self.layer2.fine_tune
        #other parameters/layers/operations
    ...

model = myModel(emb_size, pretrained, proj_size, attention_dim)
fine_tune_id = id(list(model.fine_tune)[0])
normal_params = []
for param in model.parameters():
    if id(param) != fine_tune_id:
        normal_param.append(param)
opt = optim.Adam([{'params':model.fine_tune,'lr':1e-4},
               {'params':normal_params}],lr=1e-3)
</code></pre>
What is the purpose of reduce range

Rami_Ismael: <p>I go want to make a specific change in MOVINGAVERAGEPERCHANNELMINMAXOBSERVER. I saw reduced range exists. I saw the default setting for fbgemm for activation reduce range is set to true.</p>
<p>When should you set the reduced range to true or false?</p>

Zafar: <p>Reduced range is forcing the quantization to use 7 bits instead of 8 bits. Due to internal implementation, FBGEMM needs this for the weight tensors.</p>
Concerning .detach() as input argument to a function

XHH: <p>Hi there,<br>
My question is as follows:<br>
If I have a tensor ‘a’, which comes out as intermediate output of network, and requires_grad= True,<br>
then I pass ‘a.detach()’ as input argument to function like do_sth(a.detach()), and does sth. else. However, after passing into the function and back in the main flow, ‘a’ will further generate a loss and requires loss.backward(). The code is roughly as below:<br>
a = net(input)<br>
do_sth(a.detach())<br>
loss = L1loss(a, label)<br>
loss.backward()<br>
Will the weights of the network generating ‘a’ still get backpropagated when do the loss.backward()?</p>

srishti-git1110: <p>Hi,<br>
Yes!<br>
As <code>a.detach()</code> returns a new tensor rather than modifying <code>a</code> in-place. As long as the computation graph of <code>loss</code> doesn’t break, you should be fine updating the parameters of <code>net</code>.</p>
<p>So, make sure nothing that breaks the graph happens inside your function <code>do_sth</code>.</p>