akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pybind ddp pickle

username: Lin1
cooked: <p>Hello everyone,</p>
<p>A c++ preprocessor class is implemented with pybind11. The c++ class is imported in a custominzed dataset(torch.utils.data.Dataset) and one of its funtion is called by <strong>getitem</strong>. Once I tried to use DDP for multi-process training, the errror “<strong>cannot pickle preprocessor object</strong>” occurs. It works correctly, if i use one gpu without DDP(num_worker&gt;2, num_batch&gt;2).</p>
<p>It seems ddp need to pickle everything contained by the dataset class for sharing the dataloader between different process (<a href="https://stackoverflow.com/questions/69797634/how-to-pickle-an-object-which-is-written-in-c-through-pybind11" rel="noopener nofollow ugc">link</a>).</p>
<p>I also checked the official pybind11 doc about pickling support: <a href="https://pybind11.readthedocs.io/en/stable/advanced/classes.html#pickling-support" rel="noopener nofollow ugc">link</a><br>
But there is no cues about how to pickle a general c++ classes with pointers (fill numpy arrays).</p>
<p>Am I using ddp wrongly? or is there any solution can solve it?</p>
<p>Thanks,<br>
Lin</p>

username: rvarm1
cooked: <p>DDP by itself does not do any pickling or directly interact with <code>Datset</code> or even multiprocessing communication techniques.</p>
<p>This would fall into the domain of dataloader or multiprocessing cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a></p>

username: VitalyFedyunin
cooked: <p>Hard to say without seeing reproducible example. But I would guess that you have custom pybind objects inside of DataSet structures. Try defining <code>__setstate__</code> and <code>__getstate__</code> function for them. Also in most cases (if it is not shared memory), passing memory pointers between processes will not work and you need to serialize data.</p>

username: Lin1
cooked: <p>Thanks <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> . Problem sovled by defining <strong>setstate</strong> and <strong>getstate</strong> mentioned in pybind official <a href="https://pybind11.readthedocs.io/en/stable/advanced/classes.html#pickling-support" rel="noopener nofollow ugc">doc</a></p>
<p>Just define setstate/getstate for those parameters used in the class’s construct funtion.</p>
How to enable GRAPH_DEBUG in jit？

username: pytorch_lzwhard
cooked: <pre><code class="lang-auto">#define GRAPH_DEBUG(...) \
  JIT_LOG(::torch::jit::JitLoggingLevels::GRAPH_DEBUG, __VA_ARGS__);
// use GRAPH_EXPORT to export a graph so that the IR can be loaded by a script
</code></pre>
<p>I have no idea about how to enable this log, thanks for helping me out.</p>

username: ptrblck
cooked: <p><a href="https://github.com/pytorch/pytorch/blob/c81b802ead07031aa47f33a2b713c916e983967c/torch/csrc/jit/OVERVIEW.md#jit-logging">This JIT Logging</a> doc explains how these logging utils. can be used and enabled.</p>
Server socket cannot connect!

username: just_started_coding
cooked: <p>Why would I get error message like this? I am seeting master add to local host and letting master port to be default? I am using one-node 4 gpu system. Any help will be appreciated.</p>
<p><code>python -m torch.distributed.launch --nproc_per_node=4 --nnodes=1 --node_rank=0 --master_addr="localhost" train.py configs/check.py</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bec7f1dd8793826d760cd565c477c43d4f2f2b38.png" data-download-href="https://discuss.pytorch.org/uploads/default/bec7f1dd8793826d760cd565c477c43d4f2f2b38" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bec7f1dd8793826d760cd565c477c43d4f2f2b38.png" alt="image" data-base62-sha1="rdJ68ZPiiothM5LM49VqfrYbrJu" width="690" height="188" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1256×344 19.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: wanchaol
cooked: <p><a class="mention" href="/u/just_started_coding">@just_started_coding</a> so you are launching the train program within a single node setting, I guess you don’t need to specify the master_addr in this case, according to the doc <a href="https://pytorch.org/docs/stable/distributed.html?highlight=torch%20distributed%20launch#module-torch.distributed.launch" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.12 documentation</a></p>
Autoencoder for voxel classification

username: farah_jabeen
cooked: <p>I am working on cardiac low-dose ct images and implementing a multi-scale autoencoder for anomaly detection. I have to classify voxel classification. I am not sure about that can be possible or not. If someone has an idea kindly let me know.</p>
<p>For reference architecture</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15d7d5d855752370c82abb9aef7a82c285882588.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/15d7d5d855752370c82abb9aef7a82c285882588" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15d7d5d855752370c82abb9aef7a82c285882588_2_690x308.jpeg" alt="image" data-base62-sha1="37es1hp5HZVgfebMSzazEEAIM1G" width="690" height="308" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15d7d5d855752370c82abb9aef7a82c285882588_2_690x308.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15d7d5d855752370c82abb9aef7a82c285882588_2_1035x462.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/5/15d7d5d855752370c82abb9aef7a82c285882588.jpeg 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1254×561 125 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
DDP + Model Parallelism Tutorial

username: QasimKhan5x
cooked: <p>I have an existing DDP pipeline that can fit on my single-node multiple GPU system. But when I increase the size of my model, the forward pass doesn’t fit on one GPU. Is there any tutorial regarding implementation of DDP + Model Parallelism together?</p>

username: wanchaol
cooked: <p><a class="mention" href="/u/qasimkhan5x">@QasimKhan5x</a> thanks for posting, we recently release FSDP (FullyShardedDataParallel) to deal with the case that the model size couldn’t fit into one GPU, you can take a look at this doc page and try it out <a href="https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Fully Sharded Data Parallel(FSDP) — PyTorch Tutorials 1.12.1+cu102 documentation</a></p>
Data distribution across nodes on a cluster when training with DDP

username: Subhash
cooked: <p>Hi Everyone,</p>
<p>I have a question regarding the distribution of data samples across a multi-node GPU cluster, when training using DDP.</p>
<p>Do the data subsets assigned to the nodes remain same across iterations(epochs)?</p>
<p>Thanks<br>
Subhash</p>

username: wanchaol
cooked: <p>Logically when you training with DDP, the data subsets come to each node/rank are different even within the same epoch, but they are together forming a global batch of data inputs.</p>
Allocate model and data on GPU

username: Arupreza
cooked: <p>I shift my data and model on GPU but still it gives this error.</p>
<p>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument target in method wrapper_nll_loss_forward)</p>
<p>model = ConvNet().to(device)<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)</p>
<p>def batch_gd(model, criterion, optimizer, train_loader, test_loader, epochs):<br>
train_losses = np.zeros(epochs)<br>
test_losses = np.zeros(epochs)</p>
<pre><code>for it in range(epochs):
    t0 = datetime.datetime.now()
    train_loss = []
    for sets, labels in train_loader:
        labels = labels.type(torch.LongTensor)
        sets, targets = sets.to(device), labels.to(device)
    
        # Zero the parameter gradientd
        optimizer.zero_grad()

        # Forward pass
        outputs = model(sets)
        loss = criterion(outputs, labels)

        # Backwords and optimize
        loss.backward()
        optimizer.step()

        train_loss.append(loss.item())

    # Get train loss and test loss
    train_loss = np.mean(train_loss) 

    test_loss = []
    for sets, labels in test_loader:
        labels = labels.type(torch.LongTensor)
        sets, labels = sets.to(device), labels.to(device)
        outputs = model(sets)
        loss = criterion(outputs, labels)
        test_loss.append(loss.item())
        
    test_loss = np.mean(train_loss)
    
    # Save losses
    train_losses[it] = train_loss
    test_losses[it] = test_loss
    
    dt = datetime.datetime.now() - t0
    print(f'Epoch {it+1}/{epochs}, Train Loss: {train_loss:.4f}, \n Test Loss: {test_loss:.4f}, Duration: {dt}')
    
return train_losses, test_losses
</code></pre>

username: ptrblck
cooked: <p>You are pushing the <code>labels</code> tensor to the GPU and assign it to <code>targets</code> here:</p>
<pre><code class="lang-python">sets, targets = sets.to(device), labels.to(device)
</code></pre>
<p>However, later you are using <code>labels</code> (the CPU tensor) which yields this error:</p>
<pre><code class="lang-python">loss = criterion(outputs, labels)
</code></pre>
<p>Use <code>targets</code> or reassign <code>labels</code> to itself.</p>

username: Arupreza
cooked: <p>Thanks for your reply, this is my silly mistake. I am new in PyTorch.</p>
Normally training loss suddenly returns to random initialization level

username: jsttlgdkycy
cooked: <p>I’m training an over-parameterized SuperNet, which is a Res-UNet model for denoising task. The training goes well for the first 200,000 batches. The loss descends to a small value(about 80). However, it suddenly returns to the level right after random initialization(about 3000) in several batches, and keeps this level for thousands batches without descent. If I keep training, after tens of thousands of batches, the loss sometimes may descend again, or sometimes may change to NaN. I have tried grad_clip and small learning rate, but useless. Does anyone know what’s wrong with it <img src="https://discuss.pytorch.org/images/emoji/apple/dizzy_face.png?v=12" title=":dizzy_face:" class="emoji" alt=":dizzy_face:" loading="lazy" width="20" height="20"></p>
Fail to run torchscript.ptl on Android

username: andy_wong
cooked: <p>I want to deploy my yolov5 model to Android, I convert .pt to torchscript and then torchscript.ptl. I can run pt and torchscript file successfully on my computer. But after deploying it into Android, an error occurs. does anyone know the reason?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a851e78a86db252a9ca5cce5a1adc9eee15ec9e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3a851e78a86db252a9ca5cce5a1adc9eee15ec9e" title="3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a851e78a86db252a9ca5cce5a1adc9eee15ec9e_2_225x500.jpeg" alt="3" data-base62-sha1="8lGTLDwlyqfz0Bc5VZBP9gB6K06" width="225" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a851e78a86db252a9ca5cce5a1adc9eee15ec9e_2_225x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a851e78a86db252a9ca5cce5a1adc9eee15ec9e_2_337x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a851e78a86db252a9ca5cce5a1adc9eee15ec9e_2_450x1000.jpeg 2x" data-dominant-color="ECEEF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">1080×2400 176 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d37c26bf182944681dbe7317485b2d8546c16565.png" data-download-href="https://discuss.pytorch.org/uploads/default/d37c26bf182944681dbe7317485b2d8546c16565" title="螢幕擷取畫面 2022-08-22 102814"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d37c26bf182944681dbe7317485b2d8546c16565_2_690x151.png" alt="螢幕擷取畫面 2022-08-22 102814" data-base62-sha1="uaSJPUTkoNn8OCRDMmKchugKj8p" width="690" height="151" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d37c26bf182944681dbe7317485b2d8546c16565_2_690x151.png, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d37c26bf182944681dbe7317485b2d8546c16565.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d37c26bf182944681dbe7317485b2d8546c16565.png 2x" data-dominant-color="303131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">螢幕擷取畫面 2022-08-22 102814</span><span class="informations">1015×223 27.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to inspect quantization parameters in a traced TorchScript model?

username: kohenchia
cooked: <p>Hi, tracing a simple quantized model gives me a graph that looks like this:</p>
<pre><code class="lang-auto">graph(%self.1 : __torch__.___torch_mangle_2.MyCell,
      %x : Float(10, 4, strides=[4, 1], requires_grad=0, device=cpu)):
  %22 : __torch__.torch.nn.quantized.dynamic.modules.linear.Linear = prim::GetAttr[name="linear"](%self.1)
  %25 : __torch__.torch.nn.quantized.modules.linear.LinearPackedParams = prim::GetAttr[name="_packed_params"](%22)
  %26 : __torch__.torch.classes.quantized.LinearPackedParamsBase = prim::GetAttr[name="_packed_params"](%25)
  %27 : bool = prim::Constant[value=1](), scope: __module.linear # /Users/kohenchia/development/pytorch/.virtualenv/lib/python3.8/site-packages/torch/nn/quantized/dynamic/modules/linear.py:47:0
  %Y : Float(10, 7, strides=[7, 1], requires_grad=0, device=cpu) = quantized::linear_dynamic(%x, %26, %27), scope: __module.linear # /Users/kohenchia/development/pytorch/.virtualenv/lib/python3.8/site-packages/torch/nn/quantized/dynamic/modules/linear.py:47:0
  %29 : int = prim::Constant[value=6](), scope: __module.linear # /Users/kohenchia/development/pytorch/.virtualenv/lib/python3.8/site-packages/torch/nn/quantized/dynamic/modules/linear.py:54:0
  %30 : bool = prim::Constant[value=0](), scope: __module.linear # /Users/kohenchia/development/pytorch/.virtualenv/lib/python3.8/site-packages/torch/nn/quantized/dynamic/modules/linear.py:54:0
  %31 : bool = prim::Constant[value=0](), scope: __module.linear # /Users/kohenchia/development/pytorch/.virtualenv/lib/python3.8/site-packages/torch/nn/quantized/dynamic/modules/linear.py:54:0
  %32 : NoneType = prim::Constant(), scope: __module.linear
  %33 : Float(10, 7, strides=[7, 1], requires_grad=0, device=cpu) = aten::to(%Y, %29, %30, %31, %32), scope: __module.linear # /Users/kohenchia/development/pytorch/.virtualenv/lib/python3.8/site-packages/torch/nn/quantized/dynamic/modules/linear.py:54:0
  return (%33)
</code></pre>
<p>How do I inspect the quantization parameters for the weights in this model? The weights are statically quantized.</p>
Exploding loss in encoder/decoder model

username: caridina
cooked: <p>I’m trying to build a text to speech model in PyTorch using an encoder/decoder architecture on librispeech 100hr dataset. The model essentially takes in text and outputs a mel spectrogram but I’m facing an issue where my loss explodes on the 2nd to 3rd batch irrespective of batch size.</p>
<p>I’m probably doing something stupid in my trainer but I’m not sure what! Does anyone have any insight?</p>
<p>Minimal train.py - <a href="https://gist.github.com/michaelliangaustralia/1e231659e01ee858ad964e85abe0a060" rel="noopener nofollow ugc">gist</a></p>
<p>model.py - <a href="https://gist.github.com/michaelliangaustralia/ef5e3aed0aeebe7c2688ff301514ac9a" rel="noopener nofollow ugc">gist</a></p>
<p>This is the terminal output when trying to run the training file</p>
<pre><code class="lang-auto">loss tensor(0.2049, device='cuda:0', grad_fn=&lt;MSELossBackward0&gt;)
loss tensor(8.0709e+17, device='cuda:0', grad_fn=&lt;MSELossBackward0&gt;)
loss tensor(9.1250e+29, device='cuda:0', grad_fn=&lt;MSELossBackward0&gt;)
... [loss will be inf]
</code></pre>

username: ptrblck
cooked: <p>Your loss calculation seems to delete the intermediate <code>loss</code> tensor and just uses the one from the last iteration:</p>
<pre><code class="lang-python">        for b in range(BSIZE):
            loss = 0
            batch_ms_pred_list = []
            for i in tqdm(range(audio[b].shape[0])):
                ms_pred, dec_hidden, dec_cell = decoder_model(dec_input, dec_hidden, dec_cell, enc_outputs[b].unsqueeze(0))

                if random.uniform(0, 1) &lt; p_teacher_forcing: # teacher forcing
                    dec_input = audio[b][i].unsqueeze(0).to(device)
                else: # no teacher forcing
                    dec_input = ms_pred

                # append mspred
                batch_ms_pred_list.append(ms_pred)

            # accumulate loss
            batch_ms_pred_list = torch.stack(batch_ms_pred_list).squeeze()
            loss = loss_fn(batch_ms_pred_list, audio[b].to(device))
            print('loss', loss)
</code></pre>
<p>Is this on purpose? Also, could you try to overfit a small dataset (e.g. just 10 samples) and check if your model training would still explode?</p>
Casting to int of data min and max in HistogramObserver

username: MDzz
cooked: <p>Hello,<br>
taking a look at the histogram observer, I realized that the min,max values of the incoming data are cast to int when creating or interpolating the histogram (<a href="https://github.com/pytorch/pytorch/blob/a9ba3fe1dbf2cea45c9a7e723010c27c211f7fe3/torch/ao/quantization/observer.py#L1143" rel="noopener nofollow ugc">see this line</a>). I fail to see why that is necessary, or even a good idea.<br>
I tried out just the histogram interpolation algorithm implemented in the observer, and for low-standard deviation data it gives a pretty bad approximation of the combined histogram; however, if the casting to int is removed, the approximation is far better.</p>
<p>Can anyone explain to me why the casting occurs?</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/mdzz">@MDzz</a> , sounds like a potential bug.  I filed <a href="https://github.com/pytorch/pytorch/issues/83672" class="inline-onebox" rel="noopener nofollow ugc">quantization: unexpected casting of tensor min and max to int in histogram observer · Issue #83672 · pytorch/pytorch · GitHub</a> to track investigation and resolution, thanks for reporting this.</p>

username: MDzz
cooked: <p>HI Vasiliy, thank you for looking at this, happy to help:)</p>
Cannot matmul with 3D tensor `2D @ 3D`

username: tsp_t
cooked: <p>The result of [3, 3] shaped tensor matmul with [3, 16, 1080] was expected to yield [3, 16, 1080] shaped tensor.<br>
However, I get a runtime error.</p>
<p>Is this the wrong implementation? Or am I missing something?</p>
<pre><code class="lang-python">import numpy as np
import torch

X = torch.from_numpy(np.random.randn(3,3)).cuda().float()
Y = torch.from_numpy(np.random.randn(3,16,1080)).cuda().float()
print(X.shape, Y.shape)
print(X.dtype, Y.dtype)
X @ Y
</code></pre>
<pre><code class="lang-auto">torch.Size([3, 3]) torch.Size([3, 16, 1080])
torch.float32 torch.float32
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-14-c2c71b1687e7&gt; in &lt;module&gt;
      6 print(X.shape, Y.shape)
      7 print(X.dtype, Y.dtype)
----&gt; 8 X @ Y

RuntimeError: mat1 and mat2 shapes cannot be multiplied (3240x16 and 3x3)
</code></pre>

username: KFrank
cooked: <p>Hi tsp!</p>
<aside class="quote no-group" data-username="tsp_t" data-post="1" data-topic="159724" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tsp_t/48/47302_2.png" class="avatar"> tsp_t:</div>
<blockquote>
<p>The result of [3, 3] shaped tensor matmul with [3, 16, 1080] was expected to yield [3, 16, 1080] shaped tensor.<br>
However, I get a runtime error.</p>
</blockquote>
</aside>
<p><code>torch.matmul()</code> (<code>@</code>) doesn’t perform the matrix-tensor multiplication you<br>
might expect.  Instead, for these tensors, it tried to perform “batch matrix<br>
multiplication” (broadcasting the batch dimensions), and the dimensions<br>
don’t line up properly.</p>
<p>Probably the simplest approach is to use <code>einsum()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.12.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; X = torch.randn (3, 3)
&gt;&gt;&gt; Y = torch.randn (3, 16, 1080)
&gt;&gt;&gt;
&gt;&gt;&gt; prod = torch.einsum ('ij,jkl-&gt;ikl', X, Y)
&gt;&gt;&gt;
&gt;&gt;&gt; prod.shape
torch.Size([3, 16, 1080])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Unwrapping layers from a nested Sequential object with sub-Modules?

username: neel_g
cooked: <p>Supposing I have a complex model, built up of individual <code>Sequential</code>s with their own <code>Module</code>s. In such nested condition, I wish to extract a simple list of all individual layers with arbitrary depth. Entire network is sequential so the flow of tensors wouldn’t be interruped:</p>
<pre data-code-wrap="py"><code class="lang-nohighlight">blocks = [*newmodel.children()]
interm = [*blocks[0].children()]
#.....And we go on, iterating modules appending layers to list - getting their children() etc.
</code></pre>
<p>something like this is progress but is too cumbersome for complex nets.</p>
<p>I don’t really mind using this, but if anyone has a more neater solution, I’m all years <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/rocket.png?v=12" title=":rocket:" class="emoji" alt=":rocket:" loading="lazy" width="20" height="20"></p>

username: neel_g
cooked: <p>This is what Codex suggests:-</p>
<pre data-code-wrap="py"><code class="lang-nohighlight">def get_leaf_layers(m):
    children = list(m.children())
    if not children:
        return [m]
    leaves = []
    for l in children:
        leaves.extend(get_leaf_layers(l))
    return leaves
</code></pre>
<p>I wonder if anyone can come up with a neater, more elegant solution <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: raining_day513
cooked: <p>You should use <code>self.modules()</code></p>
Seq2seq: Replace the embeddings with pre-trained word embeddings such as word2vec

username: liran
cooked: <p>Hi,</p>
<p>I am following a seq2seq tutorial:<br>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></p>
<p>I want to use pre-trained vectors (Word2Vec) instead of word2index as we can see in the tutorial. I have edited the code to get the vector of the word rather than the index:</p>
<pre><code class="lang-auto">class Lang:
    def __init__(self, name):
        self.name = name
        self.word2index = {}
        self.word2count = {}
        self.index2word = {0: "SOS", 1: "EOS"}
        self.n_words = 2  # Count SOS and EOS

    def get_word2vec(self):
        word2vec = KeyedVectors.load_word2vec_format('Models/Word2Vec/wiki.he.vec')
        return word2vec
    
    def addSentence(self, sentence):
        for word in sentence.split(' '):
            self.addWord(word)

    def addWord(self, word):
        if word not in self.word2index:
            self.word2index[word] = self.get_word2vec[word]
            self.word2count[word] = 1
            self.index2word[self.n_words] = word
            self.n_words += 1
        else:
            self.word2count[word] += 1
</code></pre>
<p>the dimension size of this word2vec is 300 dimensions</p>
<ol>
<li>Is this the right way to do that?</li>
<li>May I need to change other things in my Encoder\decoder\ NN?</li>
</ol>
<p>Thank you!</p>

username: PANKAJ_CHANDRAVANSHI
cooked: <p>hey, i am also looking for the same solution. have you found any?</p>
RuntimeError: Interrupted system call when doing distributed training

username: Vedant_Roy
cooked: <p>Hi! I was wondering if anyone else has hit the following error when doing distributed training. My configuration is 8xA100 on a single node.</p>
<p>One thing to note is I hit this error with only 2 GPUs on a single node, but the error rate increases the more GPUs I have. This only happens in the initialization phase. E.g, once training properly starts the error never happens. I’m pretty sure it has something to do with the creation of the “C10d Store”.</p>
<pre><code class="lang-auto">  File "train_mae_2d.py", line 120, in train
    run_trainer(

  File "train_mae_2d.py", line 41, in run_trainer
    trainer = make_trainer(

  File "/home/ubuntu/video-recommendation/trainer/trainer.py", line 78, in make_trainer
    return Trainer(

  File "/home/ubuntu/miniconda/envs/video-rec/lib/python3.8/site-packages/composer/trainer/trainer.py", line 781, in __init__
    dist.initialize_dist(self._device, datetime.timedelta(seconds=dist_timeout))

  File "/home/ubuntu/miniconda/envs/video-rec/lib/python3.8/site-packages/composer/utils/dist.py", line 433, in initialize_dist
    dist.init_process_group(device.dist_backend, timeout=timeout)

  File "/home/ubuntu/miniconda/envs/video-rec/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 595, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)

  File "/home/ubuntu/miniconda/envs/video-rec/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 257, in _env_rendezvous_handler
    store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout)

  File "/home/ubuntu/miniconda/envs/video-rec/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 188, in _create_c10d_store
    return TCPStore(

RuntimeError: Interrupted system call
</code></pre>
<p>Cross-posted here: <a href="https://github.com/pytorch/pytorch/issues/83824" class="inline-onebox" rel="noopener nofollow ugc">RuntimeError: Interrupted system call when doing distributed training · Issue #83824 · pytorch/pytorch · GitHub</a></p>

username: H-Huang
cooked: <p>Hi, replied to the issue on github <a href="https://github.com/pytorch/pytorch/issues/83824" class="inline-onebox" rel="noopener nofollow ugc">RuntimeError: Interrupted system call when doing distributed training · Issue #83824 · pytorch/pytorch · GitHub</a></p>
Libtorch with cloud solutions?

username: Robin_Lobel
cooked: <p>Is it possible to use libtorch with cloud solutions such as MS Azure, Google Cloud or others ?<br>
I would hate having to use python for that <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

username: Robin_Lobel
cooked: <p>…or maybe I can just upload a set of data and my saved model and ask the remote servers to do the training ?</p>

username: ptrblck
cooked: <p>If you would like to train a model, the cloud providers should already provide containers with a reasonable setup to start your training.<br>
In that case, I would recommend the Python frontend or why would you like to use libtorch for training?</p>
<p>In the case of inference, I have to pass, as I don’t know what the best approach currently is.</p>

username: Robin_Lobel
cooked: <p>Because I have a long experience with C++, but none with Python, and my brief experience with Python and its packages everywhere makes me want to stay away from it as much as possible…</p>

username: umbriel
cooked: <p>Hi,</p>
<p>I work on RL trainer (virtual robots locomotion) written on top of libtorch/C++.<br>
My local machine’s power is not enough, I’m considering moving the training to a cloud but have no clue even how to approach it.<br>
Anyone had experience launching console executable with extensive CUDA usage on some cloud server?</p>
Why torch.distributions.multivariate_normal.MultivariateNormal requires positive-definite covariance_matrix rather than positive-semidefinite

username: geng_David
cooked: <p>When I use torch.cov to compute the covariance matrix of a batch of vector (in which the batch_size may be less than the vector length,) and then use the statistics to construct the MultivariateNormal distribution, it would raise a ValueError which indicates the covariance_matrix is not a positive-definite matrix.</p>
<pre><code class="lang-auto">import torch
from torch.distributions.multivariate_normal import MultivariateNormal

data = torch.rand(2,4)
data_mean = data.mean(0)
data_cov = torch.cov(data.T)
m = MultivariateNormal(data_mean, data_cov)
</code></pre>
<p>When data (NxD) with N&lt;D, this would happen since the covariance_matrix may be positive-semidefinite. I found that in NumPy, the MultivariateNormal distribution only requires positive-semidefinite but PyTorch need positive-definite covariance_matrix.</p>
<p>To achieve my goal, I need to transform the covariance_matrix to the numpy format for random sampling and then transform the sampled results back into PyTorch tensor.</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/geng_david">@geng_David</a> ,</p>
<p>A Gaussian with zero-width is ill-defined (as you’re effectively dividing by zero), hence positive definite is required and not positive semi-definite.</p>

username: geng_David
cooked: <p>Thanks for your reply.  I understand that it would be an ill-defined normal distribution if cov-mat is positive semi-definite. I think whether providing an option like <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.multivariate_normal.html" rel="noopener nofollow ugc">scipy</a> does could be better in the case when I only need an approximation solution.</p>

username: AlphaBetaGamma96
cooked: <p>You can always add a small finite value along the diagonal to make it positive definite in the meantime. However, you can always open an <a href="https://github.com/pytorch/pytorch/issues" rel="noopener nofollow ugc">issue</a> and request that feature be added.</p>

username: geng_David
cooked: <p>I see. Thank you so much.</p>
Architercture RAFT in C++

username: ibragimov
cooked: <p>Can you please tell me if there is an implementation of the RAFT architecture in C ++? I need to embed an implementation on iOS. In the standard library that comes via cocoapods, I could not find an implementation of this architecture. The implementation of RAFT written is in Python. Do I need to convert RAFT from Python to C++ or is there already a solution?</p>
Improving accuracy of 3d image classification model

username: maryma
cooked: <p>I made an image classification model for image feature extraction. My model is a 3d model. Because I couldn’t find any 3d pretrained model. I trained the model from scratch. First I didn’t get good results then I tried to use data augmentation and now I got results that are around 65 percent but I needed higher accuracy. Do you have any idea that how can I improve accuracy results in 3d image classification?<br>
The inputs to my model are Kidney MRI. I tried to find a dataset that has a lot of 3d medical images and use transfer learning but I couldn’t find a huge dataset like imagenet. Is there any way to use image net in 3d models. Or is there any way to load weights of 2d models to 3d or vice versa.<br>
If anyone has experience in this regard, please help and share your ideas.</p>
Is distributed model performance somewhat in proportion to non-distributed model performance?

username: Messiah
cooked: <p>Hi,</p>
<p>When using the <code>torch.nn.parallel.DistributedDataParallel</code> for multi-process training, should the performance (e.g. loss) be somewhat proportionally better (where the proportion depends on the number of processes) to the performance in a single-process setting? My intuition is that since the dataset in a multi-process setting is subdivided to the individual processes, the model will have an easier time predicting these partitioned datasets and hence have better training metrics compared to when the entire dataset is predicted in a single-process setting. Can someone confirm if my intuition is correct?</p>

username: Yanli_Zhao
cooked: <p>the distributed model usually requires communication to sync model’s gradients and thus usually it is not linear scaling compared to local training</p>
Load GPU trained model to CPU

username: sakh251
cooked: <p>I trained a model on google colab with PyTorch version 1.12. trying to load it on a cpu machine. I got this error:</p>
<pre><code class="lang-auto">RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
</code></pre>
<p>I used this:</p>
<pre><code class="lang-auto">torch.load('model.pth', map_location=torch.device('cpu'))
</code></pre>
<p>I used also pickle:</p>
<pre><code class="lang-auto">class CPU_Unpickler(pickle.Unpickler):
    def find_class(self, module, name):
        if module == 'torch.storage' and name == '_load_from_bytes':
            return lambda b: torch.load(io.BytesIO(b), map_location='cpu')
        else:
            return super().find_class(module, name)
model = CPU_Unpickler(f).load()

</code></pre>
<p>now new error:</p>
<pre><code class="lang-auto">    model = CPU_Unpickler(f).load()
_pickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.
</code></pre>
<p>Thank you.</p>

username: ptrblck
cooked: <p>The error can be raised if you are trying to <code>torch.load</code> an unsupported type, such as a PNG image:</p>
<pre><code class="lang-python">torch.load("./image.png")
# UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.
</code></pre>

username: sakh251
cooked: <p>The problem was solved, I did the mapping in model.load_state_dict.</p>
<pre><code class="lang-auto">model.load_state_dict(torch.load(f"{path}), map_location=torch.device(device=device))
</code></pre>
<p>it works with:</p>
<pre><code class="lang-auto">model.load_state_dict(torch.load(f"{path}", map_location=torch.device(device=device)))
</code></pre>
What is the Search direction term in Adam optimizer?

username: BobKim
cooked: <p>Hello!<br>
I’m a graduate student studying deep learning optimizer.<br>
In general, numerical optimization algorithms can be divided into search direction term and determination step size term. But looking at Adam and other optimizer papers, there doesn’t seem to be a clear distinction.</p>
<p>In Adam optimizer, Is mt (first moment vector) the direction? if not, Which term represents direction?</p>
How to use torchlib in android?

username: dbsx
cooked: <p>I know there is a <a href="https://pytorch.org/mobile/home/" rel="noopener nofollow ugc">section</a> introducing how to take pytorch in mobile, but it seems using the python version directly.  In my case, I need to create an efficient cross-platform app (over windows, and android). So I decided to use torchlib directly.  From <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">here</a>, seems currently only windows and linux are supported. Does it mean that I can’t use binary torch library in android?</p>
Expected Result of SWA in Low Precision Image Classification

username: nisl
cooked: <p>Hi All,</p>
<p>I am trying to implement SWA for low precision learning, very similar to what has been done in <a href="https://github.com/stevenygd/SWALP/blob/master/train.py" rel="noopener nofollow ugc">SWALP</a> with BFP. However, since SWA is now already available in pytorch, I won’t have to make it from scratch, so I used AveragedModel, along with SWALR scheduler.<br>
However, as I am a bit of a beginner, I am having some doubts which I was hoping someone could clarify:</p>
<p>(a) Can I use the existing SWA functions in pytorch for low precision use cases too, or should I go for something that has been done in the SWALP repo (linked above)?</p>
<p>(b) I incorporated SWA into my Qpytorch code almost exactly the same way as the example code given by the author of SWA in this <a href="https://github.com/izmailovpavel/torch_swa_examples/blob/master/train.py" rel="noopener nofollow ugc">repo</a> , but am having a bit of a hard time understanding if SWA is making much difference in my case. Could you please tell me what to check to understand if its doing much improvements? I am training VGG on CIFAR10 but can see that final test accuracy values are not signficantly higher with SWA than without as I was expecting.<br>
So I am trying to view the weights of the AveragedModel after update_parameters to see what SWA is doing. For testing purposes, I have a single fully connected layer network and an arbitrary 3x3 input matrix. I am a bit unsure as to how exactly to hand calculate the answers for weight updates using SWA so was hoping someone could guide me on how I could test my code to know that I have used SWA correctly.</p>
<p>I realise my questions are a bit general and maybe has obvious answers but I am having a bit of a hard time finding relevant answers so would really appreciate any help. Thank you!</p>
Convert tensorflow_probability.distributions.TransformedDistribution to pytorch

username: ZSheikhb
cooked: <p>I am trying to convert a tensorflow script to pytorch. I will appreciate if someone can explain what would be the equivalent piece of code for the following line from <a href="https://github.com/progrmanial/Google-AI-Research/blob/8da9f60982aefdc020cfcf25292d8c2a8eda60ed/behavior_regularized_offline_rl/brac/networks.py" rel="noopener nofollow ugc">this script</a></p>
<pre><code class="lang-auto">import tensorflow_probability as tfp
tfd = tfp.distributions
a_distribution = tfd.TransformedDistribution(
        distribution=tfd.Normal(loc=0.0, scale=1.0),
        bijector=tfp.bijectors.Chain([
            tfp.bijectors.AffineScalar(shift=self._action_means,
                                       scale=self._action_mags),
            tfp.bijectors.Tanh(),
            tfp.bijectors.AffineScalar(shift=mean, scale=std),
        ]),
        event_shape=[mean.shape[-1]],
        batch_shape=[mean.shape[0]])
</code></pre>
<p>Thanks</p>

username: ZSheikhb
cooked: <p>I converted the above part in pytorch as following</p>
<pre><code class="lang-auto">a_distribution = TransformedDistribution(
                        base_distribution=Normal(loc=torch.full_like(mean, 0), 
                                                 scale=torch.full_like(mean, 1)), 
                        transforms=tT.ComposeTransform([
                                   tT.AffineTransform(loc=self._action_means, scale=self._action_mags, event_dim=mean.shape[-1]), 
                                   tT.TanhTransform(),
                                   tT.AffineTransform(loc=mean, scale=std, event_dim=mean.shape[-1])]))
</code></pre>
<p>But <code>TanhTransform</code> caused a <code>nan</code> value. I am wondering whether I have done this incorrectly or not?</p>
How to pass a "self" parameter?

username: Buzz
cooked: <p>Hi<br>
I got struck at a point in my code(which might seems naive to others ) where I would like to pass a self parameter. The sample model structure is following:</p>
<pre><code class="lang-auto">class A(nn.Module):
    def __init__(selft):
      super(A, self).__init__()
      self.fc = nn.Linear(768*1, 1)    
      self.r=1.0      
    def forward(self, in):   
      fc_output=self.fc(in)-self.r
      return fc_output,self.r
    def updateR(r):
     r=r+1
</code></pre>
<pre><code class="lang-auto">def train():
    model.train()
    model.zero_grad()
    output_,r= model(inp)
    ##a code will be here to update the r. How could I update the r parameter from here??
    ## updateR=(r)
    loss = someLossFunction
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
</code></pre>
<p>The sample driver code is following:</p>
<pre><code class="lang-auto">inp=someInput
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = A()
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)
epochs = 1
current = 1
while current &lt;= epochs:    
    train()
    current = current + 1
</code></pre>
<p>My objective is to find out a way to pass the self parameter in the train() to call the updateR() to update r. When I have tried doing in the following way I experienced error:</p>
<p><code>model.train()</code><br>
and<br>
<code>def train(self):</code></p>
<p>Could you please advise me how could I pass the self parameter in the train method.</p>

username: ptrblck
cooked: <p>The <code>self</code> attribute refers to the module itself and is used to define a function as a member method.<br>
If you want to manipulate <code>r</code> in <code>updateR</code>, use:</p>
<pre><code class="lang-python">def updateR(self):
    self.r = self.r + 1
</code></pre>

username: Buzz
cooked: <p>ptrblck Hi</p>
<p>Actually, I got struck at how to call this updateR() from inside the train(). Should I pass the self parameter in the train()?</p>

username: ptrblck
cooked: <p>After you’ve defined <code>updateR</code> as a member function of the model:</p>
<pre><code class="lang-python">class A(nn.Module):
    def __init__(selft):
      super(A, self).__init__()
      self.fc = nn.Linear(768*1, 1)    
      self.r=1.0      
    def forward(self, in):   
      fc_output=self.fc(in)-self.r
      return fc_output,self.r
    def updateR(self):
      self.r = self.r + 1
</code></pre>
<p>you can call it via:</p>
<pre><code class="lang-python">def train():
    model.train()
    model.zero_grad()
    output_,r= model(inp)
    mode.updateR()
</code></pre>
Providing Google Drive link as a data path - Google Colab

username: sara
cooked: <p>Hi!<br>
I am trying to reproduce the results of someone else’s code using Google Colab but the data is huge. The authors have provided the data in a shared Google Drive folder. Is it possible to specify data path as Google Drive link instead of downloading their data and then uploading in my Google Drive?</p>
<p>I have provided the link as a path but I am getting this error: (please correct me if I am wrong and the link can’t be provided as a path)</p>
<pre><code class="lang-auto">Warning: logging configuration file is not found in logger/logger_config.json.
LOG: No saved data found. Making new data directory https://drive.google.com/drive/folders/abc?usp=sharing
Traceback (most recent call last):
  File "/content/abc/train.py", line 84, in &lt;module&gt;
    main(config)
  File "/content/abc/train.py", line 31, in main
    train_data_loader = config.init_obj('train_data_loader', module_data)
  File "/content/abc/parse_config.py", line 103, in init_obj
    return getattr(module, module_name)(*args, **module_args)
  File "/content/abc/data_loader/data_loaders.py", line 32, in __init__
    os.mkdir(data_split_lists_path)
FileNotFoundError: [Errno 2] No such file or directory: 'https://drive.google.com/drive/folders/abc?usp=sharing'
</code></pre>
<p>Could someone please tell me what am I doing wrong here?<br>
Thank you!</p>

username: ptrblck
cooked: <p>I guess you should be able to use <code>pydrive</code> (or any other Python library handling Google Drive) and load the files from Google Drive with it.</p>

username: sara
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for the reply! I have figured out another way of doing this: I created a shortcut in my drive and the copied the path in Colab. It turned out that the error was due to some empty folder.<br>
Yet, I still can’t figure out, why am I getting this <code>Warning: logging configuration file is not found in logger/logger_config.json</code> even though logger_config.json is there.</p>
GPU training Error

username: asakal
cooked: <p>Hi<br>
I am training a network on M1 mackbook pro. I installed the compatible torch version for mps device.<br>
I have a 3 channel image with shapes from 2000 x 2000 and above. At first i resized them to 200x200, the model trained fine. As soon as i changed the shapes to 500x500 the model training keeps giving out the <code>Error: command buffer exited with error status.</code> error at each epoch.</p>
<p>What may be the cause of this error ?</p>
<pre><code class="lang-auto">2022-08-21 13:27:45.597158 Epoch 1 Training Loss 44.01341743328992

Error: command buffer exited with error status.
	The Metal Performance Shaders operations encoded on it may not have completed.
	Error: 
	(null)
	Caused GPU Hang Error (00000003:kIOGPUCommandBufferCallbackErrorHang)
	&lt;AGXG13XFamilyCommandBuffer: 0x175ba5a50&gt;
    label = &lt;none&gt; 
    device = &lt;AGXG13XDevice: 0x144fd3e00&gt;
        name = Apple M1 Pro 
    commandQueue = &lt;AGXG13XFamilyCommandQueue: 0x144fd5600&gt;
        label = &lt;none&gt; 
        device = &lt;AGXG13XDevice: 0x144fd3e00&gt;
            name = Apple M1 Pro 
    retainedReferences = 1```</code></pre>
Accepted image formats for Captum Feature Attribution

username: anmolseth00
cooked: <p>The Captum tutorials all seem to be using .jpg images for feature attribution. Are other image formats accepted with their tutorial notebooks? If so, where can I find a list of accepted image formats? I wasn’t able to find anything in their documentation.</p>

username: dbish
cooked: <p>Can you clarify which tutorial and lines you’re talking about and whether it’s the input in the tutorials or output for attribution?</p>
<p>If you are referring to the input images used like in this tutorial: <a href="https://captum.ai/tutorials/Resnet_TorchVision_Interpret" class="inline-onebox" rel="noopener nofollow ugc">Captum · Model Interpretability for PyTorch</a> where the input image is set as <code>img = Image.open('img/resnet/swan-3299528_1280.jpg')</code>. There is no restriction that it has to be a jpg, since the actual model is looking for a tensor, and the image transormations in the notebook just require a PIL Image type. PIL itself supports many image formats so if you swapped that line out for a png file or other type, there shouldn’t be an issue: <a href="https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html" class="inline-onebox" rel="noopener nofollow ugc">Image file formats - Pillow (PIL Fork) 9.2.0 documentation</a>.</p>
<p>If you are referring to the output attribution figures, those aren’t jpegs, they are of type  <a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.figure.html" class="inline-onebox" rel="noopener nofollow ugc">matplotlib.pyplot.figure — Matplotlib 3.5.3 documentation</a>. From that tutorial that would be like this line <code>_ = viz.visualize_image_attr(np.tran...</code> It outputs the attribution visualization and the input images in the args are numpy arrays so no specific image formats required there, you can read more on the method signature here: <a href="https://captum.ai/api/utilities.html" class="inline-onebox" rel="noopener nofollow ugc">Captum · Model Interpretability for PyTorch</a>.</p>