akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Confused about the way self-attention is calculated in nn.Transformer

meteorlin: <ol>
<li>I noticed that <code>nn.TransformerEncoderLayer</code> and <code>nn.TransformerDecoderLayer</code> will take the first dimention when calculating self-attention. Does it have any specific meaning? According to my understanding, shouldn’t it be to keep all the dimentions?</li>
<li>In addition, assuming that this operation has practical significance, when <code>batch_first</code> is <code>True</code> or <code>False</code> , the meanings of the dimention obtained by <code>[0]</code> should be different (indicating <code>batch</code> and <code>seq</code> respectively), so is there any ambiguity?</li>
</ol>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/a6d72f44a4e8b6e9d2e878f30fd8b1d3e1197f0e/torch/nn/modules/transformer.py#L544">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/a6d72f44a4e8b6e9d2e878f30fd8b1d3e1197f0e/torch/nn/modules/transformer.py#L544" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/a6d72f44a4e8b6e9d2e878f30fd8b1d3e1197f0e/torch/nn/modules/transformer.py#L544" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/a6d72f44a4e8b6e9d2e878f30fd8b1d3e1197f0e/torch/nn/modules/transformer.py#L544</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="534" style="counter-reset: li-counter 533 ;">
          <li>            x = self.norm2(x + self._ff_block(x))</li>
          <li>
          </li>
<li>        return x</li>
          <li>
          </li>
<li>    # self-attention block</li>
          <li>    def _sa_block(self, x: Tensor,</li>
          <li>                  attn_mask: Optional[Tensor], key_padding_mask: Optional[Tensor]) -&gt; Tensor:</li>
          <li>        x = self.self_attn(x, x, x,</li>
          <li>                           attn_mask=attn_mask,</li>
          <li>                           key_padding_mask=key_padding_mask,</li>
          <li class="selected">                           need_weights=False)[0]</li>
          <li>        return self.dropout1(x)</li>
          <li>
          </li>
<li>    # feed forward block</li>
          <li>    def _ff_block(self, x: Tensor) -&gt; Tensor:</li>
          <li>        x = self.linear2(self.dropout(self.activation(self.linear1(x))))</li>
          <li>        return self.dropout2(x)</li>
          <li>
          </li>
<li>
          </li>
<li>class TransformerDecoderLayer(Module):</li>
          <li>    r"""TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network.</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Retrieve intermediary value from custom backward()

hoelzerC: <p>Hi there,<br>
is it possible to retrieve an intermediary result from a custom backward pass?</p>
<p>Basically in a user-defined <code>autograd.Function</code> an intermediary value is calculated and then contracted (e.g. via <code>sum</code>) to obtain the correct backwards gradient. Since the intermediary value is also needed for other calculations besides the gradient, it would be elegant to be able to store and reuse it.</p>
<pre><code class="lang-auto">import torch

class DummyFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inp):

        factor = torch.tensor([[[1.0, 2.0], [3.0, 4.0]], [[5.0, 6.0], [7.0, 8.0]]])
        return factor * inp

    @staticmethod
    def backward(ctx, bkw):

        def dummy(tensor):
            # do sth.
            return tensor

        # how to retrieve this
        intermediary = dummy(bkw)

        res = torch.sum(intermediary, (-2, -1)).unsqueeze(-1)
        return res

inp = torch.tensor([[1.0], [2.0]], requires_grad=True)
function = DummyFunction.apply
out = function(inp)
gradient = torch.autograd.grad(outputs=out.sum(), inputs=inp)[0]
</code></pre>
Conv3d for image time-series forecasting

Didem_Durukan: <p>Hello,</p>
<p>I have an image time series forecasting which I think can be done via conv3d. However I am confused about the input shape:</p>
<p>In the documentation I saw input shape to be: (N,C,D,H,W).</p>
<ul>
<li>
<p>For my case H and W are both 256</p>
</li>
<li>
<p>C is the number of channels</p>
</li>
<li>
<p>N is the batch size (number of samples per batch)</p>
</li>
</ul>
<p>However, I am confused about the parameter D. What does it stands for? And how can I define it in my case?</p>
<p>Thanks in advance!</p>

ptrblck: <p><code>D</code> refers to the “depth” of the volume you are using, since <code>nn.Conv3d</code> expect volumes as inputs instead of 2D planes/images. The filter kernel of the <code>Conv3d</code> layer will also have an additional depth dimension and the convolution will be applied on all 3 dimensions (i.e. the filter is moving in all 3 dims).</p>

Didem_Durukan: <p>So in 3d convolution, D is always 3? Or what should I think of while deciding what D should be?</p>

ptrblck: <p>No, the depth can be any value as the other height and width dimensions.<br>
Think about the depth as e.g. the stack of medical images (e.g. CT scans).<br>
The height and width defines the “image” dimensions of each CT slice while the depth is defining the number of slices. In this case the volume would contain “'voxels” instead of pixels, since you are now using a volume to represent the scan.</p>

Didem_Durukan: <p>Ahaaa, okay so if I have 15 images depth is 15, and if they are defined as R,G,B channel is = 3 ?</p>

ptrblck: <p>Yes, each “volume” (or stack of images) can have multiple channels. For a “color volume” the channel dimension would be 3.</p>
Memory Utilization by batches in Dataloader Iterator

keshavvinayak01: <p>I’m running the preprocessing pipeline online by using a Dataloader over a self-defined Dataset, which performs some pre-processing operations defined in the <strong>getitem</strong> function.</p>
<p>I observed a large memory utilization by the batches, so I was curious if it is the preprocessed batches that are not sent to the GPU are the ones responsible for this memory bloat. How do I explicitly signal to remove this batches from memory?</p>
<p>I know I can use the delete keyword and rely on the Automatic Garbage Collector to do this(Tried it out, doesn’t affect the huge memory footprint), but is there some torch API that can help me instantly remove a preprocessed batch from the memory?</p>
<p><strong>Edit:</strong> Also, I’m not consuming the batches into an ML model on the GPU/CPU. A simple iterator over all batches without any further processing done on the emitted batches.</p>
<p>Thanks</p>
Running two separate jobs on same gpu server

mendelsontau: <p>Hi,<br>
I’m running distributed training with torchrun.<br>
I have 8 V100 GPUs on some server and I am trying to run two separate jobs (same training with different hyperparameters) such that one job will use GPUs 0-3 and the other GPUs 4-7.<br>
What is the right way to do this?<br>
Thanks</p>

ptrblck: <p>You can launch two jobs via <code>torchrun</code> by masking the corresponding devices via:</p>
<pre><code class="lang-python">CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun ...
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun ...
</code></pre>
<p>and set <code>--nproc_per_node 4</code> in both runs.</p>

mendelsontau: <p>Thanks for the prompt response.<br>
When following this i’m getting on the second run<br>
RuntimeError: The server socket has failed to listen on any local network address. The server socket has failed to bind to [::]:29500 (errno: 98 - Address already in use). The server socket has failed to bind to 0.0.0.0:29500 (errno: 98 - Address already in use).</p>

ptrblck: <p>Try to set the <code>HOST_NODE_ADDR</code> to a different port as described <a href="https://pytorch.org/docs/stable/elastic/run.html">here</a>.</p>
Why is the output of my model going to zero after a few hundred steps? Is this exploding gradients?

Alio: <p>After a few hundred steps the output of the model(s) go to zero. What are the possible causes? Is it exploding gradients?</p>
<p>My algorithm is more than 1000 lines of code so don’t know if it would be welcome here and whether I should focus on one problem. Might it be exploding gradients?</p>
<p>I have tried to clip the gradients in several ways. I don’t know if any of them work because there seems to be no difference in the ‘zeroing’ problem whether any of these attempts at clipping the gradient in the code or commented out.</p>
<pre><code>        loss_l.backward();
        torch::nn::utils::clip_grad_norm_(actor.parameters(), 0.4);

        // torch::nn::utils::clip_grad_norm_(actor_optimizer.parameters(), 0.1);
        // for (auto&amp; param : actor.parameters()) {
        //     torch::nn::utils::clip_grad_norm_(param, 0.1);
        // }
        
        // torch::nn::utils::clip_grad_norm_(actor.parameters(), 0.5);

        // for (auto&amp; param_group : actor_optimizer.param_groups()) {
        //     for (auto&amp; param : param_group.params()) {
        //         torch::nn::utils::clip_grad_norm_(param, 0.03);
        //     }
        // }

        // for (auto&amp; param : actor.parameters()) {
        //     torch::nn::utils::clip_grad_norm_(param, 1.1);
        // }

        actor_optimizer.step();
</code></pre>

ptrblck: <p>It’s hard to tell if your gradients are exploding, so I would recommend to check the norm or (some) gradients to see if their magnitude increases during the training.</p>
How to calculate second derivative for mixed precision

soonchangAI: <p>How to calculate second derivative gradients for mixed precision. I try calling backward() twice but it gave the same gradients as first derivative.</p>
<pre><code class="lang-auto">self.scaler.scale(loss).backward()
self.scaler.scale(loss).backward()
</code></pre>

ptrblck: <p>Calling <code>backward</code> multiple times will calculate the same gradients and accumulate them.<br>
You would need to backward the derivatives again as described in <a href="https://discuss.pytorch.org/t/second-order-derivatives-of-loss-function/71797/4">this post</a>.</p>
Save torch tensor in npz files

We1: <p>Hello. Though it is a little bit weird, I am wondering if saving the torch tensors (not the model weights) as numpy’s npz files will have any potential problems? What about the <em>map_location=None</em> setting if the torch tensor saved is on GPU?</p>
<p>I’ve tried to save it in npz file and load with np.load and it seems the tensor is correct (the same if I save with torch.save), even though I don’t even import torch when I np.load the saved tensor. Thank you so much in advanced.</p>
Run multi-node training inside docker

CDhere: <p>Hi! I have some questions regarding the recommended way of doing multi-node training from inside docker. Concretely, all my experiments are run in a docker container on <strong>each node</strong> and it is straightforward with <code>torch.distributed.launch</code> or <code>torchrun</code> when I only need distributed training on a single-node. However:</p>
<ol>
<li>if I need multi-node training, can I simply call <code>torch.distributed.launch</code> inside each docker separately, the same as without using docker?</li>
<li>What settings do I need when building the docker image and bringing up the docker container (e.g, network, etc)?</li>
<li>Any other recommended practice when working with docker?</li>
</ol>
<p>Thanks a lot!</p>

pritamdamania87: <aside class="quote no-group" data-username="CDhere" data-post="1" data-topic="167537">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cdhere/48/29866_2.png" class="avatar"> CDhere:</div>
<blockquote>
<ul>
<li>if I need multi-node training, can I simply call <code>torch.distributed.launch</code> inside each docker separately, the same as without using docker?</li>
</ul>
</blockquote>
</aside>
<p>This should work as long as you setup the network connectivity across docker containers appropriately such that they can talk to each other.</p>

CDhere: <p>Thank you! I just tried and it works same as usual as long as docker network is set up properly. I used <code>--network=host</code> when launching the container which can make it even more seamless.</p>
Return list of future where callback or result set inside async block

spyroot: <p>Hi Folks,</p>
<p>Can I do return list of future to a master node meanwhile on workers perform async ? On a worker, I have foo and master issue rpc_async by some reason it doesn’t</p>
<pre><code class="lang-auto">ob_rref.rpc_async().foo(self.agent_rref, n_steps, meta_tasks)
</code></pre>
<pre><code class="lang-auto">@rpc.functions.async_execution
def foo(self, agent_rref, n_steps, task_id):

       futures = []
        for i in range(self.num_task):
            print(f"Create feature {i}")
            fut = torch.futures.Future()
            futures.append(fut)

        self.loop.run_until_complete(self.bar(futures))
        return torch.futures
</code></pre>
<p>Bar run in own loop, since if I undestood mp.span create either fork or thread but in all cases it separate thread so loop on each worker are different.</p>
<pre><code class="lang-auto">async def bar(self, futures):
      # here I set each rpc future that I returned.
      futures[i].set_result(Something)

      # i.e I have

     while True:
        async_future = await queue.get()
        if async_future is None:
            break

       futures[i].set_result(async_future)
</code></pre>
Transformer with masking returns NaN values

Jeremy_Ma: <p>I have the following code in C++ to use masking for a TransformerEncoder</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;
#include &lt;vector&gt;

struct TestTransformer : torch::nn::Module {
    TestTransformer() {
        transEncoder = register_module("transEncoder", torch::nn::TransformerEncoderImpl(torch::nn::TransformerEncoderOptions(torch::nn::TransformerEncoderLayerOptions(420, 10), 4)));
    }

    torch::Tensor forward(torch::Tensor x, torch::Tensor m1, torch::Tensor m2) {
        return transEncoder-&gt;forward(x, m1, m2);
    }

    torch::nn::TransformerEncoderImpl transEncoder;
};

int main() {
    torch::Device device("cuda:0");
    TestTransformer m;
    m.to(device);
    torch::Tensor t = torch::randn({10, 1, 420});
    t = t.to(device);

    torch::Tensor mask2 = torch::tensor({1, 1, 1, 1, 1, 0, 0, 0, 0, 0}, {1, 10});
    mask2 = mask2.to(device);

    torch::Tensor out2 = m.forward(t, nullptr, mask2);
    std::cout &lt;&lt; out2.cpu().to(torch::kFloat) &lt;&lt; '\n';
    std::cout &lt;&lt; torch::any(torch::isnan(out2.cpu())).to(torch::kByte) &lt;&lt; '\n';
}
</code></pre>
<p>I get the following error</p>
<pre><code class="lang-auto">expected mask dtype to be Bool but got Float
</code></pre>
<p>However, if I change the type to bool, I get NaNs as my output.</p>
<p>This only happens if I use GPU but not CPU.</p>
<p>Is there a fix for this?</p>
Statistics tracking with distributed data parallel

jwillette: <p>I am wondering if there is a canonical way to track statistics when using distributed data parallel? For example, say I wanted to track the losses across all of the distributed instances, how can I gather the losses in one place given that they are all running in separate processes?</p>
<p>The only way I can think to do it would be to write them all to file along with information about the epoch/iteration, etc. and then combine them later.</p>
<p>Is there a better way to do this?</p>

pritamdamania87: <p>You can use the <code>gather</code> primitive from ProcessGroup to do this: <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.gather" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.13 documentation</a></p>

jwillette: <p>Thanks, this is exactly what I was looking for.</p>
What's the difference of PyTorch 2.0 and 1.14

keineahnung2345: <p>Hi, just one quick question.</p>
<p>I’ve built PyTorch master branch on 2022/12/5 and I expect its version is 2.0, but what I found is 1.14.0a0.<br>
Also in the <a href="https://download.pytorch.org/whl/nightly/torch/" rel="noopener nofollow ugc">link</a> provided by <a href="https://pytorch.org/get-started/pytorch-2.0/" rel="noopener nofollow ugc">pytorch 2.0 installation instruction page</a>, it only provides wheel like <code>torch-1.14.0.dev20221007+cpu-cp310-cp310-linux_x86_64.whl</code>.<br>
So where can I get 2.0? Or 2.0 is just an alias of 1.14?</p>
<p>Thanks in advance.</p>

ptrblck: <p>Yes, I would assume <code>1.14.0+commit</code> will be bumped into <code>2.0</code> eventually (I believe the same happened with <code>0.5.0+commit</code> and <code>1.0.0</code>).</p>

keineahnung2345: <p>Thanks for your reply!</p>
`Invalid scalar type` when dist.scatter boolean tensor

Rhett-Ying: <p>I am trying to scatter boolean tensor but failed with error: <code>Invalid scalar type</code> while it works well with <code>dtype=uint8</code>. could anyone help on this?</p>
<p>torch version: <code>1.13.0+cu117</code></p>
<pre><code class="lang-auto">def worker(rank, world_size):
    ...
    dtype = th.uint8
    dtype = th.bool
    scatter_list = [ th.tensor([True, True, True, True], dtype=dtype) for _ in range(4) ]
    gather_list = [ th.tensor([False, False, False, False], dtype=dtype) for _ in range(4) ]
    for i in range(world_size):
        dist.scatter(gather_list[i], scatter_list if i == rank else [], src=i)
</code></pre>
<pre><code class="lang-auto">File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 2290, in scatter
    work.wait()
RuntimeError: Invalid scalar type
</code></pre>

Rhett-Ying: <p>This error occurs with other calls.</p>
<pre><code class="lang-auto">import os
import torch
from torch import distributed as dist
import torch.multiprocessing as mp

def worker(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    device = 'cpu' #'cuda'
    dist.init_process_group('gloo' if device == 'cpu' else 'nccl', rank=rank, world_size=world_size)

    dtype = torch.bool
    #dtype = torch.uint8
    tensor = torch.tensor([0,1], dtype=dtype)
    dist.all_reduce(tensor)

if __name__ == '__main__':
    mp.spawn(worker, args=(2,), nprocs=2, join=True)
</code></pre>

Rhett-Ying: <p>could someone help on this?</p>

pritamdamania87: <p>This looks like a bug, can you report the issue on github: <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a></p>

Rhett-Ying: <p>reported: <a href="https://github.com/pytorch/pytorch/issues/90245" class="inline-onebox" rel="noopener nofollow ugc">[Distributed] `Invalid scalar type` when `dist.scatter()` boolean tensor · Issue #90245 · pytorch/pytorch · GitHub</a></p>
Measure inference time

zoythum: <p>I have two models that perform the same operation, one uses a library that executes sparse convolution on gpu while the other is a standard pytorch model running on cpu. I would like to analize the inference time to understand if there is an improvement in using the GPU and sparse convolution, what is the best way to do so? On cpu I usually use datetime before and after executing the model on a batch of images but it might not be the best practice.</p>

tiramisuNcustard: <p>A simple approach would be to print datetime right before training starts on the GPU and print datetime once again right after training ends. You can also choose to save the datetime objects and determine the delta_t. See <a href="https://www.geeksforgeeks.org/python-difference-between-two-dates-in-minutes-using-datetime-timedelta-method/" rel="noopener nofollow ugc">https://www.geeksforgeeks.org/python-difference-between-two-dates-in-minutes-using-datetime-timedelta-method/</a></p>

zoythum: <p>This is indeed the approach I took with the CPU model even if I’m sure it is not the best practice. The problem with the GPU one although is that multi threading requires synchronization and simply recording the time before and after the execution of the model might not yield the correct result. I am still searching for the correct way to approach this problem. Thank for your response!</p>

J_Johnson: <p>You said you want to compare inference time. Inference begins when data enters the forward pass and ends when it exits the forward pass.</p>
<pre><code class="lang-auto">def forward(self, x)
    starttime = time.time()
    x = layers(x)
    print(time.time() - starttime)
    
    return x
</code></pre>
Conv1D: how to filter multiple time series data using one single kernel?

LongWU: <p>Dear all,<br>
I have a time series data, recorded by 10 sensors, in shape [32,10,500] (batch size, sensor number, time points). I would like to use the same 1d kernel to filter these 10 sensor data (just like the wavelet filter), like below:</p>
<blockquote>
<p>input=torch.ones((32,10,1,500))<br>
m=nn.Conv2d(in_channels=10, out_channels=10, kernel_size=(1, 21), padding=(0, 10), padding_mode=‘reflect’, groups=10)<br>
output=m(input)</p>
</blockquote>
<p>However, the m.weight.shape gives me torch.Size([10, 1, 1, 21]), which means there are 10 different filters. How can I filter the 10 sensor data with the same filter?</p>
<p>Thanks in advance</p>
<p>Basically, I just want to implement a low-pass filter on time series data.</p>

ptrblck: <p>You could move the channel dimension to the batch dimension, apply the convolution, and reshape the output back. Alternatively, you could also set the weight of each filter to the same value, but you might need to redo this operation after updating the filter.</p>

LongWU: <p>Thanks for your reply. Did you mean something like below:</p>
<pre><code class="lang-auto">class aaa(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1=nn.Conv2d(10,10,(1,10),padding='same',padding_mode='reflect',bias=False,groups=10)
    def forward(self,x):
        for i in range(self.l1.weight.shape[0]):
            self.l1.weight.data[i,:,:,:]=self.l1.weight.data[0,:,:,:]
        output=self.l1(input)
        return output
</code></pre>
<p>I was expecting something more elegant. Like a switch, maybe a share_weight=True? Is there such thing?</p>

ptrblck: <p>Your approach would work, but you could also use a parametrization as seen here:</p>
<pre><code class="lang-python">conv = nn.Conv1d(3, 10, 10)
print(conv.weight)

class WeightReuse(nn.Module):
    def forward(self, w):
        out_channels = w.size(0)
        w = w[0, ...].expand(out_channels, -1, -1)
        return w

torch.nn.utils.parametrize.register_parametrization(conv, "weight", WeightReuse())
print(conv.weight)
print(conv.parametrizations.weight.original)
</code></pre>

LongWU: <p>Many thanks. I think I will just reset the weight. I am not familiar with the parameterizations…</p>
Monte carlo estimate of KL divergence

williamFalcon: <p>I recently discovered the distributions package in PT. I’d like to estimate the KL divergence which I think I can do by:</p>
<pre><code class="lang-auto">        # KL div
        q = torch.distributions.normal.Normal(z_mu, std)
        p = torch.distributions.normal.Normal(loc=torch.zeros_like(z_mu), scale=torch.ones_like(std))
        qz = q.log_prob(z)
        pz = p.log_prob(z)

        inside = (qz - pz)
        kl_loss = torch.mean(inside)
</code></pre>
<p>But, the results are really bad compared with doing it analytically.</p>
<pre><code class="lang-auto">        # kl_loss = 0.5 * torch.sum(-torch.log(z_var) + z_mu**2 - 1.0 - z_var)
        kl_loss = 0.5 * torch.sum(torch.exp(z_var) + z_mu**2 - 1.0 - z_var)
</code></pre>
<p>Can anyone spot what I’m doing wrong in the MC version?</p>

cdsjjav: <p>When calculating qz = q.log_prob(z) &amp; pz = p.log_prob(z), how do you sample z? z must be sampled from q</p>
Unexpected behavior from torchscript

pt2: <p>Hi,</p>
<p>I have encountered some unexpected behavior with mixing <code>torch.jit.script</code> and <code>torch.jit.trace</code>. Here’s a example to reproduce.</p>
<pre><code class="lang-python">import torch
import numpy as np

@torch.jit.script
def select_rows(
    nums: int,
    data: torch.Tensor,
    size: int
):
    valid_choice = torch.multinomial(torch.ones(nums).float(), size)
    return data[valid_choice]

def do_selection(x):
    return select_rows(x.shape[0], x, x.shape[0])

t_4 = torch.tensor(np.array([1, 2, 3, 4]))
t_7 = torch.tensor(np.array([1, 2, 3, 4, 5, 6, 7]))

traced_selection = torch.jit.trace(do_selection, t_4)

print(traced_selection(t_4))
&gt;&gt;&gt; tensor([3, 1, 2, 4])  # A random arrangement of the input data.

print(traced_selection(t_7))
&gt;&gt;&gt; tensor([1, 3, 2, 4])  # Another random arrangement, but of the TRACED EXAMPLE!
# Expected a random arrangement of the current input of size 7!
</code></pre>
<p>In my actual example, <code>def do_selection()</code> is extremely complicated, and cannot be scripted using <code>torch.jit.script</code> . What are my options here? Is this the expected behavior?</p>
<p>Thanks.</p>

ptrblck: <p>Yes, this is expected behavior and the second call will also warn you:</p>
<pre><code class="lang-python">TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Tensor-likes are not close!

Mismatched elements: 3 / 4 (75.0%)
Greatest absolute difference: 3.0 at index (3,) (up to 0.0 allowed)
Greatest relative difference: 2.0 at index (1,) (up to 1e-05 allowed)
  _check_trace(
</code></pre>
<p><code>torch.jit.trace</code> will not record any data-dependent control flow and will use the passed inputs to trace the model execution.<br>
You should use <code>torch.jit.script</code> instead and narrow down what exactly is failing in your code using it.</p>

pt2: <p>Thanks! I was looking for an easy way out but there wasn’t one. We were attempting to port the <a href="https://arxiv.org/pdf/2102.03334.pdf" rel="noopener nofollow ugc">ViLT, ICML 2021</a> model to a phone.</p>
<p>Took a while, but it’s working now.</p>
<p>For anyone that’s interested – <a href="https://github.com/priyamtejaswin/ViLT/blob/script/generate_assets.py" rel="noopener nofollow ugc">https://github.com/priyamtejaswin/ViLT/blob/script/generate_assets.py</a></p>
Rpc functions.async_execution rpc and futures

spyroot: <p>Hi Folks,</p>
<p>Can someone please help me understand why it is not working?  My understanding was that torch Future is asyncio.</p>
<pre><code class="lang-auto">remote worker
class foo
@rpc.functions.async_execution
def foo()
  x = []
  y = []
  // here x and y populated so x and y are Future
return x, y
</code></pre>
<pre><code class="lang-auto">class Agent()
def bar():
    ft = ob_rref.rpc_async().foo(self.agent_rref, n_steps, meta_tasks)
   self.trainer.step(*ft)

   where step expects asyncio future.
</code></pre>
<p>So ft should be a future, but I can’t wait, nor can I pass nor unpack.</p>
<pre><code class="lang-auto">class Trainer.
     def step(x,y):
        in my old code, x and y list of futures.
        so downstream code I have await x.  or gather *x
</code></pre>
<p>But it looks like torch Future not iterable</p>
<pre><code class="lang-auto">Basically, the code that I'm moving to is RPC, but before that, it was Python 3.10 asyncio 

        x  = [self._event_loop.create_future() for _ in X]
        y = [[self._event_loop.create_future() for _ in Y] for _ in range(steps)]

So it lists futures
</code></pre>
<p>So I guess how I can convert torch Future to asyncio Future?</p>

pritamdamania87: <p>torch.Future is not an asyncio future, you can refer to the docs here: <a href="https://pytorch.org/docs/stable/futures.html" class="inline-onebox" rel="noopener nofollow ugc">torch.futures — PyTorch 1.13 documentation</a>.</p>
<p>You can probably convert a torch.Future to asyncio Future as follows:</p>
<pre><code class="lang-auto">torch_fut = rpc.async(...)
asyncio_fut = asyncio.Future()

torch_fut.then(
  lambda torch.fut: asyncio_fut.set_result(torch_fut.result())
)
</code></pre>

spyroot: <aside class="quote no-group" data-username="pritamdamania87" data-post="2" data-topic="167571">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<pre><code class="lang-auto">torch_fut = rpc.async(...)
asyncio_fut = asyncio.Future()

torch_fut.then(
  lambda torch.fut: asyncio_fut.set_result(torch_fut.result())
)
</code></pre>
</blockquote>
</aside>
<p>Thank you very much,  Let me try ,   But if it wrapper around torch._C.Future what can you do from asyncio perspective.  If downstream code uses asyncio ?</p>
<p>Does torch._C.Future has a similar behavior thread future in Python ?</p>
Torch multiproccesing

spyroot: <p>Hi Folks,</p>
<p>I’m having a strange issue that I think I already spent two days :).  I guess it related to the example given.</p>
<pre><code class="lang-auto">x = queue.get()
x_clone = x.clone()
queue_2.put(x_clone)
</code></pre>
<p>So if tensor x on GPU, as far I understand, you need to clone it.</p>
<p>First question, if I have object A, which holds a reference for four tensors, what is the right way to clone an entire object with all tensors ? i.e do you need first move to CPU put the queue and then back to GPU ? Because if detach.clone while tensor on GPU the data always zeroes.</p>
<p>class A<br>
self._observations → tensor<br>
self._actions → tensor</p>
<pre><code class="lang-auto"> def clone(self, dst):
        dst._observations = self._observations.detach().clone().cpu()
        dst._actions = self._actions.detach().clone().cpu()
</code></pre>
<p>This one I use</p>
<pre><code class="lang-auto">lock network_lock:
  x = A()
  network.forward(x)
  new_x = A() 
  # a a new object.   x.clone() take new_x and detach() and move to cpu()
  # note that I've tried to detach() and clone() and different combination but can't get it work.
  # NOTE when device set to CPU everything is working
  queue.put(x.clone(new_x))
</code></pre>
<p>My second question is if a tensor on GPU what is right way to put object<br>
A that hold bunch of tensor on GPU ?</p>
<pre><code class="lang-auto"> def clone(self, dst):
        dst._observations = self._observations.detach().clone()
        dst._actions = self._actions.detach().clone()
</code></pre>
<p>In the second example, I notice as soon as the object goes to a queue.<br>
If a tensor on GPU data becomes zero, the value is zero when the consumer picks up from the queue.</p>

kwen2501: <p>Hi, this question seems to be more related to tensor copy methods. Perhaps it would get more attention if moved to the relevant sub forum? Thanks!</p>

spyroot: <p>It might, but I’m not 100 sure because I tried a different combination.  So I guess it is essential to understand how exactly Torch shares tensor between the various processes.  Specifically, mp. Multiprocessing that torch overwrites.  My guess it should somehow share pointer that own by CUDA and I’m not sure who provides synchronization access or pure delegate to python multiprocess.</p>
<p>[W CUDAGuardImpl.h:62] Warning: CUDA warning: invalid device ordinal (function uncheckedSetDevice)<br>
[W CUDAGuardImpl.h:46] Warning: CUDA warning: driver shutting down (function uncheckedGetDevice)</p>

pritamdamania87: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> <a class="mention" href="/u/simonw">@SimonW</a></p>

spyroot: <p>Folks,</p>
<p>After days of debugging, I think it is just a bug in ForkedPickle I am basically moving code to RPC.</p>
<p>I think somewhere done the line. There is a bug in PIPE. Essentially it contains a tensor that I never put in the first place. And tensors have grad to it.</p>
<p>Just FYI</p>
Pytorch model failing to load

magesh: <p>Hello All, I am trying to deploy my trained pytorch model and seeing an error trace as below</p>
<p>failed to load model ‘’: PytorchStreamReader failed locating file constants.pkl: file not found<br>
Exception raised from valid at /opt/pytorch/pytorch/caffe2/serialize/inline_container.cc:151</p>
<p>Can someone help with the above error whether something is wrong with my model training or something else ?</p>

ptrblck: <p>Your file might be corrupt and I guess <code>torch.load</code> fails to load all internal objects. Could you check if re-saving the model could help?</p>
Clip_grad_norm_ for each sample in DDP

quailwwk: <p>Hi, is there any way to perform clip_grad_norm_ for the gradient produced by each sample (rather than a batch) when using DDP?</p>

huahuanZ: <p>AFAIK, this is impossible since the gradients are reduced over the batch and the information of each sample is no longer resumable.</p>
<p>Could you tell in what situation you want to do that?</p>

pritamdamania87: <p>You can probably do this via the autograd hooks: <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.register_hook — PyTorch 1.13 documentation</a> or the DDP communication hooks: <a href="https://pytorch.org/docs/stable/ddp_comm_hooks.html" class="inline-onebox" rel="noopener nofollow ugc">DDP Communication Hooks — PyTorch 1.13 documentation</a></p>
BPTT with linear layers

clarky103: <p>I’m using a recurrent VAE to encode a path and then decode that path into a new similar path. The encoding portion consists of a GRU and 3 linear layers, and the decoding portion consists of a GRU and 2 linear layers.</p>
<p>I’m not sure how to train this network given that there are RNNs as well as linear layers. I imagine I’ll have to use BPTT because of the RNN, but I’m not sure if I can use BPTT on the entire network or if I have to single out the recurrent layers, do BPTT on them, then perform normal backprop on the other layers.</p>
How do I access the hidden states of pre-trained torch models?

desert_ranger: <p>I am trying to access the hidden states of a huggingface (PyTorch) model. By state, I mean the internal representation obtain while sending an input into a model. Could someone please help with this?<br>
I have seen some discussion on LSTM hidden states, but I don’t think that’s related to my problem.<br>
For further information, please see my SO question (although its not needed) - <a href="https://stackoverflow.com/questions/74690541/how-do-i-get-access-to-the-last-hidden-state-for-code-generation-models-in-hug" class="inline-onebox" rel="noopener nofollow ugc">pytorch - How do I get access to the "last_hidden_state" for code generation models in huggingface? - Stack Overflow</a></p>
Why give a model multiple ReLU attributes?

msailor: <p>When reading through people’s PyTorch code, I often notice people giving their models multiple ReLU attributes, e.g., like so:</p>
<pre><code class="lang-auto">class DummyModel(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.fc1 = nn.Linear(hidden_size, hidden_size)
        self.relu1 = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.relu2 = nn.ReLU()

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu1(x)
        x = self.fc2(x)
        x = self.relu2(x)
        return x
</code></pre>
<p>I’m just curious whether there is any advantage to doing this, as opposed to</p>
<pre><code class="lang-auto">class DummyModel2(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.fc1 = nn.Linear(hidden_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        x = self.relu(x)
        return x
</code></pre>
<p>Since ReLU has no parameters, it seems to me that these will have exactly the same result. So I’m curious if there is some reason to give a model more than one ReLU attribute? Is it just a way of mentally organizing the flow of their code?</p>

ptrblck: <p>I guess models developers use separate modules as it would allow users to replace each activation function separately and would thus allow to run more flexible experiments.</p>
TypeError: 'bool' object is not callable in Pytorch

l_c_le: <p>I have some Problems when i following a tutorial about autoencoder in youtube.<br>
Here is my code</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
import matplotlib.pyplot as plt

transform = transforms.ToTensor()

mnist_data = datasets.MNIST(root='./data',train=True, download=True, transform=True)

data_loader = torch.utils.data.DataLoader(dataset=mnist_data,batch_size=64,shuffle=True)

dataiter = iter(data_loader)
images, labels = next(dataiter)
print(torch.min(images), torch.max(images))
</code></pre>
<p>and i got this problems</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_9516\372505004.py in &lt;module&gt;
      1 dataiter = iter(data_loader)
----&gt; 2 images, labels = next(dataiter)
      3 print(torch.min(images), torch.max(images))

~\anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    626                 # TODO(https://github.com/pytorch/pytorch/issues/76750)
    627                 self._reset()  # type: ignore[call-arg]
--&gt; 628             data = self._next_data()
    629             self._num_yielded += 1
    630             if self._dataset_kind == _DatasetKind.Iterable and \

~\anaconda3\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
    669     def _next_data(self):
    670         index = self._next_index()  # may raise StopIteration
--&gt; 671         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    672         if self._pin_memory:
    673             data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)

~\anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py in fetch(self, possibly_batched_index)
     56                 data = self.dataset.__getitems__(possibly_batched_index)
     57             else:
---&gt; 58                 data = [self.dataset[idx] for idx in possibly_batched_index]
     59         else:
     60             data = self.dataset[possibly_batched_index]

~\anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py in &lt;listcomp&gt;(.0)
     56                 data = self.dataset.__getitems__(possibly_batched_index)
     57             else:
---&gt; 58                 data = [self.dataset[idx] for idx in possibly_batched_index]
     59         else:
     60             data = self.dataset[possibly_batched_index]

~\anaconda3\lib\site-packages\torchvision\datasets\mnist.py in __getitem__(self, index)
    143 
    144         if self.transform is not None:
--&gt; 145             img = self.transform(img)
    146 
    147         if self.target_transform is not None:

TypeError: 'bool' object is not callable
</code></pre>
<p>i wonder why this happened and how to due with this. Tks for help me!</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/dataloader-has-no-attribute-dataset/146623/12">here</a> with an answer.</p>
Masking queries in cross-attention?

jasperhyp: <p>Hi!  I am using <code>nn.MultiheadAttention</code> to do cross-attention and I need to mask both the context and the x (aka both queries and keys).  However, it looks like currently, the implementation only has <code>key_padding_mask</code>.  How can I do this for those queries (despite directly building the 2D <code>attn_mask</code>)?  Thanks!</p>