akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why do we omit the dataset size in the Exponential Moving Average adjustment term?

phamtrongthang123: <p>Hello everyone,</p>
<p>I am trying to understand the EMA feature in <a href="https://github.com/pytorch/vision/blob/main/references/classification/train.py" rel="noopener nofollow ugc">this training script</a> from pytorch/vision repository. Specifically, this part</p>
<pre><code class="lang-python">    if args.model_ema:
        # Decay adjustment that aims to keep the decay independent from other hyper-parameters originally proposed at:
        # https://github.com/facebookresearch/pycls/blob/f8cd9627/pycls/core/net.py#L123
        #
        # total_ema_updates = (Dataset_size / n_GPUs) * epochs / (batch_size_per_gpu * EMA_steps)
        # We consider constant = Dataset_size for a given dataset/setup and ommit it. Thus:
        # adjust = 1 / total_ema_updates ~= n_GPUs * batch_size_per_gpu * EMA_steps / epochs
        adjust = args.world_size * args.batch_size * args.model_ema_steps / args.epochs
        alpha = 1.0 - args.model_ema_decay
        alpha = min(1.0, alpha * adjust)
        model_ema = utils.ExponentialMovingAverage(model_without_ddp, device=device, decay=1.0 - alpha)
</code></pre>
<p>In this code, the dataset size is ommited because it is constant. However, I can’t find the reason how can we omit it even if it is constant. This adjustment may improve the result on imagenet because the dataset size is very big. But for small dataset, like 10 thousand images, and small epoch, like 100, I think it would be harmful because the ema will not average all the models.<br>
So I want to ask is there any other reason for why we omit the dataset size? Or is it just for convenient?<br>
Thank you.</p>
torch::prod(ArrayRef)

mmoelle1: <p>Hi,</p>
<p>not sure if this is the right place to ask for feature requests. What would be nice to have is a member function of the <code>ArrayRef</code> class that returns to total size</p>
<pre><code class="lang-auto">torch::prod(ArrayRef)
</code></pre>
<p>This does not work since <code>prod</code> is not implemented for objects of type <code>ArrayRef</code>.</p>
<p>Best regards,<br>
Matthias</p>

ptrblck: <p>Hi, could you create this feature request on GitHub, please?</p>

mmoelle1: <p>My apologize. After having created the feature request I was told that this functionality is already available</p>
<pre><code class="lang-cpp">torch::Tensor a(4,3,2);
a.numel();
</code></pre>
<p>I wanted to emulate that by</p>
<pre><code class="lang-cpp">torch::prod(a.sizes());
</code></pre>
<p>but that is apparently not needed anymore. Thank you!</p>
How to profile backward time of ReLU layer

cxv0519: <p>Literally, is there any way to profile backward time of relu layer automatically? (I mean by code itself, not by torch profiler)</p>
<p>I want to know elapsed time for backward computation at ReLU layer.<br>
I’m using ReLU with in-place false.</p>
<p>but with just code like below</p>
<pre><code class="lang-auto">            # forward
            start = time.time()
            output = layer(input)
            torch.cuda.synchronize()
            end = time.time()
            total = end-start
            
            # backward
            dummy_grad = torch.rand(output.size()).cuda()
            start = time.time()
            output.backward(dummy_grad)
            torch.cuda.synchronize()
            end = time.time()
            total = end-start
</code></pre>
<p>I got an error like below</p>
<pre><code class="lang-auto">Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Thank you Guys for answering this Topic.</p>

ptrblck: <p>To call <code>backward()</code> on an output tensor a computation graph with trainable tensors must be created. Set <code>.requires_grad_(True)</code> on the <code>input</code> and it should work. Also, synchronize the GPU before starting the timers, too.</p>

cxv0519: <p>Oh, Thank you for answering this Topic and nice point for synchronizing before backward.</p>
<p>And your solution worked!<br>
I made the first synthetic input with requires_grad=True option, then I got an expected result!.</p>
<p>Thank you again😊</p>
No Op registered for Plugin with domain_version of 9

MartinZhang: <pre><code class="lang-auto">onnx.onnx_cpp2py_export.checker.ValidationError:No Op registered for Plugin with domain_version of 9

==&gt; Context: Bad node spec for node. Name: OpType:Plugin
</code></pre>
<p>please help me, thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=10" title=":handshake:" class="emoji" alt=":handshake:"></p>

hitbuyi: <p>I has same problem, did you solved it?</p>
Something went wrong with cudnn？

jiexing_wen: <p>Downloaded from the official website, reinstalled cudnn, and tried using conda, but it didn’t work.</p>
<p>Traceback (most recent call last):<br>
File “D:\workspace\rl\hebbrl\hebbrl3cudacopy2.py”, line 101, in <br>
train(loader)<br>
File “D:\workspace\rl\hebbrl\AutoEncoder.py”, line 96, in train<br>
loss.backward()<br>
File “D:\Users\miaoy\anaconda3\lib\site-packages\torch_tensor.py”, line 396, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “D:\Users\miaoy\anaconda3\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 173, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution</p>

ptrblck: <p>Are you running out of memory? If not, could you post a minimal executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect-env</code>, please?</p>

jiexing_wen: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/232728b63d6fca27cb16e7398ecbda3919833a73.png" alt="image" data-base62-sha1="50YBM9LTxP0gulDOivRuJCQxoXh" width="569" height="139"><br>
I try to record the memory in gpu-z.</p>

jiexing_wen: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/232728b63d6fca27cb16e7398ecbda3919833a73.png" alt="image" data-base62-sha1="50YBM9LTxP0gulDOivRuJCQxoXh" width="569" height="139"><br>
my gpu is 2060 with 6g memory.</p>
Accessing operations in `forward()`

Samue1: <p>Let’s say I have the following model</p>
<pre><code class="lang-auto">class Model(nn.Module):

    def __init__(self):
        super().__init__()

        self.conv = torch.nn.Conv2d(in_channels=3, out_channels=4, kernel_size=(3, 3))

    def forward(self, x):
        x = self.conv(x)
        x = torch.flatten(x, start_dim=1)
        x = torch.sigmoid(x)
        return x
</code></pre>
<p>Now, if I use</p>
<pre><code class="lang-auto">for name, module in model.named_children():
    print(type(module))
</code></pre>
<p>only the convolutional operations shows up:</p>
<p><code>&lt;class 'torch.nn.modules.conv.Conv2d'&gt;</code></p>
<p>I would like to know, why the <code>flatten()</code> and <code>sigmoid()</code> do not show up?</p>
<p>If I, for example, want to replace in an existing model the <code>flatten()</code> or <code>sigmoid()</code> operation, how do I do that in this case? How can I modify the forward pass?</p>

ptrblck: <p>The functional operations used in the <code>forward</code> method are not registered as modules and thus won’t show up in <code>model.modules()</code> or <code>model.children()</code>.</p>
<aside class="quote no-group" data-username="Samue1" data-post="1" data-topic="127575">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f04885/48.png" class="avatar"> Samue1:</div>
<blockquote>
<p>If I, for example, want to replace in an existing model the <code>flatten()</code> or <code>sigmoid()</code> operation, how do I do that in this case? How can I modify the forward pass?</p>
</blockquote>
</aside>
<p>You can directly modify the <code>forward</code> method and replace these operations.<br>
In case you want to manipulate the <code>forward</code> of another pre-defined model, you could derive a custom model from it and implement your <code>forward</code> there.</p>

pzn666: <p>Hello, take Samue1’s example, if I want to get the output tensor of unregistered operation <code>torch.flatten</code> in <code>forward</code> what can I do?</p>

ptrblck: <p>You could <code>return</code> this tensor in the <code>forward</code> additionally to the last output.</p>

pzn666: <p>Thanks for your quick reply! Sorry, I didn’t make my condition clear, I cannot modify the <code>forward</code> function as I wish since my users don’t want me to. Is there any function or way to get output tensors of desired ops by name or something like in tensorflow1?</p>

ptrblck: <p>I don’t think this is easily doable. Forward hooks need <code>nn.Module</code>s, while it seems you are using the pure functional API in the <code>forward</code>. I don’t know if e.g. <a href="https://pytorch.org/docs/stable/fx.html"><code>torch.fx</code></a> would be able to manipulate these functional calls to modules, which you could then use in forward hooks, but it might be worth checking the docs.</p>

pzn666: <p>Will give it a try, thanks!</p>
Libtorch's forward function crash for cpu mode, c10 error

zhang_yunjian: <p>hey, professionals! please help me.<br>
i use libtorch to execute forward on win10, and get this error:<br>
c10::Error, the libtorch called abort directly inner.<br>
the screenshot is as following<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35a46ab372fc42ed38d8b4d612dd430bac634be6.png" data-download-href="https://discuss.pytorch.org/uploads/default/35a46ab372fc42ed38d8b4d612dd430bac634be6" title="1664509824629"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/35a46ab372fc42ed38d8b4d612dd430bac634be6_2_690x369.png" alt="1664509824629" data-base62-sha1="7ExzaIzWFZBSkp7IjaYJffZmMaW" width="690" height="369" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/35a46ab372fc42ed38d8b4d612dd430bac634be6_2_690x369.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/35a46ab372fc42ed38d8b4d612dd430bac634be6_2_1035x553.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/35a46ab372fc42ed38d8b4d612dd430bac634be6_2_1380x738.png 2x" data-dominant-color="EAEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1664509824629</span><span class="informations">1557×834 75.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
and i pressed retry, i found the result is returned, but i’m not sure if it is right.</p>
<p>please help</p>

zhang_yunjian: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5ec757a1807ceb563e375afba47f895b02d0625d.png" data-download-href="https://discuss.pytorch.org/uploads/default/5ec757a1807ceb563e375afba47f895b02d0625d" title="1664510115444"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5ec757a1807ceb563e375afba47f895b02d0625d_2_690x333.png" alt="1664510115444" data-base62-sha1="dwrYd5Er94E9Ae1qfN7HlRIZMM5" width="690" height="333" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5ec757a1807ceb563e375afba47f895b02d0625d_2_690x333.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5ec757a1807ceb563e375afba47f895b02d0625d_2_1035x499.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5ec757a1807ceb563e375afba47f895b02d0625d_2_1380x666.png 2x" data-dominant-color="E7E4E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1664510115444</span><span class="informations">1745×844 94.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to initialize label embedding with pretrained model

thomas_ng: <p>The problem is I want to initialize the label embedding with a pretrained embedding.<br>
My original network is like this</p>
<pre><code class="lang-auto">def Network(RobertaPreTrainedModel):
     self.roberta = RobertaModel(config, add_pooling_layer=False)
     self.label_emb = nn.Embedding(config.num_labels, config.hidden_size)
</code></pre>
<p>Now I want to have label embedding comes from label description like</p>
<pre><code class="lang-auto">self.label_ids = #Some ID Tensor
self.label_attention_mask = #Some MaskTensor
self.label_emb=self.roberta(self.label_ids,attention_mask=self.label_attention_mask)
</code></pre>
<p>I try like above but when casting the model to GPU by</p>
<pre><code class="lang-auto">model, optimizer, eval_dataloader = accelerator.prepare(model, optimizer, eval_dataloader)
</code></pre>
<p>, it has error like</p>
<p><code>Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! error</code><br>
Then I find out self.label_emb is still in CPU device.</p>
<p>After adding code to convert it to GPU device it can perform the forward part ( loss= model.forward()) but cannot do gradient update (optimizer.step())<br>
The error is like<br>
<code> One of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [768] is at version 2; expected version 1 instead</code></p>
<p>Then does anyone help me to solve this problem, or guide me to initialize label embedding with a pretrained model?</p>
Location of GRU implementation

John_Schwarcz: <p>Hello,<br>
I’m looking for the location of the GRU implementation so that I can make a modification to the gates.<br>
I’m found some GRU implementation but it appears to be coded in C++ which I can’t use.</p>
<p>I’m wondering if there is a Python implementation for the GRU (like exists for the LSTM at <em>/python3.10/site-packages/torch/nn/quantizable/rnn.py</em> )</p>
<p>Thanks for the help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

tom: <p>No, the easiest might be to take the LSTM and adapt it into a GRU.</p>

John_Schwarcz: <p>Thanks for your reply, this is very do-able.<br>
Do you happen to know the reason for the lack of a Python-implemented GRU?<br>
It’s confusing given evidence of a GRU implementation is lying around in places.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f869102cf074b308c53718fe5b4786f051387702.png" alt="image" data-base62-sha1="zrxvAyaKw0FDYkvJPSHlgtCzEtQ" width="673" height="26"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6abc2d1e843a634efedff29d4e998b0b0a795416.png" alt="image" data-base62-sha1="fedMd9WUa0XZwd7VgV36SPEUv30" width="645" height="83"></p>

tom: <p>My speculation would be that more or less everyone is using transformers now. <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=12" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:" loading="lazy" width="20" height="20"></p>
<p>If you base your work on the master branch and imitate the existing code well, you could see if there are any takers to add it.</p>

jerryzh168: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="162367" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>No, the easiest might be to take the LSTM and adapt it into a GRU.</p>
</blockquote>
</aside>
<p>I think you are linking the quantizable rnn file, is that what you are looking for? if not, the floating point GRU implementation is here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L792" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L792</a></p>
Modifying DDP model after calling DistributedDataParallel()

jolygod: <p>Assume that a pytorch module has to be modified after some DDP training on a single machine with multiple GPUs, say, a linear classification head has to be replaced/re-initialized (if some conditions hold) and then trained some more.</p>
<p>What would be the recommended way to do this in PyTorch?<br>
When the new head in the example is replaced, it is done on all of the processes separately implying a totally differently initialized head in each process.<br>
I currently call DistributedDataParallel() on the entire model again after the modification and continue training. I’m not sure if this is the correct approach or even sure what happens in my scenario.<br>
I think that upon the second call to DistributedDataParallel() the model on rank 0 is duplicated to all of the other processes resulting with proper training.</p>
<p>Any advice on the correct approach?</p>

kwen2501: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> any suggestion? Thanks!</p>

aazzolini: <p>The safest way of doing that would be to clone/re-create the original model, copy all the parameter from the DDP model into the clone, and then wrap the clone with DDP. (It may work if you skip some of these steps but unless perf / memory is an issue, I’d just be on the safe side. E.g.):</p>
<p>original_model = MyModel()<br>
initial_ddp_model = DistributedDataParallel(original_model)<br>
train(initial_ddp_model)</p>
<p>new_model = MyModel()<br>
new_model.load_state_dict(initial_ddp_model.module.state_dict())<br>
new_ddp_model = DistributedDataParallel(new_model)<br>
new_optimizer =…</p>
Conditional transforms for image resize

ajwitty: <p>I only want to resize images that are smaller than my desired input size. Is there way to reshape images that are smaller than a certain size and ignore all others?</p>

ptrblck: <p>Using the <a href="http://pytorch.org/docs/master/data.html#torch.utils.data.Dataset" rel="nofollow noopener">Dataset</a> class, you can lazily load your images in the <code>__getitem__</code> method.<br>
Here is a small example:</p>
<pre><code>class TrainDataset(Dataset):
    def __init__(self, image_paths, targets):
        self.image_paths = image_paths
        self.targets = targets

    def __getitem__(self, index):
        image = Image.open(self.image_paths[index])
        y = self.target[index]

        # Resize your image here
        if image.size...:
            ....
        .... # Do other stuff here

        x = torch.from_numpy(image)
    
        return x, y

    def __len__(self):
        return len(self.image_paths)
</code></pre>
<p>You can wrap your <code>Dataset</code> to a <a href="http://pytorch.org/docs/master/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">DataLoader</a> for efficient loading, preprocessing, shuffling etc.</p>
<p>Hope this helps!</p>

craymichael: <p>For what it’s worth, the alternative to this is to subclass <code>Resize</code>. The advantage of this approach is that existing torchvision <code>Dataset</code> objects (like for CIFAR-10 or ImageNet) need not be modified - rather, a new transform can simply be used.</p>
<p>My implementation:</p>
<pre><code class="lang-python">class ConditionalResize(transforms.Resize):
    """
    Resize transform but only if the input is smaller than the resize dims
    """

    @property
    def resize_height(self):
        return self.size if isinstance(self.size, int) else self.size[0]

    @property
    def resize_width(self):
        return self.size if isinstance(self.size, int) else self.size[1]

    def forward(self, img):
        """
        Args:
            img (PIL Image or Tensor): Image to be scaled.

        Returns:
            PIL Image or Tensor: Rescaled image.
        """
        r_h, r_w = self.resize_height, self.resize_width
        if isinstance(img, torch.Tensor):
            h, w = img.shape[1:]
        else:  # PIL Image
            w, h = img.size
        if w &lt; r_w or h &lt; r_h:
            return super().forward(img)
        else:
            return img
</code></pre>
Pytorch Geometric - Can I save a data object to a file?

Bixqu: <p>Well, that’s pretty much the question. I have generated a data object, and the functions that created it take about 1h to run. I’d like to save it to a file, but I can’t find anything in the docs about that.</p>
<p>If it’s possible, can someone post a snippet about how you can save and load a pytorch geometric data object from a file?</p>

ptrblck: <p>Could you link a reference to a “Geometric data object”?<br>
If it’s just a <code>tensor</code>, you can use <code>torch.save</code>. Otherwise we would need to see the implementation of this object.</p>

Bixqu: <p>Hi, <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html" rel="nofollow noopener">here it is.</a></p>

ptrblck: <p>Thanks. I think <code>torch.save</code> should work.<br>
Could you try it and report, if you are seeing any issues?</p>

Bixqu: <p>I will try, thanks. Right now I’m having some other issues unfortunately. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

DG_22: <p>I have tried this and can validate that it works. Thank you</p>
Reducing communication cost of synchronization step

alena_kopanicakova: <p>Hi all,</p>
<p>I am implementing a new distributed optimizer using PyTorch.<br>
Optimizer is converging well, but it requires synchronization of parameters every now and then.<br>
Currently, the synchronization step takes a huge amount of time, rendering the whole parallelization approach useless.<br>
At the moment, I am using a very straightforward approach to perform the synchronization step.<br>
In particular, for corr (list of all model trainable parameters), I do:</p>
<pre><code>          for c in corr: 
            dist.all_reduce(c, op=dist.ReduceOp.SUM)
</code></pre>
<p>I wonder, is it possible to reduce communication costs somehow?<br>
I have seen that there are communication hooks, but they seem to operate only on the gradientBucket vector. Is it possible to also use them in other scenarios?</p>
<p>Many thanks,<br>
Best regards,<br>
Alena</p>
De-serialize pickled files

perf: <p>Hi,</p>
<p>I am trying to find a way to open my .pkl files in C++. One way to do this is to use the PicklingTools, however, there is a conflict when building with LibTorch. I see the following error:</p>
<pre><code class="lang-auto"> undefined reference to `OC::MakeVector(std::string const&amp;, unsigned long, char const*, unsigned long)'
</code></pre>
<p>I believe PicklingTools uses OpenContainers, is there a known fix for this? Or some other recommended option to deserialize pickled files?</p>
<p>Thanks!</p>
RuntimeError:Cuda error: initialization on dataloader

yoelt11: <p>For those having the following error:</p>
<blockquote>
<p>RuntimeError: CUDA error: initialization error<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
</blockquote>
<p>when looping through the dataset, (or calling 'data = self._next_data()) e.g:</p>
<pre><code class="lang-auto">for i, (x, y) in enumerate(data_loader):
     ...
</code></pre>
<p>it might be the case that some of the tensors inside of the dataset are already in the gpu, which causes this error.</p>
<p>As a way to debug this, you can print in your dataset class:</p>
<pre><code class="lang-auto">print(x.get_device())
print(y.get_device())
</code></pre>
<p>to see which device your tensors are.</p>
<p>One thing to keep in mind is that when you save a tensor (torch.save()), also the device is saved, such when it is loaded again, it is loaded directly to the device it was saved on.</p>
<p>so in the end you should have something like this:</p>
<pre><code class="lang-auto"># in data loader tensors should be in cpu
for i, (x, y) in enumerate(data_loader):
   x = x.to(device) # where device is cuda
   y = y.to(device) # where device is cuda
</code></pre>
Error: "One of the differentiated Tensors appears to not have been used in the graph"

ximing: <p>I am trying to compute a gradient of y_hat to x (y_hat is the sum of gradients of model output to x) but it gives me the error: One of the differentiated Tensors appears to not have been used in the graph. This is the code:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self,):
        super(Model, self).__init__()
        self.weight1 = torch.nn.Parameter(torch.tensor([[.2,.5,.9],[1.0,.3,.5],[.3,.2,.7]]))
        self.weight2 = torch.nn.Parameter(torch.tensor([2.0,1.0,.4]))
   

    def forward(self, x):
        out =F.linear(x, self.weight1.T)
        out =F.linear(out, self.weight2.T)
        return out

model = Model() 
x = torch.tensor([[0.1,0.7,0.2]])
x = x.requires_grad_()
output = model(x)
y_hat = torch.sum(torch.autograd.grad(output, x, create_graph = True)[0])
torch.autograd.grad(y_hat, x)
</code></pre>
<p>I think x should be in the computational graph, so I don’t know why it gives me this error? Any thoughts would be appreciated!</p>

srishti-git1110: <p>Hi, please run the below command to see x is no more in the graph of <code>torch.autograd.grad(output, x, create_graph = True)</code>:</p>
<pre><code class="lang-auto">from torchviz import make_dot
make_dot(torch.autograd.grad(output, x, create_graph = True))
</code></pre>

ximing: <p>Hi, I did this and it gives me this graph:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97281996fb8e6bdf168c870f048ab0e755894947.png" data-download-href="https://discuss.pytorch.org/uploads/default/97281996fb8e6bdf168c870f048ab0e755894947" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97281996fb8e6bdf168c870f048ab0e755894947.png" alt="image" data-base62-sha1="lzc0JnHzIY40w6S6erzN715jEV1" width="350" height="500" data-dominant-color="EAEBE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">406×579 12.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am not sure how I can see which is x?</p>

srishti-git1110: <p>x isn’t there which is why you got the error.</p>
<p>The two blue leaf nodes are the parameters <code>weight1</code> and <code>weight2</code>.</p>
Unable to print pytorch version

Anoushka_Thakur: <p>Hi, I am fairly new to pytorch I was able to download pytorch via pip install method (without cuda for windows)… After installation I am able to import torch although i am unable to print it’s version. When do I receive the error " module ‘torch’ has no attribute '<strong>version</strong>"<br>
Please le me know</p>

ptrblck: <p>I’m unsure if it’s a formatting issue, but I assume you were using <code>print(torch.__version__)</code>?<br>
If so, is your current working directory set to any folder containing <code>torch.py</code> or any other file which might cause conflicts?</p>

Anoushka_Thakur: <p>Yes I was using print(torch.<strong>version</strong>)<br>
I was able to resolve the issue.</p>

ptrblck: <p>What was causing the issue?</p>

Anoushka_Thakur: <p>It showed that some file was not in the path and hence maybe it wasn’t able to use that print function.</p>
How to calculate armijo step length with torch.nn.BCELoss()?

Jeet: <p>I am trying to calculate the armijo step length for the steepest descent direction of 2 loss functions of the generator and the discriminator.</p>
<p>I am unable to calculate the armijo step length for the same. How to calculate the f(x+tv) where x is the current iteration and t is the learning rate and v is the direction of steepest descent respectively. I have the values of x and v but how to call the loss function? Considering the loss function I am using as nn.BCELoss(), how can I calculate the value of the function at some point in pytorch?</p>
LSTM: Padding within sequence

sesale: <p>My sequences can contain padding anywhere within the sequence as well as at the start and at the end.<br>
I am aware how to use <code>pad_sequence</code>and <code>pack_padded_sequence</code> to add padding to the sequences at the end, so that all sequences have the same length.<br>
Does anyone know whether there exists something similar that I could use in my case?</p>

srishti-git1110: <p>Do you intend to intend to use a way to pad sequences by inserting 0s anywhere in the sequence randomly, and not just at the end?</p>

sesale: <p>Yes exactly. So I have multiple time series which are missing some time steps at different points. I padded all sequences to the same length by inserting zeroes where data is missing and I now want to “tell” the model to not pay attention to these time steps.</p>

srishti-git1110: <p>So, essentially it boils down to modifying the attention mask so that it has a 0 for all values that are 0 in the sequence, and 1 otherwise. Is that it? What model are you using?</p>
<p>If yes, can we then construct the attention mask like:</p>
<pre><code class="lang-auto">seq = torch.tensor([1.0, 2.0, 0, 3, 0, 8, 9, 0])
attn = torch.where(seq!=0, 1, 0)
print(attn)
</code></pre>
<p>gives:</p>
<pre><code class="lang-auto">tensor([1, 1, 0, 1, 0, 1, 1, 0])
</code></pre>

sesale: <p>Yes, exactly. Thank you!<br>
I am using an LSTM. How can I now add the attention mask to the model?<br>
Another thing is, I want to label the sequences (many-to-one), so I just need the last timestep’s hidden state. Once I know how to tell the model which entries to pay attention to, how can I make sure that it returns the hidden state of the last timestep that is not padded?</p>
Torch argmax not finding max value in array

ori: <p>I am trying to find argmax of an array, but it returns empty value. Any idea whats wrong.<br>
<code>torch.argmax(torch.tensor([[ 1.92, -1.27, -0.86]], dtype=torch.float32))</code> retunrs empty array</p>

srishti-git1110: <p>Your code runs fine for me.<br>
Could you please re-run it?</p>
Vision evaluation metrics

Rafa: <p>Hello everyone,</p>
<p>I’m currently working on a Mask-RCNN project. I’ve been following the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial</a> and it helped me a lot.</p>
<p>But I’d like to be able to get some evaluation metrics in order to compare various models. Things like training and validation losses, Precision, Recall, F1 curve, mAP, losses, and confusion matrix. I haven’t been able to find a neat implementation. I’d like to be able to store information somewhere (like a csv file) and do some plotting.</p>
<p>So far I haven’t been able to find many tutorials explaining this part. Could anyone point me to a project or an example that does this?</p>
Question about src_mask in TransformerEncoder

jrmiranda: <p>I’m trying to train a Transformer model similar of how BERT was trained, where elements of the input sequence are masked randomly. By reading the docs, I found that the expected shape of <code>src_mask</code> is (N*num_heads, S, S). I suppose that the batch size is multiplied by the number of heads because the model will use a different mask for each head (is that correct?), so I’m just repeating the mask for each head. I’m using the following code to generate the masks:</p>
<pre><code class="lang-auto">num_heads = 8
batch_size = 32
seq_len = 50
prob = 0.3

src_mask = torch.rand(batch_size, seq_len, 1) &lt; prob  # generate random mask for each sequence (N, S, 1)
src_mask = src_mask.repeat(1, 1, seq_len)  # make a square matrix for each mini-batch (N, S, S)
src_mask = src_mask.repeat_interleave(num_heads, dim=0)  # repeat for each head (N*num_heads, S, S)
</code></pre>
<p>Is this approach correct? Thanks.</p>
Concatenate feature maps with different dimension

Anita: <p>Hi. I was trying to merge feature maps that I got from two different encoders which have different dimensions.</p>
<p>simclr_features: torch.Size([543, 512]) imagenet_features: torch.Size([543, 1024])</p>
<p>so I wanted: torch.Size([543, 1536]). what is a possible solution to do that?</p>

srishti-git1110: <p>Hi,<br>
Use torch.cat:</p>
<pre><code class="lang-auto">x = torch.randn(3, 5)
y = torch.randn(3, 6)
z = torch.cat((x, y), dim=1)
print(z.shape)
</code></pre>
<p>out -</p>
<pre><code class="lang-auto">torch.Size([3, 11])
</code></pre>
'ImageClassifierOutput' object has no attribute 'requires_grad' to conversion of model from pytorch to ONNX

dgrnd4: <p>Hi there, I have the above error while running the following piece of code, from the <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html" rel="noopener nofollow ugc">tutorial</a> :</p>
<pre><code class="lang-auto">
import onnxruntime

ort_session = onnxruntime.InferenceSession(model_onnx)

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

# compute ONNX Runtime output prediction
ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(x)}
ort_outs = ort_session.run(None, ort_inputs)

# compare ONNX Runtime and PyTorch results
np.testing.assert_allclose(to_numpy(torch_out), ort_outs[0], rtol=1e-03, atol=1e-05)

print("Exported model has been tested with ONNXRuntime, and the result looks good!")
</code></pre>
<p>Can you please help me with that? Many THANKS!!!</p>
<p>The whole error that I get is this one:</p>
<pre><code class="lang-auto">
---------------------------------------------------------------------------

AttributeError Traceback (most recent call last)

[&lt;ipython-input-29-129a29912fd2&gt;](https://9b63epfevop-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220926-060050-RC01_476951119#) in &lt;module&gt; 11 12 # compare ONNX Runtime and PyTorch results ---&gt; 13 np.testing.assert_allclose(to_numpy(torch_out), ort_outs[0], rtol=1e-03, atol=1e-05) 14 15 print("Exported model has been tested with ONNXRuntime, and the result looks good!")

[&lt;ipython-input-29-129a29912fd2&gt;](https://9b63epfevop-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220926-060050-RC01_476951119#) in to_numpy(tensor) 4 5 def to_numpy(tensor): ----&gt; 6 return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy() 7 8 # compute ONNX Runtime output prediction

AttributeError: 'ImageClassifierOutput' object has no attribute 'requires_grad'
</code></pre>
With torch.cuda.graph(g): capture fails on multiple gpu

taoye9: <p>hi, I encounter a silent failure when I try to capture  cuda graphs on mutiple GPUs using <code>with torch.cuda.graph(g):</code> context.  It turns that <code>with torch.cuda.graph(g0, stream=s):</code> works</p>
<p>The problem is that only one of the two cuda graphs seems successuly catpures the warpped functions. The other <code>graph.replay</code> seems do nothing.  By chance, I found that this problem can be fixed by explicatly create a new stream and using <code>with torch.cuda.graph(g0, stream=s):</code>,  would someone from the community explains why and what is the expected way to use cuda graph on mutiple gpus?</p>
<h1>
<a name="more-details-1" class="anchor" href="#more-details-1"></a>more details</h1>
<p>What I intented to do is basically using cuda graph to accerlate inplace add of two tensor list on two different GPU serparately.  The following code (mostly adpoted from <code>torch.cuda.make_graphed_callables</code>)  fails as  when call <code>g1.replay()</code> nothing happens. the output place_holder tensor remains unchanged.</p>
<pre><code class="lang-auto">data_cuda0 = [torch.ones((1), dtype=torch.float, device="cuda:0") for _ in range(3)]
output0 = torch.zeros((1), dtype=torch.float, device="cuda:0")

data_cuda1 = [torch.ones((1), dtype=torch.float, device="cuda:1") for _ in range(3)]
output1 = torch.zeros((1), dtype=torch.float, device="cuda:1")

torch.cuda.set_device("cuda:0")
g0 = torch.cuda.CUDAGraph()

# warmup
s = torch.cuda.Stream()
s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):
    for i in range(3):
        output0.zero_()
        for t in data_cuda0:
            output0.add_(t)
torch.cuda.current_stream().wait_stream(s)

with torch.cuda.graph(g0):
    for t in data_cuda0:
        output0.add_(t)
output0.zero_()
g0.replay()
print(output0) # tensor([3.], device='cuda:0')

torch.cuda.set_device("cuda:1")
g1 = torch.cuda.CUDAGraph()

# warmup
s = torch.cuda.Stream()
s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):
    for i in range(3):
        output1.zero_()
        for t in data_cuda1:
            output1.add_(t)
torch.cuda.current_stream().wait_stream(s)

with torch.cuda.graph(g1):
    for t in data_cuda1:
        output1.add_(t)
output1.zero_()
g1.replay()
print(output1) #output tensor([0.], device='cuda:1')
</code></pre>
<pre><code class="lang-auto"></code></pre>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce this error, please?</p>

taoye9: <p>thanks for reply. i’ve uploaded a executable demo code</p>
[Maxpooling] what should the indices be when there're multiple max values?

ryanbty: <p>What to do when there are multiple values in the kernel that equals to the max? For example, for these values when we use torch.argmax :</p>
<pre><code class="lang-auto">torch.tensor([[1., 1.],
       [1., 1.]])
</code></pre>
<p>The max is simply 1. What should max indices look like? should it be True’s for <strong>all</strong> occurrence of max:</p>
<pre><code class="lang-auto">torch.tensor([[ True,  True],
       [ True,  True]])
</code></pre>
<p>Or the <strong>first</strong> occurrence of the max:</p>
<pre><code class="lang-auto">torch.tenso([[ True, False],
       [False, False]])
</code></pre>
<p>It is possible to simply retrieves all the maximum indices in the equality maximum value case using Pytorch ?</p>

ptrblck: <aside class="quote no-group" data-username="ryanbty" data-post="1" data-topic="162390">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/5f8ce5/48.png" class="avatar"> ryanbty:</div>
<blockquote>
<p>For example, for these values when we use torch.argmax :</p>
</blockquote>
</aside>
<p><code>torch.argmax</code> will return the index of the first max. value when duplicates are seen as described in the <a href="https://pytorch.org/docs/stable/generated/torch.argmax.html?highlight=argmax#torch.argmax">docs</a>.</p>
<p><code>nn.MaxPool2d</code> will return the max. value, so the index doesn’t matter unless you use <code>return_indices</code> in which case also the first index should be returned:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.ones(1, 1, 2, 2)
pool = nn.MaxPool2d(2, return_indices=True)

out = pool(x)
print(out)
# (tensor([[[[1.]]]]), tensor([[[[0]]]]))
</code></pre>

ryanbty: <aside class="quote no-group quote-modified" data-username="ryanbty" data-post="1" data-topic="162390">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/5f8ce5/48.png" class="avatar"> ryanbty:</div>
<blockquote>
<p><code> </code></p>
</blockquote>
</aside>
<p>Thanks for your answer. But it is possible to simply retrieves all the maximum indices ? Not only the first occurrence ?</p>

ptrblck: <p>Yes, you should be able to get the max. value and get the indices of all duplicates afterwards:</p>
<pre><code class="lang-python">x = torch.ones(1, 1, 2, 2)
x[0, 0, 0, 1] = 0.

max_val = x.max()
print((x == max_val))
# tensor([[[[ True, False],
#           [ True,  True]]]])

print((x == max_val).nonzero())
# tensor([[0, 0, 0, 0],
#         [0, 0, 1, 0],
#         [0, 0, 1, 1]])
</code></pre>
Weight sharing perceptrons

mfatih7: <p>Hello<br>
I want to implement a network having weight-sharing perceptrons.<br>
If I use structures like below</p>
<pre><code class="lang-auto">class ABC(nn.Module):
    def __init__(self, n, ch_in, d):
        super(ABC, self).__init__()
           
        self.n = n
        self.ch_in = ch_in
        self.d = d
        
        self.linear = nn.Linear(self.ch_in, self.d)
        
    def forward(self, x):        
        
        outt = []        
        for i in range(self.n):
            outt.append(self.linear( x[:, self.ch_in*i:self.ch_in*(i+1)]))
        out = torch.cat(outt, dim=1)
        
        return out
</code></pre>
<p>How will be the performance in GPU?<br>
Should I avoid using lists?</p>

mfatih7: <p>When loops and lists are replaced by conv layers the processing speed becomes 3 times faster in my trials.</p>
CPU Memory Allocation/Virtual Memory Allocation per GPU

Richard_Pruss: <p>Is there some way to reduce the CPU memory allocation on init of torch?</p>
<p>When we run torch.is_available() it allocates 11GB for one GPU and 44.2GB when we use six GPUs.</p>
<p>Then when we start the workers in the training loop that CPU allocation is copied to each worker, so we see this massive memory use.</p>
<p>The 11GB is much bigger than the model + n x workers * batch size we roughly expect things to be.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d556107e8eb505d58c9604339f58ede9d4eb13c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7d556107e8eb505d58c9604339f58ede9d4eb13c" title="Screen_Shot_2022-09-14_at_10.41.23_am"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d556107e8eb505d58c9604339f58ede9d4eb13c_2_690x314.jpeg" alt="Screen_Shot_2022-09-14_at_10.41.23_am" data-base62-sha1="hSKBFkRNrqVgnzCRHIuuE9HNXu4" width="690" height="314" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d556107e8eb505d58c9604339f58ede9d4eb13c_2_690x314.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d556107e8eb505d58c9604339f58ede9d4eb13c_2_1035x471.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d556107e8eb505d58c9604339f58ede9d4eb13c_2_1380x628.jpeg 2x" data-dominant-color="2B3D2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen_Shot_2022-09-14_at_10.41.23_am</span><span class="informations">1917×873 518 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Sorry if this is dead obvious but we found 3 open issues on this and no obvious smoking gun of what we might be doing wrong or is 11GB of CPU memory the expected overhead of using a GPU?</p>
<p>Thanks,<br>
Ric</p>
<p>P.S. Some small technical details.<br>
Using the current Nvidia Docker image for Pytorch<br>
(FROM <a href="http://nvcr.io/nvidia/pytorch:22.08-py3" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:22.08-py3</a>)<br>
Model and Training loop is Detectron2 Masked RCNN</p>

Richard_Pruss: <p>Is this the correct area to post this?  Should I move the question somewhere else?</p>

ptrblck: <p>No, that’s the right place. Sorry for not responding before as I’ve seen this post but it dropped from my list. Will get back to you after checking with our CUDA team.</p>
Inserting custom layer in a pretrained model

Vibhu: <p>Hi,<br>
I am looking for a way to slightly modify Hugging Face GPT-2’s architecture by inserting a custom feedforward layer inside a GPT-2 decoder block, right after the masked self-attention sublayer. I want to then initialize all original parameters with pre-trained GPT-2 weights and the newly added ones randomly. Is there a way to achieve this by inheriting Hugging Face’s GPT-2 model, instead of copying Hugging Face’s modeling_gpt2 file and then making changes to it?<br>
I’d be really grateful if someone could guide me or point me in the right direction.</p>
<p>Thanks</p>

srishti-git1110: <aside class="quote quote-modified" data-post="2" data-topic="162140">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krypticmouse/48/52544_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/new-layer-in-bert/162140/2">New layer in BERT</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    You can simply replace the 6th layer with another layer. This layer will have 6th layer encoder as first layer and your custom layer as the second layer. For example, let’s try adding another encoder in the 5th layer:- 
from torch import nn
from transformers import BertForSequenceClassification, BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

class CustomBertLayer(nn.Module):
    def __init_…
  </blockquote>
</aside>

<p>Does this help?<br>
It’s not about GPT-2, but maybe you could replicate this for your use case.</p>
How can I get intermediate output from a torch._C.Graph

nicheng0019: <p>In one .py file, I save a  torch.jit model:</p>
<blockquote>
<p>torch.jit.save(torch.jit.trace(model_quantized, example_inputs=input), “model.pt”)</p>
</blockquote>
<p>in another .py file, I load and run the model:</p>
<blockquote>
<p>model_quantized = torch.jit.load(“model.pt”)<br>
model_quantized(input)<br>
graph = model_quantized.graph</p>
</blockquote>
<p>how can I modify the graph to get intermediate output?</p>

krypticmouse: <p>Is <a href="https://discuss.pytorch.org/t/get-intermediate-layer-outputs-from-traced-graph/37976/5">this</a> what you are looking for?</p>
How to use libtorch on ios11

whying: <p>I found that the libtorch support ios 12  at least from this link<a href="https://github.com/pytorch/pytorch/blob/master/ios/LibTorch.podspec" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/ios/LibTorch.podspec</a>. How can I use libtorch on iso11? Whether the way of building from the source is feasible?</p>

cccclai: <p>Build from source for iOS is possible, check <a href="https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source" class="inline-onebox" rel="noopener nofollow ugc">iOS | PyTorch</a></p>
<p>Regarding iOS 11, maybe worth a shot, I’m not sure</p>
Batched checking if 2D image is connected

Ryan_Keathley1: <p>I am trying to create a vectorized 2D game with an initial phase where the player places 1’s into a 2D zero tensor (with a batch dimension). The ones indicate the presence of a wall at that square.<br>
We start with an empty zero tensor, which is obviously one path-connected-component, if we take movement to be one step in the x or y direction.<br>
With each placement of a wall we have to verify that we did not disconnect the space defined by the zeros in the image.<br>
Does anyone have any idea how this might be achieved in an <em>efficient</em> (batched) way?</p>

Ryan_Keathley1: <p>I just realized that I can use repeated convolutions with a filter that looks like:<br>
0 1 0<br>
1 0 1<br>
0 1 0<br>
and zeroing out any activations on a wall square<br>
after placing a single 1. on a walkable square</p>