akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
IterDataPipe and DDP

Miquel_Espinosa: <p>Is there any way of combining an <code>IterDataPipe</code> dataset with DDP?<br>
I have tried to convert my IterDataPipe to Map-style dataset with <code>to_map_datapipe()</code> but without success.<br>
Also, I am not sure if there are better ways to do this.</p>
<p>Thanks</p>

ejguan: <aside class="quote no-group" data-username="Miquel_Espinosa" data-post="1" data-topic="168629">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miquel_espinosa/48/47180_2.png" class="avatar"> Miquel_Espinosa:</div>
<blockquote>
<p>Is there any way of combining an <code>IterDataPipe</code> dataset with DDP?</p>
</blockquote>
</aside>
<p>Could you please elaborate on it?</p>

Miquel_Espinosa: <p>In the documentation it seems that using <code>torch.utils.data.distributed.DistributedSampler</code> is not compatible with an Iter-style dataset. I was just wondering if there is any way of making it compatible.<br>
Some people have suggested some <a href="https://discuss.pytorch.org/t/iterable-pytorch-dataset-with-multiple-workers/135475/3">solutions</a> when using multiple workers in <code>IterDataPipes</code>.</p>
How to set up annotation cvs file when each label image contains multiple categories?

Fredrik_OLSSON: <p>Hi,<br>
I am very new to Pytorch. I am trying to set up a custom dataset where I want to use Unet and train it for semantic segmentation. I have 41 images and 41 label images where each label image may contain 5 different categories. The label images are png images with a specific grayscale for background, category 1, category 2 etc. How should I set up my annotation csv file?</p>
<p>Thanks for any help.</p>
Validation loss and accuracy - Cora GNN

Akash_142: <p>Hi,</p>
<p>I am learning about GCN (Graph convolution network) and I found some standard code online for training GCN using the Cora dataset.</p>
<p>I got the following results for two GCN models after the val_loss converges:</p>
<p>6 GCN hidden layers:<br>
Train loss: 0.0066 | Train acc: 1.0000<br>
Val loss: 13.1973 | Val acc: 0.5040<br>
Test loss: 10.6857 | Test acc: 0.5430</p>
<p>16 GCN hidden layers:<br>
Train loss: 0.6381 | Train acc: 0.7643<br>
Val loss: 4.1288 | Val acc: 0.2800<br>
Test loss: 3.8559 | Test acc: 0.2910</p>
<p>What I don’t understand is how is the Validation accuracy of 16 layers lower than 6 layers even though the Validation loss of 16 layers is less than that of 6 layers model.</p>
<p>Here is my function for calc Validation:</p>
<pre><code class="lang-auto">def eval_step(model: torch.nn.Module, data: Data, loss_fn: LossFn, stage: Stage)-&gt; Tuple [float, float]:
    model.eval()
    mask=getattr(data, f"{stage}_mask")
    logits = model(data.x, data.edge_index)[mask]
    preds = logits.argmax(dim=1)
    y = data.y[mask]
    loss = loss_fn(logits, y)
    acc = accuracy(preds, y)
    return loss.item(), acc
</code></pre>
<p>Thank you for your time and help!</p>
Module 'torchvision.datasets' has no attribute 'matcifar'

Dota2Player: <p>did anyone know how to solve this problem<br>
i have read the document of torchvision,but i couldn’t find ‘matcifar’,<br>
help  pls<br>
my pytorch vision is 0.3.1,torchvision is 0.2.0</p>

Dota2Player: <pre><code class="lang-auto">import torchvision.datasets as dset
train_dataset = dset.matcifar(imdb, train=True, d=3, medicinal=0)

test_dataset = dset.matcifar(imdb, train=False, d=3, medicinal=0)
</code></pre>

Dota2Player: <p>need help guys, i can’t google this either</p>

ptrblck: <p><code>matcifar</code> is not in <a href="https://pytorch.org/vision/stable/datasets.html"><code>torchvision.datasets</code></a>, so I’m unsure where this dataset is coming from.</p>

Dota2Player: <p>yes， i’m so confused, i really couldn’t find matcifar in [<code>torchvision.datasets</code>], but it’s used indeed in some code from part of paper i have read</p>

ptrblck: <p>I assume you’ve used a repository from the authors? If so, did you see, if the authors have forked <code>torchvision</code> and have added this particular dataset?</p>

Jeavan_Cooper: <p>I wonder how did you fix the problem,cause i am facing the exact same problem. Matcifar is in someone’s code and I can’t run the code.</p>
What's torch._C.infer_size() means?

dasen_W: <p>location:  torch\distributions\kl.py</p>
<p><code>combined_batch_shape = torch._C._infer_size(other._unbroadcasted_scale_tril.shape[:-2], self._unbroadcasted_scale_tril.shape[:-2])</code></p>
<blockquote>
<p><a href="https://pytorch.org/docs/stable/_modules/torch/distributions/kl.html#kl_divergence" class="inline-onebox" rel="noopener nofollow ugc">torch.distributions.kl — PyTorch 2.1 documentation</a></p>
</blockquote>

ptrblck: <p>This internal method seems to be used to compute the size needed to expand the tensors as described in the comment:</p>
<pre><code class="lang-auto">    # Expands term2 according to
    # inv(qcov) @ pcov = inv(q_tril @ q_tril.T) @ (pW @ pW.T + pD)
</code></pre>

dasen_W: <p>May I ask what is the difference between this internal method and the internal method of expanding the tensor size of the MultivariateNormal distribution?</p>
<p>like <code>torch.distributions.multivariate_normal.MultivariateNormal.expand()</code></p>
<p>Thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20"></p>

ptrblck: <p><code>_infer_size</code> expects <code>torch.Size</code> objects as its input and will also output a <code>torch.Size</code> object which can then be used for further shape processing of tensors.</p>
<p><code>torch.distributions.multivariate_normal.MultivariateNormal.expand</code> is defined as:</p>
<blockquote>
<p>Returns a new distribution instance (or populates an existing instance<br>
provided by a derived class) with batch dimensions expanded to<br>
<code>batch_shape</code>. This method calls :class:<code>~torch.Tensor.expand</code> on<br>
the distribution’s parameters. As such, this does not allocate new<br>
memory for the expanded distribution instance. Additionally,<br>
this does not repeat any args checking or parameter broadcasting in<br>
<code>__init__.py</code>, when an instance is first created.</p>
</blockquote>

dasen_W: <p>Thank you for your answer, I seem to understand the difference between the two, I am trying to use my own defined method to achieve the effect achieved by the _infer_size() method.</p>
<p>Thank you very much for your help! <img src="https://discuss.pytorch.org/images/emoji/apple/hugs.png?v=12" title=":hugs:" class="emoji" alt=":hugs:" loading="lazy" width="20" height="20"></p>
Gradient Rescaling in Backpropagation

seer_mer: <aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1511.06581">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1511.06581" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/1511.06581" target="_blank" rel="noopener nofollow ugc">Dueling Network Architectures for Deep Reinforcement Learning</a></h3>

  <p>In recent years there have been many successes of using deep representations
in reinforcement learning. Still, many of these applications use conventional
architectures, such as convolutional networks, LSTMs, or auto-encoders. In this
paper, we...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Dueling DQN, a simple improvement over DQN, splits fully connected layers into two branches. quote from the paper:<br>
“we rescale the combined gradient entering the last convolutional layer by 1/√2. This simple heuristic mildly increases stability”</p>
<p>How do we achieve this in PyTorch? I noticed that most of the popular Pytorch-based RL frameworks did not even implement this rescaling (tianshou, torchrl, …), is there any reason behind this?</p>
<p>Thanks in advance</p>

vmoens: <p>Hi <a class="mention" href="/u/seer_mer">@seer_mer</a><br>
I personally did not test it on my TorchRL implementation. My understanding is that because it “feeds” two networks, the gradient is rescaled as it there was only one.<br>
It feels a bit ad-hoc, not well documented and I don’t see any tests or precise mathematical justification for this choice. If we were to adopt this in TorchRL, we’d have to justify it just by quoting the paper (without being able to put a precise rationale behind it). Also, it’s the kind of things that is surprising to the user if they want to play around with the algorithm (what should I do with this line? Is it crucial?). Finally, it would be surprising that the return really depends on that trick.<br>
Overall, I think our role of developers of this kind of library is less to reproduce exactly a paper’s result rather than providing the tools for you to build upon their results. In this case, I think we all judge that this does not constitute the core contribution of the paper.<br>
Hope that answers the question.</p>

seer_mer: <p>Thank you, that helped me to understand the reason behind the design choice.</p>
How to train a model with a subset of the dataset in a subset of epochs

setare: <p>Hi, I have 25000 images dataset for train model, and I should train for 50000 epochs.<br>
I want to train this model with 1000 random images for each 10 epochs in other words, for 10 epoch my model run for randomly 1000 image in continue, in the next 10 epoch my model run for randomly 1000 image. How can I do that? What changes should I make in which part of source code?<br>
I attach source code that I want to add this condition.<br>
It is worth mentioning that I should set only batch=2 and batch=3, 4,… not work so I have a limit of batch_size.</p>
<p>for load collate function:</p>
<pre><code class="lang-auto">def collate_fn(data_bunch):

  '''
  A function for the dataloader to return a batch dict of given keys

  data_bunch: List of dictionary
  '''

  dict_data_bunch = {}

  for i in data_bunch:
    for (key, value) in i.items():
      if key not in dict_data_bunch:
        dict_data_bunch[key] = []
      dict_data_bunch[key].append(value)

  for key in list(dict_data_bunch.keys()):
      dict_data_bunch[key] = torch.stack(dict_data_bunch[key], axis = 0)

  if 'img' in dict_data_bunch:
    ## Pre-processing for ViT
    dict_data_bunch['img'] = vit_feat_extract(list(dict_data_bunch['img']),return_tensors = 'pt')['pixel_values']

  return dict_data_bunch
</code></pre>
<p>for load dataset:</p>
<pre><code class="lang-auto">class DataModule(pl.LightningDataModule):

  def __init__(self, train_dataset, val_dataset,  batch_size = 1):

    super(DataModule, self).__init__()
    self.train_dataset = train_dataset
    self.val_dataset = val_dataset
    self.batch_size = batch_size

  def train_dataloader(self):
    return DataLoader(self.train_dataset, batch_size = self.batch_size, 
                      collate_fn = collate_fn, shuffle = True, num_workers = 2, pin_memory = True)
  
  def val_dataloader(self):
    return DataLoader(self.val_dataset, batch_size = self.batch_size,
                    collate_fn = collate_fn, shuffle = False, num_workers = 2, pin_memory = True)
</code></pre>
Overriding a Tensor at a given index using `data_ptr`

tbouvier: <p>Hi everyone.</p>
<p>I’m using C++ libtorch for performance reasons. I have the following tensor representing a batch of images <code>[128, 3, 224, 224]</code>. I would like to replace the image at the given index <code>k</code> as follows:</p>
<pre><code class="lang-auto">auto nbytes = tensor[0].nbytes();
void* ptr = tensor.data_ptr() + k * nbytes; // I think this is fine as PyTorch is indexing row-wise
*ptr = ...
</code></pre>
<p>Does that look correct to you?</p>
<p>The reason I’m not using the <code>.index_put_</code> method of the tensor is because the latter increases its version counter of the tensor marking it as ‘dirty’, which I do not want.</p>
<p>Additional question: how do I make sure to free the overwritten memory in the above example?</p>
<p>Thanks</p>
Unexpected behavior when using two multiprocessing dataloaders

mendelsontau: <p>Hi,<br>
I am training a model on two different tasks (that require different data) in parallel. I’m using 2 data loaders one for each dataset where one dataset set is Iterable and the other is map-style. My training loop looks something like this</p>
<p>dataloader2_iterator = iter(dataloader2)<br>
for batch1 in dataloader1:<br>
batch2 = next(dataloader2_iterator)<br>
train_model(batch1, batch2)</p>
<p>When both dataloaders are multiprocess (i.e. num_workers &gt; 0) training works fine for a while then crashes with the following error:<br>
File “/home/gamir/DER-Roei/alon/anaconda3/envs/open_clip/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 652, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/gamir/DER-Roei/alon/anaconda3/envs/open_clip/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1347, in _next_data<br>
return self._process_data(data)<br>
File “/home/gamir/DER-Roei/alon/anaconda3/envs/open_clip/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1373, in _process_data<br>
data.reraise()<br>
File “/home/gamir/DER-Roei/alon/anaconda3/envs/open_clip/lib/python3.10/site-packages/torch/_utils.py”, line 461, in reraise<br>
raise exception<br>
RuntimeError: Caught RuntimeError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/home/gamir/DER-Roei/alon/anaconda3/envs/open_clip/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py”, line 302, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/home/gamir/DER-Roei/alon/anaconda3/envs/open_clip/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py”, line 52, in fetch<br>
return self.collate_fn(data)<br>
File “/home/gamir/DER-Roei/alon/anaconda3/envs/open_clip/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 180, in default_collate<br>
return [default_collate(samples) for samples in transposed] # Backwards compatibility.<br>
File “/home/gamir/DER-Roei/alon/anaconda3/envs/open_clip/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 180, in<br>
return [default_collate(samples) for samples in transposed] # Backwards compatibility.<br>
File “/home/gamir/DER-Roei/alon/anaconda3/envs/open_clip/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 146, in default_collate<br>
return torch.stack(batch, 0, out=out)<br>
RuntimeError: torch.cat(): input types can’t be cast to the desired output type Long</p>
<p>When only one of the dataloaders is multiprocess and the other is single process this doesn’t happen and training works fine all the way. I’ve debugged everything and there seems to be no reason for this error besides possible collision between the workers of the dataloaders when both are multiprocesses.</p>
<p>Does anyone know might be going wrong here and how I can fix this?<br>
Thanks</p>

ejguan: <p>What PyTorch version are you using?</p>
Supporting SNN in the future?

Chaofei: <p>PyTorch is one of the most popular deep learning frameworks, and spiking neural network is an emerging network model for training AI tasks. Do Pytorch have plans to support SNN in the future?</p>

ptrblck: <p><a href="https://github.com/jeshraghian/snntorch"><code>snntorch</code></a> or <a href="https://github.com/norse/norse"><code>norse</code></a> might be interesting for your use case. I’m not aware of any plans to add these libs into the native frameworks as I also don’t see a huge usage of these types of models.</p>
Torch-tensorrt::compile gets stuck bug caused by elimination exception

bjaeger: <p>What I want to do: TensorRT optimization of a PyTorch trained model which was previously saved as a torchscript-Model.<br>
But after calling:</p>
<pre><code class="lang-auto">auto trt_mod = torch_tensorrt::torchscript::compile(module, compile_settings);
</code></pre>
<p>… the process gets stuck in an infinite(?) loop. I can also observe that the GPU load drops back to 0% after about 1s.</p>
<p>According to this link: <a href="https://github.com/pytorch/TensorRT/pull/1409" class="inline-onebox" rel="noopener nofollow ugc">fix: fix compilation stuck bug caused by elimination exception by bowang007 · Pull Request #1409 · pytorch/TensorRT · GitHub</a> the issue should already be solved.</p>
<p>Version:</p>
<ul>
<li>Torch-TensorRT v1.3.0</li>
<li>PyTorch 1.13</li>
<li>CUDA 11.7</li>
<li>TensorRT 8.5</li>
<li>cudnn 8.5</li>
</ul>

ptrblck: <p>Also tracked <a href="https://github.com/pytorch/TensorRT/issues/1560">here</a>.</p>
Initialization gain for SiLU activation

Lelouch1: <p>As the current function <a href="https://pytorch.org/docs/stable/nn.init.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.init — PyTorch 1.13 documentation</a> does not give the gain for the SiLU activation <a href="https://pytorch.org/docs/stable/generated/torch.nn.SiLU.html" class="inline-onebox" rel="noopener nofollow ugc">SiLU — PyTorch 1.13 documentation</a> , do you know what value I should use ?</p>

tom: <p>I guess it boils down to how much and what kind of work you want to put into it.<br>
Some ideas:</p>
<ul>
<li>“if it is close to ReLU” use ReLU gain,</li>
<li>check the literatur (links given in the SiLU documentation),</li>
<li>experiment yourself.</li>
</ul>
<p>For the last one, maybe the discussion we had regarding SELU might be some inspiration:</p>
<aside class="quote quote-modified" data-post="3" data-topic="20854">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/calculate-gain-tanh/20854/3">Calculate_gain('tanh')</a> 
  </div>
  <blockquote>
    So I know nothing about these the paper you cite. And I haven’t looked at the papers for He’s and Glorot’s initialization, but here is Thomas Theory of Weight Initialization for the Theory Adverse™: 

Why do we care about initialization? We want “stability” of activation distributions in deep networks. If no “signal” arrives in layer 100, we are screwed.
How do we measure “signal”? Let’s just take standard deviation.
What is then a good gain? One where the standard deviation “converges” to a rea…
  </blockquote>
</aside>

<p>Of course, we (I) got scolded by the people who prefer computing the gain from the fixed point calculation, but if your goal is to train networks successfully, that might not be as crucial.</p>
<p>Best regards</p>
<p>Thomas</p>
Grouped linear layer

mathfinder: <p>I want to implement a linear function: y = [w_1x_1+b_1; w_2x_2+b_2;…;w_kx_k+b_k] the size of input x is (b,k*c), where b is the batch size, k is the number of groups, c is the number of channels. As shown below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b61331ee69510b37b604e91b3332252b4c8ed8dc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b61331ee69510b37b604e91b3332252b4c8ed8dc" title="linear"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b61331ee69510b37b604e91b3332252b4c8ed8dc_2_615x500.jpeg" alt="linear" data-base62-sha1="pYI1xnTgtpbB0A0u4SODqS1Knj6" width="615" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b61331ee69510b37b604e91b3332252b4c8ed8dc_2_615x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b61331ee69510b37b604e91b3332252b4c8ed8dc_2_922x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/b61331ee69510b37b604e91b3332252b4c8ed8dc.jpeg 2x" data-dominant-color="D9D7CA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">linear</span><span class="informations">1024×832 29.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And a Grouped convolution layer is also needed. The grouped convolution layer just like using the convolution setting groups=k. The difference is that the weight and bias is not expected to be shared between groups.<br>
Although this operation can be implemented with multiple linear or convolutional layers, I don’t think it is convenient.</p>
<p>Actually, I hope that different data may be processed with different weights. I want to transfer the batch to the channel, like (b, c, h, w)-&gt;(1, bxc, h, w), so that a sample corresponds to a group of channels, and each group of channels is processed with different weights. And then reshaping it back.</p>

aliutkus: <p>did you find an elegant way to do that ?</p>

aliutkus: <p>answering to my own question, I guess the easiest is just to go for <code>torch.matmul</code></p>

anton: <p>For <code>num_blocks</code> independent transforms, each from <code>dim_in</code> to <code>dim_out</code>, here is a workaround:</p>
<pre><code class="lang-auto">block_linear = torch.nn.Conv2d(
            in_channels=num_blocks * dim_in,
            out_channels=num_blocks * dim_out,
            kernel_size=1,
            groups=num_blocks,
            bias=True,
        )
</code></pre>
<p>Then, for <code>x</code> of size <code>[batch x num_blocks x dim_in]</code>, the mapping looks as follows:</p>
<pre><code class="lang-auto">x = x.reshape(batch, num_blocks * dim_in, 1, 1)
x = block_linear(x)  # output is of size [batch x (num_blocks * dim_out) x 1 x 1]
x = x.reshape(batch, num_blocks, dim_out)
</code></pre>
PyTorch matmul >3x slower than Tensorflow on AMD CPU

pytorch-user: <p>Hi, I recently noticed that matrix multiplication on my AMD Ryzen CPU is significantly faster in Tensorflow than in PyTorch. Is there any way to fix this, by using a different BLAS backend or something? I installed both frameworks using pip (torch==1.13.1 and tensorflow==2.11.0).</p>
<p>Code:</p>
<pre><code class="lang-auto">import tensorflow as tf
import torch
from timeit import default_timer as timer

#tf.config.threading.set_inter_op_parallelism_threads(1)
#tf.config.threading.set_intra_op_parallelism_threads(1)
shape = (8192, 8192)
with tf.device('/CPU:0'):
    x = tf.zeros(shape)
    tf.matmul(x, x)  # warmup
    start = timer()
    tf.matmul(x, x)
    end = timer()
    print(f'Tensorflow time: {end-start:g} s')

#torch.set_num_threads(1)
#torch.set_num_interop_threads(1)
x = torch.zeros(*shape)
torch.matmul(x, x)  # warmup
start = timer()
torch.matmul(x, x)
end = timer()
print(f'PyTorch time: {end-start:g} s')
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Tensorflow time: 2.02568 s
PyTorch time: 6.78812 s
</code></pre>
<p>The problem also exists for batch matrix multiplication, and also on a M1 Mac (to a lesser extent), but apparently not on an Intel CPU. I also noticed that Tensorflow uses the 16 virtual cores fully while PyTorch only uses the 8 logical ones, but the problem persists when limiting both to only use one thread (12.7s vs 34.5s).</p>
How to generate the score for a determined region of an image using Mask R-CNN

Mingqi_Jiang: <p>I want to change the RegionProposalNetwork of Mask R-CNN to generate the score for a determined region of an image using Mask R-CNN.</p>
<pre><code class="lang-auto">import torch
from torch import nn
import torchvision.models as models
import torchvision
from torchvision.models.detection import MaskRCNN
from torchvision.models.detection.anchor_utils import AnchorGenerator

model = models.detection.maskrcnn_resnet50_fpn(pretrained=True)

class rpn_help(nn.Module):
    def __init__(self,) -&gt; None:
        super().__init__()
    def forward(self,) :
        proposals=torch.tensor([ 78.0753,  12.7310, 165.6465, 153.7253])
        proposal_losses=0
        return proposals, proposal_losses

model.rpn= rpn_help
model.eval()
model(input_tensor) # input_tensor is an image
</code></pre>
<p>It takes error like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49272712ada9c5eea1cd386f7a77fe89abcd5f79.png" data-download-href="https://discuss.pytorch.org/uploads/default/49272712ada9c5eea1cd386f7a77fe89abcd5f79" title="WeChatbb686829d6c0f06106e53c1e3feecb55"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49272712ada9c5eea1cd386f7a77fe89abcd5f79_2_690x236.png" alt="WeChatbb686829d6c0f06106e53c1e3feecb55" data-base62-sha1="ar8JQuKaiqqlyZnP8reXd9p4Bo5" width="690" height="236" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49272712ada9c5eea1cd386f7a77fe89abcd5f79_2_690x236.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49272712ada9c5eea1cd386f7a77fe89abcd5f79_2_1035x354.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49272712ada9c5eea1cd386f7a77fe89abcd5f79_2_1380x472.png 2x" data-dominant-color="229624"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WeChatbb686829d6c0f06106e53c1e3feecb55</span><span class="informations">1572×538 268 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anyone know how to generate the score for a determined region of an image using Mask R-CNN?</p>
How can we do inference for the weight file trained by DDP (2 GPUs)?

salidw: <p>I found the below error when I did inference in the same way of weight file trained by single GPU.</p>
<p>“RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.”</p>
<p>Should I still need to initialize the distributed for inference on a single GPU ?</p>
<p>Please advise. Thanks.</p>

agu: <p>If you have a <code>DistributedDataParallel</code> model <code>ddp_model</code>, you can try to save the model parameters via <code>sd = ddp_model.module.state_dict()</code>. (Note the extra <code>.module</code>.) This saves the model parameters as if they were from a local non-DDP-version of the model. In that case, you should be able to load the model parameters via <code>local_model.load_state_dict(sd)</code> when running on a single CPU process / single GPU and without initializing a process group.</p>

salidw: <p>Hi Andrew, noted and thanks. Let me try and update you the result <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

salidw: <p><a class="mention" href="/u/agu">@agu</a>  I had revised my state_dict by using “model_ddp.module.state_dict()” and then used torch.save(dict_model,‘‘weight_ddp.pth’’). When I run my inference.py. There was still RuntimeError at the line of<br>
==&gt; “checkpoint = torch.load(checkpoint_file)”</p>
<p>The codes are:<br>
‘’’<br>
dict_model = {<br>
‘model’: model_ddp,<br>
‘state_dict’: model_ddp.module.state_dict(),<br>
‘optimizer’: optimizer.state_dict(),<br>
‘epoch’: epoch+1,<br>
}<br>
‘’’</p>
<p>checkpoint_file = ‘weight_ddp.pth’<br>
checkpoint = torch.load(checkpoint_file)  # RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.<br>
model = checkpoint[‘model’]<br>
model.load_state_dict(checkpoint[‘state_dict’])<br>
model.eval()</p>

ptrblck: <p>Avoid storing the entire model and just save the <code>state_dict</code> instead.<br>
In particular, remove the <code>'model': model_ddp</code> key-value pair from <code>dict_model</code>.</p>

salidw: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Noted and thanks.</p>
What's the best practice of batching and padding with transformer inference

DogeWatch: <p>I found that use batch size = 1 and do not pad is the fastest way to inference, is this conclusion correct? the best practice is do not batch and pad ?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10efc5b2679dad49c4464caeed57f2358c3d9b9d.png" data-download-href="https://discuss.pytorch.org/uploads/default/10efc5b2679dad49c4464caeed57f2358c3d9b9d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10efc5b2679dad49c4464caeed57f2358c3d9b9d_2_690x424.png" alt="image" data-base62-sha1="2pPlEmMjNh0EDHq1OnhMDQwpaJn" width="690" height="424" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10efc5b2679dad49c4464caeed57f2358c3d9b9d_2_690x424.png, https://discuss.pytorch.org/uploads/default/original/3X/1/0/10efc5b2679dad49c4464caeed57f2358c3d9b9d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/0/10efc5b2679dad49c4464caeed57f2358c3d9b9d.png 2x" data-dominant-color="F3F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">794×488 35.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

dreidizzle: <p>Well, if your batch size is 1 why pad - you only pad to make sure your data if B X L X D where N is the batch size, L is the max length, and D is the vector dimension - i.e. you pad so you get a tensor? Batches of size 18 would I guess take a longer time than batches of size 1, in general. However, typically have Predict Time(batch of size 18) &lt;&lt;&lt; 18*Predict Time(batch of size 1), so this is why batching helps.</p>

DogeWatch: <p>when batch size is 1, longest padding is same with do not pad. In my experiment, I found that after processing 4030 data, the total time of batch=1 is much less than that of batch=18, which is why I think batching is useless.</p>

dreidizzle: <p>Maybe this is due to the GPU but this is still surprising to me. In the case when you have batch 1, why do you need to pad? You are saying 1 predict is 584 s and a batch of size 18 is 948 s if you pad each sequence using the maximum length sequence in the batch? But then, your average predict time for the 18 case is 984 s / 18 no? That’s less than 584 s … Is this the right way to interpret the table? Let me know if I’m missing something …</p>

DogeWatch: <p>the total time is the time costs of processing 4030 items, so the average predict time of batch=1 and do not pad is  584 / 4030 seconds, and the average predict time of batch=18 and longest padding is 809 / 4030 seconds</p>
What version of torch does whisper want? (Raspbian)

bitrat: <p>Hi, I’m trying to run whisper on Raspbian buster.  There are apparently no wheels available for this platform (arm7l).  I compiled pytorch from source, but I realised my version (1.0.0) is quite old, so now compiling from the 1.7 branch</p>
<p>The error message from the earlier attempt was:</p>
<pre><code class="lang-auto">$ pip install torch===1.0.0 git+https://github.com/openai/whisper.git
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting git+https://github.com/openai/whisper.git
  Cloning https://github.com/openai/whisper.git to /tmp/pip-req-build-scewa0hz
  Running command git clone --filter=blob:none --quiet https://github.com/openai/whisper.git /tmp/pip-req-build-scewa0hz
  Resolved https://github.com/openai/whisper.git to commit 0b5dcfdef7ec04250b76e13f1630e32b0935ce76
  Preparing metadata (setup.py) ... done
ERROR: Could not find a version that satisfies the requirement torch===1.0.0 (from versions: none)
ERROR: No matching distribution found for torch===1.0.0
</code></pre>
<p>I get much the same error if I don’t specify the version.</p>
<p>I ran the setup.py script after building pytorch and it’s imported by python (3.9) when run interactively.</p>
<p>My questions are:</p>
<ul>
<li>how can I get the installer to tell me which version it does want?</li>
<li>is it finding any version at all?</li>
<li>are there similar version requirements for torchvision, torchaudio, etc?</li>
<li>what branch should I be using, 1.7 or master?</li>
</ul>
<p>I realise this is a lot of questions, but really I’d just be grateful for any info anyone can provide to help me get whisper running on the R-pi.  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>EDIT: <a href="https://github.com/openai/whisper/discussions/210" rel="noopener nofollow ugc">This</a> thread might help me, but I’d still appreciate any other advice.</p>
Pytorch version does not match the image name (Docker)

olream: <p>i pull the ‘pytorch/pytorch:1.9.1-cuda11.1-cudnn8-devel’ image. When i run it to create a new container, the pytorch version is upgraded to 1.10.0 in base conda environment.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23bddcbd190a768e3265c3b86ce059b04b9842c4.png" data-download-href="https://discuss.pytorch.org/uploads/default/23bddcbd190a768e3265c3b86ce059b04b9842c4" title="iShot_2022-12-19_08.30.51"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23bddcbd190a768e3265c3b86ce059b04b9842c4_2_689x135.png" alt="iShot_2022-12-19_08.30.51" data-base62-sha1="56bulzpeUPbiiOZlPnaGPli5o5m" width="689" height="135" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23bddcbd190a768e3265c3b86ce059b04b9842c4_2_689x135.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23bddcbd190a768e3265c3b86ce059b04b9842c4_2_1033x202.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/23bddcbd190a768e3265c3b86ce059b04b9842c4.png 2x" data-dominant-color="F0EDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">iShot_2022-12-19_08.30.51</span><span class="informations">1148×226 43.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Array ordering for Tensor creation

ignatius: <p>I want to create a <code>at::Tensor</code> from an flat array of floats. Let’s say I want to flat a multidimensional array with size (32, 3, 28, 28) to a flat array, and then use <code>torch::from_blob</code></p>
<p>Should I use row-major order flattening, or column-major ordering?</p>
<pre><code class="lang-auto">float data[32*3*28*28];
// fill data somehow... row-major or column-major falttening...
at::Tennsor tensor_data = torch::from_blob((float*)data, {32, 3, 28, 28});
</code></pre>

edeguine: <p>Did you ever figure this out?<br>
I am trying to convert an RGB8 image to import as a tensor and cannot figure out if i should swap rows and columns.</p>
<p>More precisely.<br>
I have a PNG of width 450 and height 400<br>
In Python I do</p>
<pre><code class="lang-auto">with open('test.png', "rb") as image_file:
    image = Image.open(image_file)
    image = image.convert("RGB")
data_transforms = transforms.Compose([transforms.ToTensor()])
image = data_transforms(image)
image = image.unsqueeze(0)
</code></pre>
<p>which give me a tensor of size torch.Size([1, 3, 400, 450]) (BxCxHxW)</p>
<p>but i don’t know how to do this properly in ios.</p>
CRNN datset and dataloader

Kalwerake: <p>Hi,</p>
<p>I’m trying to build a CRNN with a convolutional layer followed by LSTM. The inputs are sequences of correlation matrices ,How do I design the dataset class? Is there a way to use VIdeoFrameDataset, but set it to accept every frame? There’s only 100 and each one is important. Thank you</p>
Nightly Build Release Notes

provolone: <p>Where can I find the release notes, or documentation on the latest updates, for the PyTorch nightly build?</p>
<p>Thanks!</p>

ptrblck: <p>There are no daily release notes for nightly builds, but you could check the commit history before the branch was cut.</p>

fxmarty: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Looking at <a href="https://github.com/pytorch/pytorch/pull/26921" class="inline-onebox" rel="noopener nofollow ugc">[DO NOT DELETE OR MERGE] Nightly release by ezyang · Pull Request #26921 · pytorch/pytorch · GitHub</a> it seems like not all commits are included in the nightlies?</p>
<p>I don’t really understand either why <a href="https://github.com/pytorch/pytorch/tree/nightly" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch at nightly</a> is some commits ahead of master.</p>

ptrblck: <p>I don’t know how the <code>nightly</code> branch gets updated, but I would guess it might pick a “stable” commit from the <code>trunk</code> to avoid releasing “broken” binaries. <a class="mention" href="/u/malfet">@malfet</a> might know how the branch gets updated with more details.</p>

malfet: <p>Aggregated daily commits a pushed to the nightly branch. Since those commits are missing in trunk, GitHub thinks that it is ahead of it, though in reality it’s always behind.</p>

fxmarty: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/malfet">@malfet</a> Thanks a lot for your answer!</p>
Do I have to reset my lstm hidden state after each forward pass in reinforcment learning?

Defalt: <p>I want to make a model to learn a simple 2 player card game, where there is some randomness in the game, since a player can draw a random card. To, in a way, remember the cards drawn and what cards where played, I want to use some lstm cells, but I am not sure if I need to pass the previous hidden state of the last lstm cell to the next forward call or not.</p>
<p>My input for getting an action to play would be:<br>
<code>(1, representation_of_the_current_game_state), which means: </code>(batch_size, representation_of_the_current_game_state),`</p>
<p>while for training:<br>
<code>(game_states, representation_of_this_game_state)</code>.</p>
<p>I’m using PPO as my training algorithm and passing all game states of the game played as my batch. Thanks in advance</p>

J_Johnson: <p>The hidden state contains the learned representation of “memory” the model passes through time iterations. In your example of a card game, you are likely using the model forward pass per “turn”. And if you want the model to pass some memory values onward to the next “turn”, then you’ll need to do so via the hidden state. Resetting the hidden state will remove that memory.</p>
<p>In a game of cards, it may be helpful having a memory state as the current game state likely won’t represent the cards already used and no longer available in the deck - i.e. reasoning by deduction.</p>
<p>Keep in mind when passing the hidden layer, to use <code>.detach()</code> between each forward pass so as not to have issues with carrying gradients between passes.</p>

J_Johnson: <aside class="quote no-group" data-username="Defalt" data-post="1" data-topic="167138">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/b9bd4f/48.png" class="avatar"> Defalt:</div>
<blockquote>
<p>and passing all game states of the game played as my batch.</p>
</blockquote>
</aside>
<p>This might not be compatible with an RNN. To make it compatible, you can instead use batches to pass n-games turn at each time step. I.e. have 60 games playing at one time with each forward pass being turn <code>t+=1</code>.  You can use a <code>done</code> boolean array to represent the games finished, and filter for active games with <code>games[~done]</code> - that is games not done.</p>
Why is tensorboard reporting no TensorCores?

MartinPerry1: <p>I have this simple code:</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
from torch.profiler import profile, record_function, ProfilerActivity


#https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html

if __name__ == '__main__':

    #https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html
    torch.set_float32_matmul_precision("high")    
    #same as
    #torch.backends.cuda.matmul.allow_tf32 = True

    model = models.resnet50().cuda()
    model.eval()
    inputs = torch.randn(5, 3, 224, 224).cuda()
   
    traceHandler = torch.profiler.tensorboard_trace_handler('trace_test')

    with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
                 record_shapes=True,
                 profile_memory=True,
                 with_stack=True,
                 on_trace_ready=traceHandler,
                 ) as prof:
        with record_function("model_inference"):
            with torch.cuda.amp.autocast():
                model(inputs)
</code></pre>
<p>When I open generated image in TensorBoard, I can see:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/aff7b86c9c4655aa8bf3abf8a0dc75fe09bc146c.png" data-download-href="https://discuss.pytorch.org/uploads/default/aff7b86c9c4655aa8bf3abf8a0dc75fe09bc146c" title="result_tb"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/aff7b86c9c4655aa8bf3abf8a0dc75fe09bc146c.png" alt="result_tb" data-base62-sha1="p6Gilg9lY4Kn2pw8a0rj921AhYM" width="690" height="360" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">result_tb</span><span class="informations">867×453 13.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why are TensorCores not used? Is this problem with TensorBoard results and profile module?</p>
<p>I have RTX3080 Ti, torch 1.13.1 torch-tb-profiler 0.4.0 and CUDA 11.7. Python is 3.8.13.</p>

eqy: <p>I’m not 100% sure what is being reported by TensorBoard is accurate here. Could you check if toggling <code>https://pytorch.org/docs/stable/backends.html#torch.backends.cudnn.torch.backends.cudnn.allow_tf32</code> is changing the performance of your workload?</p>
Can we change the communication speed of GLOO/NCCL manually?

Yuki_Takezawa: <p>PyTorch supports GPU-to-GPU communication in a distributed learning environment via gloo and NCCL.</p>
<p>I want to analyze the phenomenon of distributed learning with a high latency value, a low bandwidth value, or a slow communication speed. So, is it possible to manually change (i.e. decrease) the communication speed between GPUs? Additionally, can we manually change the latency value and bandwidth of the GPU-to-GPU communication?</p>
<p>Thank you.</p>

mrshenli: <p>Hey <a class="mention" href="/u/yuki_takezawa">@Yuki_Takezawa</a>, AFAIK, there is no user APIs to directly change Gloo/NCCL speed. If you would like to mimic the behavior of a slower interconnect, can this be done at a higher layer? E.g., can you wrap collective APIs to manually insert delay?</p>

Yuki_Takezawa: <p>Thank you for your suggestion.</p>
<p>Yes, we can insert a delay (e.g., <code>time.sleep(1.0)</code>) before <code>dist.send()</code> and can simulate the slow communication network. However, I want to simulate a slow communication network in a more realistic environment.</p>
<p>For instance, if we run distributed learning of PyTorch with multiple virtual machines, we can manually set the communication speed by <code>tc</code> command. However, since it is complicated to set up this environment, I am looking for an easier way.</p>

Chenning_Li: <p>Good point, Yuki. Actually, I have the same demand to simulate a wireless network. Wonder if you have determined the solution?</p>
<p>Thanks</p>

Yuki_Takezawa: <p>I found that using docker is a relatively easy way.<br>
The environment of a network between docker containers (e.g., latency, bandwidth) can be also set by tc command.</p>
Setting the values at specific indices of a 2d tensor

standshik: <p>The torch.topk returns the topk values and indices of a 2d tensor. How do I set those index values to 1 and everything else to zero?<br>
To give an example, suppose I have a 2d tensor a =tensor([[1,2,3],[6,5,4]]). If I use value,index = torch.topk(a,2,largets=True) then index = tensor([[2,1],[0,1]]). I want the output to be a tensor b = tensor([[0,1,1],[1,1,0]]) without using a for loop.</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">b = torch.zeros_like(a)
b[torch.arange(b.size(0)), index.t()] = 1.
print(b)
# tensor([[0, 1, 1],
#         [1, 1, 0]])
</code></pre>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor []] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to com

s_n: <p>Hi,</p>
<p>Can someone help me to solve the below error? I am not sure why its occurring. In one of the posts (<a href="https://discuss.pytorch.org/t/one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-cuda-floattensor-3-48-3-3-is-at-version-2-expected-version-1-instead/83241" class="inline-onebox">One of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 48, 3, 3]] is at version 2; expected version 1 instead</a>), <a class="mention" href="/u/alband">@albanD</a> pointed out that the error is occurring because of doing optimizer.step() before calling loss.backward(). But in my code I call loss.backward and after that I call optimizer.step().</p>
<p>Also, I get this error for the second iteration of the for loop (which is over the minibatches).</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor []] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>

ptrblck: <p>Check if you are reusing some outputs or activations from the previous iteration without detaching them from the old computation graph, as this would also cause the issue as seen in this small example:</p>
<pre><code class="lang-python">model = nn.Linear(10, 10)
data = torch.randn(1, 10)
target = torch.randn(1, 10)
optimizer = torch.optim.Adam(model.parameters(), lr=1.)
criterion = nn.MSELoss()

# iter0 - works
optimizer.zero_grad()
out = model(data)
loss = criterion(out, target)
loss.backward()
optimizer.step()

# iter1 - works for a new forward pass and thus a new computation graph
optimizer.zero_grad()
out = model(data)
loss = criterion(out, target)
loss.backward()
optimizer.step()

# iter2 - fails since you are trying to backpropagate through iter2 and iter1
optimizer.zero_grad()
out = model(out) # !!!
loss = criterion(out, target)
loss.backward()
# RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
</code></pre>

s_n: <p>Hi,</p>
<p>Thanks!<br>
Actually, that was the reason for the error.</p>