akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to compute mean pixel values of training images and subtract them from all images

Gordon_Lim: <p>I want to compute the mean pixel values of <em>all</em> training images and subtract them from all images as opposed to finding the mean of a single image and subtracting it from the pixels in that image.</p>
<p>This is the specific line in the paper which are trying to reproduce the results for:</p>
<blockquote>
<p>After resizing the images, we compute the mean pixel values<br>
of the training images and subtract them from all images to<br>
center the training data.</p>
</blockquote>
<p>and here’s the link to the full paper <a href="https://arxiv.org/pdf/1704.03557.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1704.03557.pdf</a></p>
<p>This is what I have now:</p>
<pre><code class="lang-auto">class RvlCdipDataset(Dataset):
    def __init__(self, labels_file, img_dir, transform=None, target_transform=None):
        self.img_labels = pd.read_table(labels_file, header=None, names = ["image_path", "label"],sep=" ")
        self.img_dir = img_dir
        self.transform = transform
        self.target_transform = target_transform

    def __len__(self):
        return len(self.img_labels)

    def __getitem__(self, idx):
        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])
        image = Image.open(img_path).convert('RGB')
        label = self.img_labels.iloc[idx, 1]
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)
        return image, label

dataset = RvlCdipDataset('labels/train.txt', 
                         'images/', 
                         transform=transforms.Compose([transforms.ToTensor(),
                                                       transforms.Resize((224,224))]))
</code></pre>
How to monitor and when to retrain the object detection model in production?

iamexperimentingnow: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , I recently moved regression model to production and I’m monitoring the model drift and data drift using statistical tests, based on their distributions i retrain the model.</p>
<p>Could you please tell me how to monitoring the object detection model and detect drifts ?</p>
<p>Do you use statistical test to detect drifts? If yes, how do you do that? And which value you as an input to detect?</p>
<p>Please guide me? Even if you could provide any relevant article also would help me</p>
<p>Thanks in advance😊</p>
Memory Fills Up Reading Data From HDF5 File From Pytorch Dataloader

Curiouser: <p>I created the <strong>getitem</strong> method of the DataLoader class in the Pytorch library as follows.</p>
<pre><code>    with h5py.File(self.path_video,'r') as hf:
        sample = hf[video_id][...]
</code></pre>
<p>HDF5 files were also created as key-value pairs.</p>
<p>Sometimes, when reading data, the data in the file is read to the end with a memory occupancy rate of 30-40%. Sometimes this memory starts to fill up, and when it is completely full, the program crashes.</p>
<p>When the memory is full, I get the following error and the program crashes.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e5af286ef9edf5842ce4bcf7f323b90f01d8b4a7.jpeg" alt="3" data-base62-sha1="wLSC7ES7pVcLleniZ4g84eILoYT" width="605" height="407"></p>
<p>How can i solve this problem.</p>
Pytorch equivalent of tensorflow keras StringLookup?

jiwidi: <p>I’m working with pytorch and I’m really missing a layer from keras: <code>tf.keras.layers.StringLookup</code> that helped with processing of ids. Is there any workaround to do something similar with pytorch?</p>
<p>And example of the functionality im looking for:</p>
<pre><code class="lang-auto">vocab = ["a", "b", "c", "d"]
data = tf.constant([["a", "c", "d"], ["d", "a", "b"]])
layer = tf.keras.layers.StringLookup(vocabulary=vocab)
layer(data)

Outputs:
&lt;tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[1, 3, 4],
       [4, 1, 2]])&gt;
</code></pre>

jiwidi: <p>I found a solution:</p>
<pre><code class="lang-auto">from torchnlp.encoders import LabelEncoder

data = ["a", "c", "d", "e", "d"]
encoder = LabelEncoder(data, reserved_labels=['unknown'], unknown_index=0)

enl = encoder.batch_encode(data)

print(enl)
</code></pre>
<pre><code class="lang-auto">tensor([1, 2, 3, 4, 3])
</code></pre>
What's Happening inside MultiHeadAttention Module in New Torch Version?

Haider_Ali_Shuvo: <p>What sort of projections exactly happens inside Multi-Head-Attention Module ?<br>
In Pytorch 1.9.1 All of Key,Query and Value needed to be of same dimentions(Embed Dimention) , Then Key,Query was mapped to Key Dimentions(called Kdim) and Value was mapped to Value Dimention(Called Vdim) before applying attention mechanism . And There was a mapping from Num HeadxValue dimention to Embed Dimention in the output . This was consistent with Transformer Paper .<br>
In Pytorch 1.10 , This seems to have changed . Query , Key and Value each can be of separate dimentions now . What sort of projection and calculatons are happening inside the Block in new updated Torch ?</p>
RuntimeError: shmem_size <= sharedMemPerBlock INTERNAL ASSERT FAILED

qianertongre: <p>When running the codes from <a href="https://github.com/CSUBioGroup/DeepLncLoc" class="inline-onebox" rel="noopener nofollow ugc">GitHub - CSUBioGroup/DeepLncLoc: A deep learning-based lncRNA subcellular localization predictor</a>, I got the error message:<br>
Traceback (most recent call last):<br>
File “train.py”, line 34, in <br>
model.cv_train(dataClass, trainSize=1, batchSize=batchSize, stopRounds=-1, earlyStop=10,<br>
File “D:\GitHub\DeepLncLoc\model\DL_ClassifierModel.py”, line 24, in cv_train<br>
res = self.train(dataClass,trainSize,batchSize,epoch,stopRounds,earlyStop,saveRounds,optimType,lr,weightDecay,<br>
File “D:\GitHub\DeepLncLoc\model\DL_ClassifierModel.py”, line 57, in train<br>
loss = self._train_step(X,Y, optimizer)<br>
File “D:\GitHub\DeepLncLoc\model\DL_ClassifierModel.py”, line 175, in <em>train_step<br>
loss.backward(retain_graph=True)<br>
File “C:\ProgramData\Anaconda3\envs\newenv\lib\site-packages\torch\tensor.py”, line 195, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File "C:\ProgramData\Anaconda3\envs\newenv\lib\site-packages\torch\autograd_<em>init</em></em>.py", line 97, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: shmem_size &lt;= sharedMemPerBlock INTERNAL ASSERT FAILED at C:/w/1/s/tmp_conda_3.8_075542/conda/conda-bld/pytorch_1579852615070/work/aten/src/ATen/native/cuda/AdaptiveAveragePooling.cu:668, please report a bug to PyTorch.<br>
I use a 1060 6G graphic card with 16G memory.<br>
I tried Pytorch 1.4 with cuda 9.2 and 1.8  with cuda 10.2, but it seems that the version is not a problem.<br>
Could anyone help me to know how I could decrease the shmem_size or any methods to get rid of this error? Thank you!</p>
Constructing parameter groups in pytorch

blade: <p>In the <code>torch.optim</code> <a href="https://pytorch.org/docs/stable/optim.html" rel="noopener nofollow ugc">documentation</a>, it is stated that model parameters can be grouped and optimized with different optimization hyperparameters. It says that</p>
<blockquote>
<p>For example, this is very useful when one wants to specify per-layer<br>
learning rates:</p>
<pre><code>optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>
<p>This means that <code>model.base</code>’s parameters will use the default<br>
learning rate of <code>1e-2</code>, <code>model.classifier</code>’s parameters will use a<br>
learning rate of <code>1e-3</code>, and a momentum of <code>0.9</code> will be used for all<br>
parameters.</p>
</blockquote>
<p>I was wondering how to define such groups that have <code>parameters()</code> attribute. What came to my mind was something in the form of</p>
<pre><code>class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.base()
        self.classifier()

        self.relu = nn.ReLU()

    def base(self):
        self.fc1 = nn.Linear(1, 512)
        self.fc2 = nn.Linear(512, 264)

    def classifier(self):
        self.fc3 = nn.Linear(264, 128)
        self.fc4 = nn.Linear(128, 964)

    def forward(self, y0):

        y1 = self.relu(self.fc1(y0))
        y2 = self.relu(self.fc2(y1))
        y3 = self.relu(self.fc3(y2))

        return self.fc4(y3)
</code></pre>
<p>How should I modify the snippet above to be able to get <code>model.base.parameters()</code>? Is the only way to define a <code>nn.ParameterList</code> and explicitly add <code>weight</code>s and <code>bias</code>es of the desired layers to that list? What is the best practice? Is there any other ways other than using <code>nn.Sequential</code>?</p>

tom: <p>While structuring it (with separate nn.Module classes for base and classifier) would look more natural to me in your example, you can pass any list/iterator to the optimizers for the parameters, not just ParameterLists / Sequential.</p>
<p>For example:</p>
<pre><code class="lang-auto">some_params = []
some_params.extend(fc1.parameters())
some_params.extend(fc2.parameters())

other_params = [p for p in model.parameters() if p not in set(some_parameters)]
</code></pre>
<p>You could also filter by name with <code>[p for n, p in model.named_parameters() if n.endswith('bias')]</code> or so.</p>
<p>Best regards</p>
<p>Thomas</p>
How to use MaxPool2d

weizhen_song: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f54f265359dbf32021552f232595e6b0f0e26f26.png" data-download-href="https://discuss.pytorch.org/uploads/default/f54f265359dbf32021552f232595e6b0f0e26f26" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f54f265359dbf32021552f232595e6b0f0e26f26_2_563x500.png" alt="image" data-base62-sha1="z06yzUID9AqPoNTcBvmXdxtqehM" width="563" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f54f265359dbf32021552f232595e6b0f0e26f26_2_563x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f54f265359dbf32021552f232595e6b0f0e26f26_2_844x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f54f265359dbf32021552f232595e6b0f0e26f26.png 2x" data-dominant-color="2D2D27"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">945×839 90.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>when I use it, it aways say:</p>
<p><strong>UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  …\c10/core/TensorImpl.h:1156.)<br>
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)</strong></p>
<p>how can I deal it?<br>
I hope get your help</p>

ptrblck: <p>You can ignore it and update PyTorch (and torchvision) to the latest release, as the warning was unhelpful and was removed.</p>

Brando_Miranda: <p>but updating libraries is scary <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"> what if it breaks all my code…</p>
<p>ok will do this monday I guess when I have more energy…</p>
<p>thanks btw! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

ptrblck: <p>You could also ignore the warning, if you prefer to keep an older release.<br>
Before changing your current setup, create a new conda/pip environment, install the latest libs there, and check your code. This would keep your current setup while you can test the updated one.</p>

Brando_Miranda: <p>I know…not looking forward to it…thanks regardless <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why are the sizes of `tensor` and `tensor.detach()` the same?

AlJazari: <p>I was expecting that the size of a given tensor with <code>x.auto_grad=True</code> should be different from <code>x.detach()</code> or <code>y = x.detach().clone()</code> Can someone explain why the sizes of both objects are the same?</p>
<p>Thanks</p>
<p><strong>EDIT:</strong> I mean the actual size on desk not the shape of the tensor.</p>

zetyquickly: <p>Hello <a class="mention" href="/u/aljazari">@AlJazari</a><br>
Let me clarify, what do you expect to see when tensor requires and does not requires grad?</p>
<p>If you detach tensor from the graph then it’s <code>requires_grad</code> becomes <code>False</code> and <code>grad</code> property becomes <code>None</code>. In other words, the size of a tensor (means shape) stays the same, but <code>grad</code> property disappears</p>

AlJazari: <p>I was assuming the graph itself is attached to the tensor, so I was expecting that the size (i.e., actual size on desk in bytes) is different for a tensor with a graph versus a tensor detached from the graph.</p>
<p>For example, if the information in the graph is irrelevant to me, then the memory occupied by storing the tensor on desk is the same whether this tensor includes the graph or detached from it.</p>
Convert tensor of size 768 to 128

Alymostafa: <p>I want to make a projection to the tensor of shape <code>[197, 1, 768]</code> to <code>[197,1,128]</code> using<br>
<code>nn.Conv()</code></p>

ptrblck: <p>You could use a kernel size and stride of 6, as that’s the factor between the input and output temporal size:</p>
<pre><code class="lang-python">x = torch.randn(197, 1, 768)
conv = nn.Conv1d(in_channels=1, out_channels=1, kernel_size=6, stride=6)
out = conv(x)
print(out.shape)
&gt; torch.Size([197, 1, 128])
</code></pre>

Alymostafa: <p>Is there a solution that is independent of batch size (second dimension)?</p>

ptrblck: <p>The second dimension of the input is the channel dimension while the first dimension is the batch dimension. <code>nn.Conv1d</code> is independent w.r.t. the batch size.</p>
Indexing 3D tensor with another 3D

Sentient07: <p>Hello.</p>
<p>I would like to do this operation below without the pythonic for loop in plain pytorch. How do I go about?</p>
<pre><code class="lang-auto">torch.stack([v[f] for v,f in zip(verts_padded, faces_padded)])
</code></pre>
<p>Where, <code>verts_padded</code> is of shape (B, N, 3) <code>faces_padded</code> is of shape (B,M,3).</p>
<p>I see a related question from 3yrs ago but unanswered : <a href="https://discuss.pytorch.org/t/indexing-3d-tensor-using-3d-index/32372/2" class="inline-onebox">Indexing 3D Tensor using 3D Index - #2 by HuynhLam</a></p>
<p>Thank you</p>

ptrblck: <p>Given your shapes and the posted for loop, this would work:</p>
<pre><code class="lang-python">B, N, M = 4, 5, 6
verts_padded = torch.randn(B, N, 3)
faces_padded = torch.randint(0, B, (B, M, 3))

out = torch.stack([v[f] for v,f in zip(verts_padded, faces_padded)])

res = verts_padded[torch.arange(B).unsqueeze(1).unsqueeze(2), faces_padded]

print((out == res).all())
&gt; tensor(True)
</code></pre>

Sentient07: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> sorry for my late reply, I did not receive notification about your response.</p>
<p>Your solution works, but it’s still not intuitive for me regarding the unsqueeze that you do. Can you please elaborate on it? Thanks</p>

ptrblck: <p>The <a href="https://numpy.org/doc/stable/reference/arrays.indexing.html#purely-integer-array-indexing">numpy - Advanced Indexing docs</a> can explain it better than I could, so take a look at the linked section and in particular into the usage of <code>np.newaxis</code>:</p>
<pre><code class="lang-python">rows = np.array([0, 3], dtype=np.intp)

columns = np.array([0, 2], dtype=np.intp)

rows[:, np.newaxis]
array([[0],
       [3]])

x[rows[:, np.newaxis], columns]
array([[ 0,  2],
       [ 9, 11]])
</code></pre>
<p>and the comparison to the indexing approach without broadcasting (previous code snippet).</p>
Embedding dimension mismatch in CVAE

AvirupJU: <p>I am trying to implement a conditional VAE but I’m getting the following error from the encoder embeddings:<br>
<code>RuntimeError: Tensors must have same number of dimensions: got 4 and 2</code><br>
Here’s how I’ve implemented the encoder and decoder networks:</p>
<pre><code class="lang-auto">class Unet_Encoder(nn.Module):

    def __init__(self, num_classes, in_channels=3):

        super(Unet_Encoder, self).__init__()

        self.embed = nn.Embedding(num_classes, 256*256)

        self.down_1 = Unet_DownBlock(in_channels+1, 32, normalize=False)

        self.down_2 = Unet_DownBlock(32, 64)

        self.down_3 = Unet_DownBlock(64, 128)

        self.down_4 = Unet_DownBlock(128, 256)

        self.down_5 = Unet_DownBlock(256, 256)

        self.linear_encoder = nn.Linear(256 * 8 * 8, 512)

        self.dropout = nn.Dropout(0.5)

    def forward(self, x, labels):

        embedding = self.embed(labels)

        embedding = embedding.view(labels.shape[0], 1, 256, 256)

        x = torch.cat([x, embedding], dim=1)

        x = self.down_1(x)

        x = self.down_2(x)

        x = self.down_3(x)

        x = self.down_4(x)

        x = self.down_5(x)

        x = torch.flatten(x, start_dim=1)

        x = self.linear_encoder(x)

        x = self.dropout(x)

        return x

class Unet_Decoder(nn.Module):

    def __init__(self, num_classes, out_channels=3):

        super(Unet_Decoder, self).__init__()

        self. embed = nn.Embedding(num_classes, 128)

        self.linear_1 = nn.Linear(256, 8*8*256)

        self.dropout = nn.Dropout(0.5)

        self.deconv_1 = Unet_UpBlock(256, 256)

        self.deconv_2 = Unet_UpBlock(256, 128)

        self.deconv_3 = Unet_UpBlock(128, 64)

        self.deconv_4 = Unet_UpBlock(64, 32)

        self.final_image = nn.Sequential(*[nn.ConvTranspose2d(32, out_channels,

                                        kernel_size=4, stride=2,

                                        padding=1), nn.Tanh()])

    def forward(self, x, labels):

        embedding = self.embed(labels)

        x = torch.cat([x, embedding], dim=1)

        x = self.linear_1(x)

        x = x.view(-1, 256, 8, 8)

        x = self.dropout(x)

        x = self.deconv_1(x)

        x = self.deconv_2(x)

        x = self.deconv_3(x)

        x = self.deconv_4(x)

        x = self.final_image(x)

        return x
</code></pre>
<p>It is to be noted that when I’m testing the entire model on a dummy tensor and label it works just fine:</p>
<pre><code class="lang-auto">x = torch.randn((1,1,256,256))

    model = VAE_Model(num_classes=11)

    label = torch.tensor([5]) 

    out = model(x, label)

    print(label.shape) 

    print(out.shape)
</code></pre>
<p>Output:<br>
torch.Size([1])<br>
torch.Size([1, 1, 256, 256])</p>
<p>The issue is when I’m trying to train it on MNIST dataset I’m getting the said error.<br>
I tried printing the shape of the labels and it is showing torch.Size([12]) which is correct because my batch size is 12. Where am I going wrong?</p>
Why nn.ModuleList is not a subclass of Sequence

techkang: <p>As the code shown below, nn.ModuleList is not subclass of MutableSequence.</p>
<pre><code class="lang-auto">In [4]: from collections.abc import MutableSequence, Mapping

In [5]: import torch.nn as nn

In [6]: issubclass(nn.ModuleList, MutableSequence)
Out[6]: False

In [7]: issubclass(nn.ModuleDict, Mapping)
Out[7]: False
</code></pre>
<p>The source code of  <code>ModuleList</code> is</p>
<pre><code class="lang-auto">class ModuleList(Module):
</code></pre>
<p>Should we change the code to</p>
<pre><code class="lang-auto">class ModuleList(Module, MutableSequence):
</code></pre>
<p>Why or why not?</p>

tom: <p>The mutable sequence type comes with <a href="https://docs.python.org/3/library/stdtypes.html#mutable-sequence-types" rel="noopener nofollow ugc">quite a few functions that need to be implemented</a>, I’m not sure all of them are there today. Is it worth to implement them? I don’t know.</p>
<p>Best regards</p>
<p>Thomas</p>

techkang: <p>Thanks. I also find that <code>ModuleDict</code> implement all the interface except for <code>get</code> in <code>Mapping</code>. I hope that someday they will implement this interface so I can reopen this question.</p>

tom: <p>You could always send a patch it and see if they like it. If you test on ModuleDict, you’ll see what the general sentient is without risking to waste too much effort…</p>

techkang: <p>I came across this idea because when I want to reverse a ModuleList instance, my IDE (PyCharm) said that <code>reversed</code> function only accept <code>Sequence</code> or <code>Reversible[_T]</code>.</p>
Why DistributedSampler older versions have different way of sampling indices?

vadimkantorov: <p>Current PyTorch core one splits the whole dataset in interleaved fashion:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/d5988c5eca0221e9ef58918e4f0b504940cb926a/torch/utils/data/distributed.py#L116">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/d5988c5eca0221e9ef58918e4f0b504940cb926a/torch/utils/data/distributed.py#L116" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/d5988c5eca0221e9ef58918e4f0b504940cb926a/torch/utils/data/distributed.py#L116" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/d5988c5eca0221e9ef58918e4f0b504940cb926a/torch/utils/data/distributed.py#L116</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="106" style="counter-reset: li-counter 105 ;">
          <li>        if padding_size &lt;= len(indices):</li>
          <li>            indices += indices[:padding_size]</li>
          <li>        else:</li>
          <li>            indices += (indices * math.ceil(padding_size / len(indices)))[:padding_size]</li>
          <li>    else:</li>
          <li>        # remove tail of data to make it evenly divisible.</li>
          <li>        indices = indices[:self.total_size]</li>
          <li>    assert len(indices) == self.total_size</li>
          <li>
          </li>
<li>    # subsample</li>
          <li class="selected">    indices = indices[self.rank:self.total_size:self.num_replicas]</li>
          <li>    assert len(indices) == self.num_samples</li>
          <li>
          </li>
<li>    return iter(indices)</li>
          <li>
          </li>
<li>def __len__(self) -&gt; int:</li>
          <li>    return self.num_samples</li>
          <li>
          </li>
<li>def set_epoch(self, epoch: int) -&gt; None:</li>
          <li>    r"""</li>
          <li>    Sets the epoch for this sampler. When :attr:`shuffle=True`, this ensures all replicas</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><code>indices = indices[self.rank:self.total_size:self.num_replicas]</code></p>
<p>Code in <a href="http://github.com/nvlabs/wetectron" rel="noopener nofollow ugc">github.com/nvlabs/wetectron</a> (which is derived from older version of Detectron2):<br>
<a href="https://github.com/NVlabs/wetectron/blob/69436c0c6bb77e8dddb6f3f2b5d912f0d1ecd8fb/wetectron/data/samplers/distributed.py#L57" class="inline-onebox" rel="noopener nofollow ugc">wetectron/distributed.py at 69436c0c6bb77e8dddb6f3f2b5d912f0d1ecd8fb · NVlabs/wetectron · GitHub</a><br>
<code>indices = indices[offset : offset + self.num_samples]</code></p>
<p>Why did older Detectron2 use a different way of sampling?</p>
GhostBatchNorm with channels_last

Minh_Nguyen: <p>I have tried implementing GhostBatchNorm as follows:</p>
<pre><code class="lang-auto">class GhostBatchNorm(nn.BatchNorm2d):
    def __init__(self, num_features, num_splits, **kw):
        super().__init__(num_features, **kw)

        running_mean = torch.zeros(num_features * num_splits)
        running_var = torch.ones(num_features * num_splits)

        self.weight.requires_grad = False
        self.num_splits = num_splits
        self.register_buffer("running_mean", running_mean)
        self.register_buffer("running_var", running_var)

    def train(self, mode=True):
        if (self.training is True) and (mode is False):
            # lazily collate stats when we are going to use them
            self.running_mean = torch.mean(
                self.running_mean.view(self.num_splits, self.num_features), dim=0
            ).repeat(self.num_splits)
            self.running_var = torch.mean(
                self.running_var.view(self.num_splits, self.num_features), dim=0
            ).repeat(self.num_splits)
        return super().train(mode)

    def forward(self, input):
        n, c, h, w = input.shape
        if self.training or not self.track_running_stats:
            return F.batch_norm(
                input.contiguous().view(-1, c * self.num_splits, h, w),
                self.running_mean,
                self.running_var,
                self.weight.repeat(self.num_splits),
                self.bias.repeat(self.num_splits),
                True,
                self.momentum,
                self.eps,
            ).view(n, c, h, w)
        else:
            return F.batch_norm(
                input,
                self.running_mean[: self.num_features],
                self.running_var[: self.num_features],
                self.weight,
                self.bias,
                False,
                self.momentum,
                self.eps,
            )
</code></pre>
<p>However, when using this GhostBatchNorm implementation with channels_last, I actually saw an increase in training time compared to channels_first. This is different from when I try torch.nn.BatchNorm2d, using channels_last decreases training time compared to channels_first.</p>
<p>So can anyone explain the reason for the training time increase and have suggestions on how to implement GhostBatchNorm with channels_last in Pytorch? Thanks in advance.</p>
Training model with ImageNet

KanZa: <p>I am training a model on ImageNet Dataset but cannot resolve the following error</p>
<p>lib/python3.6/site-packages/torchvision/datasets/folder.py", line 231, in <strong>getitem</strong><br>
path, target = self.samples[index]<br>
IndexError: list index out of range</p>

ptrblck: <p>The error is quite strange, as the dataset should check the number of samples and use it as the range for the indices. Could you add a print statement into the <code>__getitem__</code> and check which value <code>index</code> has as well as was <code>len(dataset)</code> returns?</p>

KanZa: <p>I resolved the error as the number of training images was set as 50,000 images for other datasets so indexing was bigger than the images I wanted to train.<br>
Thank you!</p>
Need help to optimize speed of gather/scatter heavy model

nlgranger: <p>I am trying to optimize a model with a lot of gather, scatter operations.<br>
So far, my optimization to the code have focused on:</p>
<ul>
<li>use inplace ops and <code>out=</code> parameter to avoid extra copies</li>
<li>replace long indices by int indices everywhere (cuda seems to be designed for int indices right?), my layers now use torch.index_select and torch.index_add and torch.searchsorted, all of which support int32 indices.</li>
<li>fp16 everywhere with padding to make shapes multiples of 64</li>
</ul>
<p>Using the revamped torch profiler (thanks!) I can see that most of the time is still spent in index_add_ and index_select operations.<br>
This is happening on an Nvidia A100 GPU, on a P1000 most of the time is spent in algebra operations as expected.<br>
Are there any tips or known limitations or good practice to optimize this kind of workload?</p>
How to apply temporal conv on 1D data?

seyeeet: <p>I am a little confuse regarding applying temporal convolution on the 1D data and appreciate it if someone can help me understand if I am doing it right.<br>
I have temporal data (<code>x</code>) in shape of <code>B x C x T</code> (batch,channels,times). If I define <code>conv=nn.conv1d(C,C,3)</code>, would it means that I am applying convolution temporally with kernel 3 on the temporal dimension when I do <code>conv(x)</code>?</p>

ptrblck: <p>Yes, your description is correct.<br>
The conv layer will use a kernel size of 3 and will apply the convolution on the <code>T</code> dimension creating 3 activations maps.</p>
<pre><code class="lang-python">B, C, T = 3, 4, 10
x = torch.randn(B, C, T)
conv = nn.Conv1d(C, C, 3)
out = conv(x)
print(out.shape)
&gt; torch.Size([3, 4, 8])
</code></pre>
How to create Pytorch dataset with filepath label format

Gordon_Lim: <p>I want to create a Pytorch dataset for RVL-CDIP to feed to a pytorch dataloader. This is the format</p>
<p>imagesq/q/o/c/qoc54c00/80035521.tif 15<br>
where 15 is a label</p>
<p>is there a pytorch datasets function to create said dataset.</p>

ptrblck: <p>I would suggest to write a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>. This would allow you to define the logic to load and process each sample manually. Based on your description I guess the path and label might be stored as a string or <code>tuple</code> etc. so splitting them into a list containing image paths and the corresponding target tensor should work.</p>
Not sure how to save model in h5 format

guen_gn: <p>I am using following as reference. It appears there are two main ways to save the model:<br>
<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a></p>
<ul>
<li>save dictionary state: recommended way.</li>
<li>save entire model: not recommended as path is saved and can break if it is opened somewhere else.</li>
</ul>
<p>TF can conveniently store in h5 format which I can dump quickly using h5 (hdf5) format, but this is not possible with Pytorch?</p>

guen_gn: <p>At least according to this, hdf5 format is not great as it accumulates over time. Is there a better format these days?</p><aside class="quote quote-modified" data-post="2" data-topic="39556">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/imaluengo/48/9847_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/save-torch-tensors-as-hdf5/39556/2">Save torch tensors as hdf5</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    HDF5 is not a great format for appending information over-time… It will end up generating a very large binary file to handle new data. 
I’d recommend doing it for a fixed size. E.g. first create a dataset of a fixed size: 
N = 100 # find the length of my dataset
data = h5_file.create_dataset('data', shape=(N, 3, 224, 224), dtype=np.float32, fillvalue=0)

Then populate it 
for i in range(N):
    img = ... # load image
    data[i] = img

h5_file.close()

However, if you really really want to do re…
  </blockquote>
</aside>


Programmer-RD-AI: <p>Pytorch usually use <code>.pth</code> or <code>.pt</code></p>
<p>those are the formats that are mostly used in pytorch</p>
Check fburl.com/missing_ops for the fix.{prim::is_quantized, }

cuberfan: <p><strong>pytorch</strong>: 1.9.1</p>
<p><strong>android</strong>: 11</p>
<p><strong>android gradle</strong>:</p>
<pre><code>implementation 'org.pytorch:pytorch_android_lite:1.9.0'
implementation 'org.pytorch:pytorch_android_torchvision:1.9.0'
</code></pre>
<p><strong>problem</strong>:</p>
<pre><code>com.facebook.jni.CppException: Following ops cannot be found. Check fburl.com/missing_ops for 
the fix.{prim::is_quantized, } ()
</code></pre>
<p><strong>Test code</strong>:</p>
<pre><code>class AnnotatedConvBnReLUModel(torch.nn.Module):
    def __init__(self):
        super(AnnotatedConvBnReLUModel, self).__init__()
        self.conv = torch.nn.Conv2d(3, 5, 3, bias=False).to(dtype=torch.float)
        self.bn = torch.nn.BatchNorm2d(5).to(dtype=torch.float)
        self.hs = torch.nn.Hardswish()
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv(x)
        x = self.bn(x)
        x = self.hs(x)
        x = self.dequant(x)

        return x
</code></pre>
<p>The android lib  “pytorch:pytorch_android_lite” dose not support torch.nn.Hardswish ?</p>
<p>model = torchvision.models.quantization.mobilenet_v3_large(pretrained=False, quantize=True)</p>
<p>and the mobilenet_v3_large has the same problem, why?</p>

JacobSzwejbka: <p>Are you using a custom build of pytorch?</p>

cccclai: <p>Did you run <code>optimize_for_mobile</code> when exporting the model?</p>

cuberfan: <pre><code>def convert_to_mobile(self):
    logging.info("mode: convert_to_mobile")
    self.net.eval()

    logging.info("step1: quantize model")
    model_quantizer = ModelQuantizer()
    quantized_model = model_quantizer.quantize_model(self.cfg, self.net, self.cur_device, "qnnpack")
    # quantized_model = self.net

    logging.info("step2: script/trace model")
    traced_script_model = torch.jit.script(quantized_model)

    logging.info("step3: model optimization")
    traced_script_model_optimized = optimize_for_mobile(traced_script_model)
    # traced_script_model_optimized = traced_script_model

    logging.info("step4: save mobile model")
    mobile_model_path = self.cfg['common']['convert_model_mobile']
    traced_script_model_optimized._save_for_lite_interpreter(mobile_model_path)</code></pre>

cuberfan: <p>yes, Is there a problem with the custom build of pytorch ?</p>

HDCharles: <p>i’d try it with a hardswish module rather than a functional, it looks like pytorch_android_lite doesn’t like the quantization check in the quantized hardswish op.</p>
<p>alternatively you can just remove the line: <a href="https://github.com/pytorch/pytorch/blob/871a31b9c444e52ba0cc6667fb317c11802ac4de/torch/nn/quantized/functional.py#L460" class="inline-onebox" rel="noopener nofollow ugc">pytorch/functional.py at 871a31b9c444e52ba0cc6667fb317c11802ac4de · pytorch/pytorch · GitHub</a></p>
<p>In your custom version of pytorch and see if that works.</p>

cuberfan: <p>The func  “torch.nn.functional.hardsigmoid” can work instead of torch.nn.HardSigmoid</p>
Problem in C++ parser data in TorchScript for Frontend in network Inference

dingyongchao: <p>I am working for torchscript parser as frontend for our network inference framework. I found the way for parser float torchscript model, as convolution for example;</p>
<pre><code class="lang-auto">Input:const torch::jit::Node* node
const auto&amp; inputs = node-&gt;inputs();
const auto stride = getValue&lt;std::vector&lt;int64_t&gt;&gt;(inputs[3]);
const auto padding = getValue&lt;std::vector&lt;int64_t&gt;&gt;(inputs[4]);

template &lt;typename T&gt;
static inline T getValue(const torch::jit::Value* value) {
    auto optional_ivalue = toIValue(value);
    T res;
    if (!optional_ivalue) {
        MNN_ERROR("getValue: must Constant Node.");
        return res;
    }
    c10::IValue&amp; val = optional_ivalue.value();
    auto optional_res = val.toOptional&lt;T&gt;();
    if (!optional_res) {
        // MNN_ERROR("getValue: value is None.");
        return res;
    }
    return optional_res.value();
}
</code></pre>
<p>However, when I try to parser or unpack qat torchscript model, I found that In pytorch 1.10, the input of convolution number is 4, which include input, __packed_params, scale, zero_point. So I have big problem in unpacking __packed_params in C++.<br>
So if there any easy API such as “toIValue” that I can use for unpack __packed_params, Thanks.</p>
Masking several value inside a tensor with keeping the gradient

thecho7: <p>Hi, I wonder how to change the value inside a tensor with keeping its gradient.</p>
<pre><code>a = torch.Size([5, 1280, 2])
index = torch.Tensor([100, 200, 300, 400, 500])
r = 5

a[0] = [[x1, y1], [x2, y2], ..., [x_95, y_95], [x_96, y_96], ..., [x_105, y_105], ..., [x1279, y1279]]
a[1] = [[x1, y1], [x2, y2], ..., [x_195, y_195], [x_196, y_196], ..., [x_205, y_205], ..., [x1279, y1279]]
a[2] = [[x1, y1], [x2, y2], ..., [x_295, y_295], [x_296, y_296], ..., [x_305, y_305], ..., [x1279, y1279]]
a[3] = [[x1, y1], [x2, y2], ..., [x_395, y_395], [x_396, y_396], ..., [x_405, y_405], ..., [x1279, y1279]]
a[4] = [[x1, y1], [x2, y2], ..., [x_495, y_495], [x_496, y_496], ..., [x_505, y_505], ..., [x1279, y1279]]
</code></pre>
<p>It means that every <strong>index[i]</strong> indicates the index of 5 <strong>a[i]</strong>.<br>
The resulting tensor is expected like below</p>
<pre><code>a[0] = [[0, 0], [0, 0], ..., [x_95, y_95], [x_96, y_96], ..., [x_105, y_105], ..., [0, 0]]
a[1] = [[0, 0], [0, 0], ..., [x_195, y_195], [x_196, y_196], ..., [x_205, y_205], ..., [0, 0]]
a[2] = [[0, 0], [0, 0], ..., [x_295, y_295], [x_296, y_296], ..., [x_305, y_305], ..., [0, 0]]
a[3] = [[0, 0], [0, 0], ..., [x_395, y_395], [x_396, y_396], ..., [x_405, y_405], ..., [0, 0]]
a[4] = [[0, 0], [0, 0], ..., [x_495, y_495], [x_496, y_496], ..., [x_505, y_505], ..., [0, 0]]
</code></pre>
<p>Any help? Please comment on any unclear explanations.</p>
<p>Thanks,</p>
Scikit-image graph module on pytorch

yegane: <p>Hi,<br>
I am trying to use skimage.future.graph package in pytorch. Is there an easy way to move the computations into pytorch and so I could move the computations to GPU?</p>

ptrblck: <p>It depends which methods are called internally in the package. Since <code>skimage</code> would use numpy arrays, you won’t be able to directly push the data to the GPU and use it for the computation. Instead you could check which method are needed and try to rewrite them in PyTorch.</p>
How to efficiently reduce GPU memory for knowledge distillation training

JimW: <p>I am new to Pytorch, and I am working on some knowledge distillation task:<br>
For instance, we have a large teacher network (pre-trained with imagenet) and a small student network.<br>
We need to freeze the teacher network, and use its prediction to guidance the training of the small student network. In other words, teacher network is not trainable, but student network will be updated.</p>
<p>Currently, I simply used “with torch.no_grad()” on teacher network, but the training still consumed too much GPU memory. Because there is no need to backpropagate on the big teacher at all, I was wondering if there are some other ways to handle gradient and activations.</p>
<p>What can I do to further improve the GPU memory utility?</p>

tom: <p>If you use torch.no_grad you are disabling the autograd recording of things for backpropagation, so you would have a hard time reducing memory use further (one idea could do (part of?) the teacher computation in fp16 instead of fp32), but you have to watch out for overflows.</p>
<p>In the student, you could use checkpointing (i.e. keeping less activations between forward and backward and recomputing them) to save memory.</p>
<p>Best regards</p>
<p>Thomas</p>
Loss vs Loss Function vs Criterion

Robin_Lobel: <p>Pure vocabulary question.<br>
How would you define and differentiate those 3 terms : Loss, Loss Function, Criterion</p>
<p>Is the “Loss Function” part of the larger ensemble “Criterion”, and “Loss” the value resulting from the “Loss Function” ?<br>
I see that documentation and tutorials often talks about loss functions, but in practice code snippet often use the term criterion instead, as the variable pointing the loss function. But this word criterion is never discussed when introducing loss functions.<br>
Are there criterions which are not loss function ?<br>
Is “criterion” widely understood or should I always prefer “loss function” when speaking about it ?</p>
<p>Side question: Metrics, Evaluation Metric.<br>
For instance, Accuracy is one of those metrics.<br>
Is loss part of the metrics, but not the evaluation metrics, as opposed to accuracy ?</p>

tom: <p>As you note, this is not completely distinct. “criterion” is typically a callable (function or nn.Module instance) that computes the loss (value), “loss function” makes this explicit in the name. “loss” is - in my experience - more often used for the loss value but sometimes (for example in naming the loss Modules) it is also used for the loss function or even the class whose instance computes the loss function.</p>
<p>I think criterion is relatively widely understood, I have not seen ones that were not loss functions.</p>
<p>Loss typically is differentiable, while metrics don’t need to be. I would consider metrics to comprise both losses and non-differentiable metrics (such as accuracy). It is completely OK to use validation loss as a evaluation metric, but you cannot differentiate accuracy and so you cannot do gradient descent on it.</p>
<p>Best regards</p>
<p>Thomas</p>
Compute interpolation of two tensors

papyrus: <p>Let’s say I have:</p>
<ul>
<li>two tensors, <code>T0</code> with size <code>B x a x b</code> and <code>T1</code> with size <code>B x a x b</code>
</li>
<li>a list <code>interps</code> of interpolations: <code>[0.25, 0.5, 0.75]</code>
</li>
</ul>
<p>and I want to fill a new tensor <code>T3</code> with size <code>B x (len(interps) + 2) x a x b</code> so that:</p>
<pre><code class="lang-auto">T3[:, 0, ...] = T0
T3[:, 1, ...] = T0 * (1-interps[0]) + T1 * (interps[0])
T3[:, 2, ...] = T0 * (1-interps[1]) + T1 * (interps[1])
T3[:, 3, ...] = T0 * (1-interps[2]) + T1 * (interps[2])
T3[:, 4, ...] = T1
</code></pre>
<p>How could I do this effectively, without a for loop? I managed to do something alike with <code>torch.kron</code> and simpler tensors, but I struggle doing it with tensors that have those shapes.</p>
<p>Thanks for the help</p>

tom: <p>This is just what broadcasting is good for (<code>None</code> below inserts a singleton dimension for broadcasting, <code>:</code> uses all of a given dimension. One could use unsqueeze or view, but I like this to align things.)</p>
<pre><code class="lang-python">T0, T1 = torch.randn(2, 3, 4, 4)

interps = torch.tensor([0.0, 0.25, 0.5, 0.75, 1.0])

T3 = T0[:, None, :, :] * (1 - interps[None, :, None, None]) + T1[:, None, :, :] * interps[None, :, None, None]
</code></pre>
<p>or equivalently</p>
<pre><code class="lang-python">T3 = torch.lerp(T0[:, None, :, :], T1[:, None, :, :], interps[None, :, None, None])
</code></pre>
<p>The latter can be a bit faster (useful if your computation is bottlenecked on these operations), but <code>torch.lerp</code> is a tad exotic and so chances are that people trying to understand the code will have to look up what it does.</p>
<p>Best regards</p>
<p>Thomas</p>

papyrus: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="135352">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p><code>interps[None, :, None, None]</code></p>
</blockquote>
</aside>
<p>Nice! I tried to do something similar but couldn’t make it work.</p>
<p>Thanks a lot!</p>

tom: <p>One thing to keep in mind - and I did this wrong at first, too - is that if the tensors have a differing number of dimensions, the smaller ones get padded in front. In particular, it is crucial that you give interp the last two dimensions while you could (works, but is less clear IMHO) leave out the front one.</p>
PyTorch equivalent of Keras 'Add' type layer

YadneshD: <p>My keras summary model shows that it has a add layer to sum few layers. Screenshot in drive link below -</p><aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/1Iulv86KcQSO1iu2PirD1pl3AAoMTghbd/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/1Iulv86KcQSO1iu2PirD1pl3AAoMTghbd/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1Iulv86KcQSO1iu2PirD1pl3AAoMTghbd/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1Iulv86KcQSO1iu2PirD1pl3AAoMTghbd/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">Screenshot from 2021-10-28 17-25-56.png</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I am a beginner and I don’t know how to implement the add layer in PyTorch. Actually I need to sum few layers convolution layers with previous layers. Please see the attached screenshot. <img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef8fff2d112e6e21cb21d741cac0f8a02027c0cc.png" alt="Screenshot from 2021-10-28 17-21-56" data-base62-sha1="ybgCSDWfhC9oJrBVUrzWckhL5zC" width="597" height="449"></p>
<p>I am converting a model from keras to PyTorch. All other conv2d, batch_normalization2d layers are similar in pytorch but I am not getting the equivalent of ‘Add’ type layer of Keras. Please help!</p>

ptrblck: <p>You can directly add PyTorch tensors in your forward without the need to use a layer:</p>
<pre><code class="lang-python">def forward(self, x):
    out1 = self.layer1(x)
    out2 = self.layer1(x)
    out3 = self.layer1(x)
    out = out1 + out2 + out3
    ...
</code></pre>
<p>Alternatively, you could also create a custom <code>nn.Module</code>, pass the inputs to this module, and add them in the <code>forward</code> (although it sounds quite cumbersome given that a <code>+</code> will also do it <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"> ).</p>