akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to annotate inside polygons(holes) in semantic segmentation

username: Shashank_Shekhar
cooked: <p>In semantic segmentation how do annotators annotate holes(inside polygons)?<br>
As seen in the below image for swimmer and race cars, their are multiple polygons one representing the object and other inside polygons(holes) which representing the background. How do we annotate it and while training will this be ignore label?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/487c7e3be814e4e8beeb7106808ea9e110b31a5b.png" data-download-href="https://discuss.pytorch.org/uploads/default/487c7e3be814e4e8beeb7106808ea9e110b31a5b" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/487c7e3be814e4e8beeb7106808ea9e110b31a5b_2_690x338.png" alt="Capture" data-base62-sha1="alf6omzjIk6F7EhDRnzkrcoM0Wf" width="690" height="338" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/487c7e3be814e4e8beeb7106808ea9e110b31a5b_2_690x338.png, https://discuss.pytorch.org/uploads/default/original/3X/4/8/487c7e3be814e4e8beeb7106808ea9e110b31a5b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/8/487c7e3be814e4e8beeb7106808ea9e110b31a5b.png 2x" data-dominant-color="6C707B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">995×488 665 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Upsample 2x2->5x5

username: 111186
cooked: <p>my input size is 128x5x5， after maxpooling the size is 128x2x2. i want use the upsample to make the 128x2x2 to 128x5x5. how to change the code</p>
<pre><code class="lang-auto">x2 = F.interpolate(x1, size=[5,5])
</code></pre>

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>Try the following code snippet.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

x = torch.randn((128, 2, 2))
x = x.unsqueeze(dim=0)
ups = nn.Upsample(size=[5, 5])
out = ups(x).squeeze(0)
print(out.shape)
</code></pre>
<p>There are various modes of upsampling. Please check the doc here - <a href="https://pytorch.org/docs/master/generated/torch.nn.Upsample.html" rel="nofollow noopener">https://pytorch.org/docs/master/generated/torch.nn.Upsample.html</a></p>
<p>Thanks<br>
Regards<br>
Pranavan</p>

username: 111186
cooked: <p>i will try. thank you so much</p>
Distributed error this morning

username: pytorchnewbie
cooked: <p>Hi, I’m not sure why but today when I run the same code that I ran last night I get this error:</p>
<pre><code class="lang-auto">   from ignite.utils import setup_logger
  File "/home/mia/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/__init__.py", line 2, in &lt;module&gt;
    import ignite.distributed
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/__init__.py", line 1, in &lt;module&gt;
    from ignite.distributed.auto import *
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/auto.py", line 10, in &lt;module&gt;
    from ignite.distributed import utils as idist
  File "/home/useranaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/utils.py", line 8, in &lt;module&gt;
    from ignite.distributed.comp_models import (
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/comp_models/__init__.py", line 2, in &lt;module&gt;
    from ignite.distributed.comp_models.native import has_native_dist_support
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/comp_models/native.py", line 22, in &lt;module&gt;
    class _NativeDistModel(ComputationModel):
  File "/home/user/anaconda3/envs/idk/lib/python3.7/site-packages/ignite/distributed/comp_models/native.py", line 305, in _NativeDistModel
    "AND": dist.ReduceOp.BAND,
AttributeError: type object 'torch.distributed.ReduceOp' has no attribute 'BAND'
</code></pre>
<p>Any idea?</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/pytorchnewbie">@pytorchnewbie</a> what is your pytorch version ?<br>
It should be &gt;= 1.3</p>
RanodmSearchCV in pytorch

username: 887574002
cooked: <p>Hi, I have a question regarding hyperparameter tunning in PyTorch. I have a neural net in PyTorch that I want to tune its hyperparameters using RandomsearchCv in sklearn. But, I didn’t find any wrapper for pytorch model that makes it possible to use RandomSearchCv in sklearn. Any help is appreacited.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p><a href="https://github.com/skorch-dev/skorch">Skorch</a> provides a scikit-learn compatible layer on top of PyTorch.</p>
Modify Layer Name

username: Abhinav_Rai
cooked: <p>I am loading ResNet50 using torchvision. However, I want to modify the layer names, specifically add a prefix before each name. Ex. layer1.0.conv1 -&gt; prefix_layer1.0.conv1</p>
<p>Is it possible to do so for models imported from torchvision ?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You could create a custom model and create the new child modules using the new attribute names and by assigning the pretrained <code>torchvision</code> modules.<br>
This would also need changes in the <code>forward</code> method, since now all layers have different names.<br>
Also, using this method you won’t be able to load the pretrained <code>state_dict</code> from <code>torchvision</code>'s ResNet.</p>
<p>So, while it would be possible, I can’t think of a clean solution without some disadvantages.</p>
<p>What’s your use case that you need to change the attribute names?</p>
Result of convolution changes every time the cell is compiled

username: albusdemens
cooked: <p>I wrote a simple script which applies a 2D convolution operation to an input image. I am running the code as a cell in a Jupyter Notebook. Every time I compile the cell, the colors of the processed image change. Why is that? Should I change how convolution is calculated?</p>
<p>Here is my code:</p>
<blockquote>
<p>import torch<br>
import torchvision<br>
import torchvision.transforms as transforms<br>
import numpy as np<br>
import matplotlib.pyplot as plt<br>
import imageio<br>
from PIL import Image</p>
<p>A = imageio.imread(“Gioconda.jpg”)</p>
<p>del transposed_image, fc, fc1, result, image_d</p>
<p>!!! from [H, W, C] to [C, H, W]<br>
transposed_image = A.transpose((2, 0, 1))<br>
!!! add batch dim<br>
transposed_image = np.expand_dims(transposed_image, 0)</p>
<p>conv2 = torch.nn.Conv2d(in_channels=3, out_channels=3, kernel_size=10, stride=10, padding = 10)</p>
<p>image_d = torch.FloatTensor(transposed_image)<br>
fc = conv2(image_d)<br>
fc1 = fc.permute(0, 2, 3, 1)[0]<br>
result = fc1.data.numpy()<br>
max_ = np.max(result)<br>
min_ = np.min(result)<br>
result -= min_<br>
result /= max_</p>
<p>plt.figure(figsize=(16,8))<br>
plt.subplot(1,2,1)<br>
plt.imshow(A)<br>
plt.subplot(1,2,2)<br>
plt.imshow(result)<br>
plt.show()</p>
</blockquote>

username: ptrblck
cooked: <p>I assume you mean the colors of <code>result</code> change?<br>
If that’s the case, then it would be expected, since you are applying a random <code>nn.Conv2d</code> layer to the image.</p>

username: albusdemens
cooked: <p>Thanks for the hint! In your opinion, what’s the more pytorchy way to apply a convolution layer to all the channels of the image?</p>

username: ptrblck
cooked: <p>The “standard” conv layer will already use kernels, which use all input channels to create an activation map.<br>
If you would like to apply a single channel filter to each input channel, you could use a grouped convolution via <code>groups=in_channels</code>.</p>
How to improve model performance with limited samples?

username: gamebo5000
cooked: <p>I am using a pre-trained MobileNetV2 model on a custom dataset. I configured the outputs from 1000 to 3 since that is the number of classes I am working with. Class A is 8000 images Class B is 5000 images and class C is 500 images. I took 100 of each out and use those as the validation set and the rest as the training set.  I ran the experiment for 90 epochs with the learning rate being adjusted by 0.1 every 30 epochs and  these hyperparameters:</p>
<p>learning rate = 0.001<br>
weight decay = 4e-5<br>
momentum = 0.9<br>
batch size  = 64<br>
optimizer  = SGD<br>
dropout = 0.2</p>
<p>after completing my experiment I am noticing that class A and B have accuracies between 80% and 97% in the last few runs and Class C is in the 70 percentile range.</p>
<p>I understand this because of how small the data sample is for class C compared to A and B, and I am looking for ways to run around that since even if I add more data it still would be less than half the size of the other two classes. This is my data augmentation technique:</p>
<blockquote>
<p>train_transform = transforms.Compose([<br>
transforms.RandomResizedCrop(224,scale = (0.2,1.0)),<br>
transforms.RandomHorizontalFlip(),  # reverse 50% of images<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406],<br>
[0.229, 0.224, 0.225])<br>
])</p>
<pre><code>test_transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],
                             [0.229, 0.224, 0.225])
    ])
</code></pre>
</blockquote>
<p>Cross-validation is the only technique I found while searching this problem that I am not using and using cross-validation in Pytroch is proving rather tricky for me. One technique I read about in a paper is where the author claimed the augmented their class data to be 6,000 images each, but they didn’t say how they did this.  Are there anyways I can increase the detection accuracy of class C with limited samples available and by association increase the overall accuracy of my model from 84% to 95+%?</p>
<p><strong>Update</strong><br>
I came across the concept of weight loss in cross-entropy loss. the formula I found to determine each class weight loss is 1/class_size.  So in my case class, A weight would be 1/8000 = 1.25*10^(-4), if this is not the correct way determine it please let me know.</p>
Replacing for loop for max operation

username: legoh
cooked: <p>Hi, I need a bit of help replacing a for-loop. Here’s a code snippet</p>
<pre><code class="lang-auto">        for _ in range(n_iter):
            # Message passing
            for i in range(n):
                m_beta_alpha_k[:, :, i] = g[:, i, :] - torch.max(
                    torch.cat((m_alpha_beta[:, :i, :], m_alpha_beta[:, (i + 1):, :]), dim=1), dim=1)[0]
                m_alpha_beta_k[:, :, i] = g[:, :, i] - torch.max(
                    torch.cat((m_beta_alpha[:, :i, :], m_beta_alpha[:, (i + 1):, :]), dim=1), dim=1)[0]

            m_alpha_beta = m_alpha_beta_k
            m_beta_alpha = m_beta_alpha_k
</code></pre>
<p>Each tensor is a 3D tensor, where we have [batch, i, j], where [i, j] forms a matrix and we have a batch size of them to operate on. As shown in the loop, at each iteration, I wish to exclude a row of the [i, j] matrix, and take a <code>max</code> operation in each column for the remaining matrix. Is there a way to do this efficiently in torch?</p>
Batch dimension for calculating loss

username: Marla
cooked: <p>I’m attempting to train an RNN, and am unsure how to handle batches when calculating the loss.</p>
<p>A criterion such as nn.MSELoss() requires the prediction and target to be of the same shape.</p>
<p>For an RNN, ‘output’ is given as (seq_len, batch, num_directions * hidden_size) with ‘batch’ no longer represented (as per the documentation).</p>
<p>I’ve shaped my target data/ labels to be the same as ‘input’: (seq_len, batch, input_size), so I have a target value for each batch and feature.</p>
<p>Thanks in advance!</p>

username: vdw
cooked: <aside class="quote no-group" data-username="Marla" data-post="1" data-topic="90858">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/51bf81/48.png" class="avatar"> Marla:</div>
<blockquote>
<p>For an RNN, ‘output’ is given as (seq_len, batch, num_directions * hidden_size) with ‘batch’ no longer represented (as per the documentation).</p>
</blockquote>
</aside>
<p>What do you mean the <code>batch</code> is no longer represented? It’s in the second dimension :). If you need it in the first one you could simply do <code>output.transpose(0,1)</code></p>
Can I use this CNN to train data rather than text?

username: ChineseBest
cooked: <pre><code class="lang-auto">class CNNnet(torch.nn.Module):
    def __init__(self):
        super(CNNnet,self).__init__()
        self.conv1 = torch.nn.Sequential(
            torch.nn.Conv1d(in_channels=32,
                            out_channels=2,
                            kernel_size=1,
                            stride=2,
                            padding=1),
            torch.nn.Sigmoid(),
            torch.nn.MaxPool1d(kernel_size=1)
        )

        self.conv2 = torch.nn.Sequential(
            torch.nn.Conv1d(in_channels=2,
                            out_channels=2,
                            kernel_size=1,
                            stride=2,
                            padding=1),
            torch.nn.Sigmoid(),
            torch.nn.MaxPool1d(kernel_size=1)
        )

        self.conv3 = torch.nn.Sequential(
            torch.nn.Conv1d(in_channels=2,
                            out_channels=2,
                            kernel_size=1,
                            stride=2,
                            padding=1),
            torch.nn.Sigmoid(),
            torch.nn.MaxPool1d(kernel_size=1)
        )
        
        self.mlp1 = torch.nn.Sequential(torch.nn.Linear(4,10),torch.nn.Sigmoid(),torch.nn.Linear(10,4))
       
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.mlp1(x.view(x.size(0),-1))
        return x
</code></pre>
<p>I mean, such as prediction work?</p>

username: ptrblck
cooked: <p>Your model might be able to train, if the data provides enough information.<br>
It’s a bit hard to provide a general statement, as it depends on the use case you are working with.<br>
However, I would assume the more “temporal” structure your data has, the better.</p>

username: ChineseBest
cooked: <p>Thanks a lot for your reply</p>
UserWarning: The given NumPy array is not writeable?

username: spadel
cooked: <p>I get the following notification when I load my dataset into the DataLoader:</p>
<pre><code class="lang-auto">/pytorch/torch/csrc/utils/tensor_numpy.cpp:141: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.
</code></pre>
<p>I get this error when I want to convert my labels from a numpy array to a tensor. I tried changing the flags but it doesn’t work. Also I tried to copy the array as proposed, but I still get the error. I also can’t change the writeable flag at all. I tried the following in code:</p>
<pre><code class="lang-auto">        print(y.flags)
        y.setflags(write=1)
        print(y.flags)
        y = y.copy()
        print(y.flags)
        y.setflags(write=1)
        print(y.flags)
        y = torch.as_tensor(y, dtype=torch.long)
</code></pre>
<p>I check the flags three times after each operation and it always returns:</p>
<pre><code class="lang-auto">  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : False
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False
</code></pre>
<p>Can someone help?</p>

username: swalton
cooked: <p>Have you tried just using <code>from_numpy</code>? I faced this a similar issue and tried a bunch of copies but the simple solution worked the best.</p>
Variation of results

username: Nilakshan_Kunanantha
cooked: <p>I have quantized version of resnet18 network. When do the inference, I fed a single image from sequentially sampled data loader.<br>
I’ve tried to printed the image tensor and input to the first convolution layer,got some mismatches.</p>
<p>To print input image fed from data loader, I used:</p>
<pre><code class="lang-auto">img = image.detach().numpy()
img = np.transpose(img,(2,3,1,0))
</code></pre>
<p>To get the input to first conv layer:</p>
<pre><code class="lang-auto">layer_input={}
def get_input(name):
	def hook(model, input, output):
		layer_input[name] = input

	return hook

model.conv1.register_forward_hook(get_input('conv1'))

qdm = torch.nn.quantized.DeQuantize()
deqout = qdm( val )
deqout = deqout.numpy()
deqout = np.transpose( deqout, (2, 3, 1, 0) )
</code></pre>
<p>Image data:</p>
<blockquote>
<p>tensor([[[[-0.5082, -0.3883, -0.4226,  …,  0.9303,  0.3823,  0.6392],<br>
[-0.6281, -0.6965, -0.4397,  …,  0.8104,  0.5878,  0.2111],<br>
[-0.5767, -0.1486,  0.0741,  …,  0.7419,  0.8961,  0.2282],</p>
</blockquote>
<p>Input to conv layer:</p>
<blockquote>
<p>-0.52449334,-0.5619572,-0.7492762,-0.3746381,-0.41210192,-0.5619572,-0.41210192,-0.03746381,0.07492762,0.0,-0.26224667,-0.59942096,-0.18731906,-0.41210192,-0.7118124 ,-0.7118124</p>
</blockquote>

username: dskhudia
cooked: <p>These should be close to each other. Just to confirm, your “img” input is after the normalize transformation mentioned here: <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</a></p>
<pre><code class="lang-auto">        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225]),
</code></pre>
Building pytorch from source issue

username: sathvik_udupa
cooked: <p>After I updated pytorch, it doesn’t support my GPU anymore(3.5 compute compatibility) so I uninstalled and have been trying to install from source from <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">this</a>.</p>
<blockquote>
<p>python setup.py install</p>
</blockquote>
<p>Using this code, my system freezes when the build starts.<br>
Do I need to adjust build options? I do not know what to change.</p>
<p>Will I be facing any issue if I install previous version of pytorch which was compatible for my gpu?</p>
<p>Any help appreciated. Thanks</p>
<p>system information:<br>
OS: Ubuntu 18.04<br>
RAM: 8GB<br>
CPU: Intel(R) Core™ i5-5200U CPU @ 2.20GHz<br>
GPU: GeForce 920M 2GB<br>
NVIDIA driver: 440.33.01<br>
cuda: 10.2</p>

username: albanD
cooked: <p>Hi,</p>
<p>What do you mean by the system freezer? Do you run out of ram?</p>

username: sathvik_udupa
cooked: <p>Yes, cpu and ram usage goes ~100%, no other processes running. No response after 15+ mins, had to hard reboot everytime.</p>

username: albanD
cooked: <p>You’re running out of RAM during the compilation <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
You might want to reduce the number of threads that it can use to reduce the memory usage by setting <code>MAX_JOBS=2</code> as an env variable before calling the setup.py script. Note that this will slow down the compilation quite a lot.</p>

username: sathvik_udupa
cooked: <p>Okay thanks, I’ll try it.<br>
As an alternative, if I install the pytorch 1.2 wheel file from the older versions page, will I be facing any performance issues on model building? I’ve tried finding information on it online but not finding anything relevant.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Older versions might not have all the most recent features, bugfix and perf improvements. So it will depend a lot on what you’re using.</p>
Torch and pytorch both installed?

username: fjsaezm
cooked: <p>Hello!<br>
I created a conda environment using a provided .yml file <a href="https://github.com/loeweX/Greedy_InfoMax" rel="noopener nofollow ugc">on this repo</a>.<br>
However, when installing torchaudio , torch 1.5.1 was automatically installed and I needed pytorch 1.0.0 (required for the code). I ended up having</p>
<ul>
<li>
<p>torch 1.5.1</p>
</li>
<li>
<p>pytorch 1.0.0<br>
on the same environment. Is his a problem? I think it is, because now I’m getting the following error when I execute the audio training:</p>
<blockquote>
<p>File “/home/fjaviersaezm/anaconda3/envs/infomax/lib/python3.6/site-packages/torchaudio/functional.py”, line 40, in <br>
<span class="mention">@torch.jit.ignore</span></p>
</blockquote>
</li>
</ul>
<blockquote>
<p>AttributeError: module ‘torch.jit’ has no attribute ‘ignore’</p>
</blockquote>
<p>Can I uninstall torch 1.5.1? How can I solve the “ignore” problem?<br>
Thanks guys!</p>

username: vincentqb
cooked: <p>torchaudio requires more recent version of torch, see <a href="https://github.com/pytorch/audio/blob/master/README.md#dependencies" rel="nofollow noopener">dependencies</a>. It appears the environment is looking at pytorch 1.0 and functionalities required by torchaudio are missing.</p>
Pytorch model not training

username: ayush-1506
cooked: <p>I’m trying to train a basic conv net using pytorch. This is what my training loop looks like:</p>
<pre><code class="lang-auto">for epoch in range(20):  # loop over the dataset multiple times
    model.train()
    print("================ ", epoch)

    running_loss = 0.0
    for i, data in enumerate(trainloader):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 20 == 0:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0
</code></pre>
<p>Now for some reason, I always get the same loss:</p>
<pre><code class="lang-auto">1,     1] loss: 0.000
[1,    21] loss: 0.028
[1,    41] loss: 0.036
[1,    61] loss: 0.050
[1,    81] loss: 0.028
[1,   101] loss: 0.042
[1,   121] loss: 0.034
[1,   141] loss: 0.030
[1,   161] loss: 0.030
[1,   181] loss: 0.022
[1,   201] loss: 0.036
================  1
[2,     1] loss: 0.002
[2,    21] loss: 0.028
[2,    41] loss: 0.036
[2,    61] loss: 0.050
[2,    81] loss: 0.028
[2,   101] loss: 0.042
[2,   121] loss: 0.034
[2,   141] loss: 0.030
[2,   161] loss: 0.030
[2,   181] loss: 0.022
[2,   201] loss: 0.036
</code></pre>
<p>Why is my model not training? Am I missing something?</p>

username: ayush-1506
cooked: <p>Training loop taken from <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></p>

username: ayush-1506
cooked: <p>Spent 8 hours carefully debugging the train loop to see where things might be going wrong only to realize later that my architecture was wrong. duhh</p>
To get a middle activation map

username: chetan06
cooked: <p>I was trying to visualize a layer of pretrained model but when I am trying to get the middle layer of model like alexnet then I am getting 2d tensor with one column but I am expecting a feature map with 3 dimensions.</p>
<p>To get this output I am replacing all the layer that follows the desired layer with following layer</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    def __init__(self):
         super()
   def forward(self, x):
        return x
</code></pre>

username: ptrblck
cooked: <p>An easier way to get the intermediate activations would be forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
<p>I think you are seeing a flattened tensor, since you are only replacing the modules, while the functional API call in the <code>forward</code> (such as <code>x = x.view(x.size(0), -1)</code>) are still used.</p>

username: chetan06
cooked: <p>Thanks for the answer</p>
Computing every iteration, every 20 epochs

username: pytorchnewbie
cooked: <p>Hi!</p>
<p>What if I wanted to compute something on every batch output (so every iteration), but I only wanted to do this every 20 epochs? How could I do this? Is there a sort of “detach_event_handler” as a counterpart to attach_event_handler? Or is there some other way to do this? Thanks!</p>

username: RaLo4
cooked: <p>I might have misunderstood your question but could you not just use:</p>
<pre><code class="lang-auto">if(epoch%20 == 0):
</code></pre>

username: pytorchnewbie
cooked: <p>I think I didn’t explain well my question.</p>
<p>Consider the following:</p>
<p>trainer.add_event_handler(Events.ITERATION_COMPLETED(every=1), foo)</p>
<p>I want to run foo() on every iteration, but only every 20 epochs.</p>
<p>I cant just do:</p>
<p>if (epoch%20==0):</p>
<p>trainer.add_event_handler(Events.ITERATION_COMPLETED(every=1), foo)</p>
<p>Because the first time epoch gets to 20, the handler will be attached to the trainer and then foo() will be called on every single iteration from then on, whether epoch%20 or not.</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/pytorchnewbie">@pytorchnewbie</a> I think the point was about using <code>if epoch %20 ==0</code> inside <code>foo</code> which is one of possible implementations.</p>
<p>If you would like to setup customized event filtering other than <code>every</code>, please see the usage of <code>event_filter </code> argument : <a href="https://pytorch.org/ignite/engine.html#ignite.engine.events.Events" rel="nofollow noopener">https://pytorch.org/ignite/engine.html#ignite.engine.events.Events</a></p>

username: RaLo4
cooked: <p>either this.<br>
Or, as far as I understand you want to call foo() on every iteration of every 20th epoch right?<br>
So this:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):    #your epoch loop
    for iteration, data in enumerate(your_dataloader, 0):     #your iteration loop
        if((epoch+1)%20 == 0): 
            foo() #function gets called on every iteration of every 20th epoch
</code></pre>
<p>should work.<br>
<a class="mention" href="/u/pytorchnewbie">@pytorchnewbie</a> Is this not the behavior you asked for?</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/ralo4">@RaLo4</a> The question is marked with “ignite” category and about how to do that with PyTorch-Ignite</p>
Custom operation with quantization

username: zzoon91
cooked: <p>Hi,</p>
<p>I want to run MLP model on my accelerator.<br>
My plan is to add a custom jit operation that operates like the nn.Linear layer but faster by using my accelerator.<br>
However, the accelerator only supports int8 operation, so I need some quantization as well.</p>
<p>How can I add the custom operation that utilizes torch.quantization?</p>
<p>Thanks in advance!</p>
How to get the version numbers of a Module's Parameters?

username: Florian_Dietz
cooked: <p>How do I find out what the version numbers of a Module’s Parameters are?</p>
<p>I know that pytorch keeps track of this information, because it will raise an exception if you try to backpropagate through a graph that was based on Parameters that have been updated since.</p>
<p>I need this because I have a pretty complex and unusual computational graph that is dynamically generated, and I would like to perform sanity checks on the version numbers.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can access these with the read-only <code>._version</code> attribute on Tensors.<br>
Note though that this is an internal implementation. And the absolute values might change (though equal or not equal will of course still be true).</p>

username: Florian_Dietz
cooked: <p>Thanks!<br>
Two follow-up questions:</p>
<ul>
<li>I want to store the version numbers of all Module parameters at certain points, and then later check if any of these checkpoints used Parameters that are now outdated. Will this work reliably?</li>
<li>Can I assume that this gets serialized correctly when using <code>state_dict()</code>, and will continue to work after reloading? Or can this cause the version numbers to jump and maybe even become smaller?</li>
</ul>

username: albanD
cooked: <p>Hey,</p>
<p>Yes checking if they changed or not will always work reliably.<br>
Since the deserialized Tensor doesn’t share data with the original one anymore. There will be no link between their version counters. So I would expect that it gets reset to 0 on loading.</p>
torchtext.data.Field didn't padding

username: Lymo
cooked: <p>Hi, I’m using Bert to do <strong>relation classification</strong>.</p>
<p>Here is my setting for <code>torchtext.Field</code></p>
<pre><code class="lang-python"># Initialize tokenizer
pretrain_model = "bert-base-uncased"
additional_special_tokens = ['[E1]', '[/E1]', '[E2]', '[/E2]']
tokenizer = BertTokenizer.from_pretrained(pretrain_model, do_lower_case=True, additional_special_tokens = additional_special_tokens)

# Model parameters
MAX_SEQ_LEN = 512
PAD_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)
UNK_INDEX = tokenizer.convert_tokens_to_ids(tokenizer.unk_token)

batch_size = 32
# Fields

label_field = Field(sequential=False, use_vocab=False, batch_first=True, dtype=torch.float)
text_field = Field(use_vocab=False, tokenize=tokenizer.encode, lower=False, include_lengths=False, batch_first=True,
                   fix_length=MAX_SEQ_LEN, pad_token=PAD_INDEX, unk_token=UNK_INDEX)
fields = [('label', label_field),  ('text', text_field)]
</code></pre>
<p>And I try to verify if it works correctly, (from <a href="https://torchtext.readthedocs.io/en/latest/data.html#torchtext.data.RawField.preprocess" rel="nofollow noopener">torchtext docs</a>):</p>
<pre><code class="lang-python">temp_preprocess = text_field.preprocess(train_data.iloc[0:10, -1])
</code></pre>
<p>Unfortunately, it didn’t work. It looks the same as what it used to be.<br>
No padding, no tokenization:</p>
<pre><code class="lang-auto">[CLS] sen. charles e. schumer called on federal safety officials yesterday to reopen their investigation into the fatal crash of a passenger jet in [E2] belle_harbor [/E2] , [E1] queens [/E1] , because equipment failure , not pilot error , might have been the cause . [SEP]
[CLS] but instead there was a funeral , at st. francis de sales roman catholic church , in [E2] belle_harbor [/E2] , [E1] queens [/E1] , the parish of his birth . [SEP]
[CLS] rosemary antonelle , the daughter of teresa l. antonelle and patrick antonelle of [E2] belle_harbor [/E2] , [E1] queens [/E1] , was married yesterday afternoon to lt. thomas joseph quast , a son of peggy b. quast and vice adm. philip m. quast of carmel , calif. . [SEP]
</code></pre>
<p>I also tried using <code>TabularDataset</code> to read data from files:</p>
<pre><code class="lang-python">train_td = TabularDataset(path='./origin_data/train_filtered_bf.txt', format='tsv', fields=fields)
test_td = TabularDataset(path='./origin_data/test_filtered_bf.txt', format='tsv', fields=fields)
print(train_td[1].text)
</code></pre>
<p>And the result didn’t have any padding token as well:</p>
<pre><code class="lang-auto">[101, 101, 2021, 2612, 2045, 2001, 1037, 6715, 1010, 2012, 2358, 1012, 4557, 2139, 4341, 3142, 3234, 2277, 1010, 1999, 30525, 9852, 1035, 6496, 30522, 1010, 30524, 8603, 30523, 1010, 1996, 3583, 1997, 2010, 4182, 1012, 102, 102]
</code></pre>
<p>I’m wondering did I missed any things? Please help me with this. Thanks in advance.</p>
PyTorch is not linked with support for cpu devices

username: thoval
cooked: <p>I’m desperately trying to load a .pt model file in an iOS application. The Sample applications work as expected. When using pod install the most recent version of LibTorch is currently 1.3.1. This version didn’t manage to load the model. When searching for the error message I found suggestions to use a nightly build of 1.4.0. I did find a prebuilt nightly build and also tried building it from source myself. Both ways led to the same error. When calling “_impl = torch::jit::load(filePath.UTF8String);” it raises an exception with the message: “PyTorch is not linked with support for cpu devices (getDeviceGuardImpl at /Users/distiller/project/c10/core/impl/DeviceGuardImplInterface.h:216)”.</p>
<p>Has anybody experienced the same issue or have an idea how to solve it?<br>
Thanks in advance!</p>

username: Lambda
cooked: <p>UP, I’m currently having the exact same issue, Using pod install for LibTorch 1.4.0 on an Xcode IOS simulator project, I cannot manage to load a .pt model file leading to the “Pytorch is not linked with support for cpu devices” Error</p>
<p>Looking for an idea to solve this issue.</p>

username: developer
cooked: <p>UP, I’m currently having the exact same issue, Using pod install for LibTorch 1.5.0 on an Xcode IOS Device project, I cannot manage to load a .pt model file leading to the “Pytorch is not linked with support for cpu devices” Error</p>
<p>Looking for an idea to solve this issue.</p>
Feature Request Torchvision Object detection Models TPU Support

username: oke-aditya
cooked: <h2>Feature Proposal: -</h2>
<p>Torchvision Object detection models should support TPU.</p>
<p>TPUs will greatly lower training time for models like Faster RCNN and coming soon Retina Net. Torch natively supports TPU through xla library. It is possible to train the classification models on TPU as well. But when we try object detection models on TPU it gives the following error</p>
<pre><code class="lang-auto"> File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/generalized_rcnn.py", line 70, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/rpn.py", line 493, in forward
    boxes, scores = self.filter_proposals(proposals, objectness, images.image_sizes, num_anchors_per_level)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/rpn.py", line 416, in filter_proposals
    keep = box_ops.batched_nms(boxes, scores, lvl, self.nms_thresh)
RuntimeError: Cannot access data pointer of Tensor that doesn't have storage
</code></pre>
<p>This error is because some features such <code>rpn.py</code> , <code>ops.batched_nms</code> is not TPU compatible.<br>
Will we get any support for TPU soon for these models ?<br>
I created an <a href="https://github.com/pytorch/vision/issues/2486" rel="nofollow noopener">issue</a> on GitHub as well but I got no response. So I thought maybe I will add here to discuss.</p>
Multi image input model with dataloader

username: moonchildz
cooked: <p>Hi everyone.</p>
<p>I want to make a model that uses 4 image input and 1 image output.</p>
<p>Since I’m very new with pytorch, I only know that I should use Dataloader,</p>
<p>and failed to make my own source code</p>
<p>can anybody give me some hints and refereneces ?</p>

username: ekilic
cooked: <p>Hi Dear,</p>
<p>You should code your own custom dataset and model.</p>
<p>Custom dataset will be like this.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, image_paths):
        self.image_paths = image_paths

    def __getitem__(self, index)
        #read input images and output image
        
        return iImg1, iImg2, iImg3, iImg4, oImg

    def __len__(self):
        return len(self.image_paths)
</code></pre>
<p>Your custom model will be like this.</p>
<pre><code class="lang-auto">class CustomModel(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        """
        In the constructor we instantiate two nn.Linear modules and assign them as
        member variables.
        """
        #your model arch

    def forward(self, iImg1, iImg2, iImg3, iImg4):
        # 4 image input for model
        #operations regarding model
 
        return oImg #output a image

</code></pre>
<p>You must check in docs writing custom <a href="https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_module.html" rel="nofollow noopener">model</a> and <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">dataset</a>.</p>

username: moonchildz
cooked: <p>You are my best. Thank you so much.</p>
Optimization with unitary constraints

username: sakuraiiiii
cooked: <p>Hi. I want to take a constrained optimization. Specifically, the problem is to minimize a function f(U1, U2, …), with U_i is a unitary matrix.<br>
For example,</p>
<pre><code class="lang-python">import torch
from torch import nn
import numpy as np

Ui = []
for i in range(4):
    H = np.random.rand(4, 4)
    np.add(H.T.conjugate(), H, H)
    np.multiply(.5, H, H)
    
    val, vec = np.linalg.eig(H)
    unitary = np.dot(vec * np.exp(1.j * val), vec.T.conjugate())
    
    Ui.append(np.stack([unitary.real, unitary.imag], axis=0))
    
guess_U = torch.from_numpy(np.stack(Ui, axis=1)).float().unsqueeze(0)
guess_U.requires_grad = True

layer1 = nn.Conv3d(2, 16, kernel_size=3, stride=1)
layer2 = nn.Linear(16*2*2*2, 1)
func = lambda x: layer2(nn.ReLU()(layer1(x)).view(1, -1))

optimizer = torch.optim.Adam([guess_U], lr=3e-4)
for epoch in range(1000):
    optimizer.zero_grad()
    out = func(guess_U)
    out.backward()
    optimizer.step()
    with torch.no_grad():
        ...  # Unitary matrix constraints
</code></pre>
<p>Since PyTorch is only supported real values, considering U as A+iB, with A and B are real matrices, I have turned the problem as<br>
$$<br>
\min_{A, B} f(A, B),<br>
s.t. A A^T+B B^T = 1 and B A^T - A B^T=0.<br>
$$<br>
Since the constraints asre with respect to inner product and transpose, I don’t know how to implement it with PyTorch.</p>
nn.CrossEntropyLoss() for text with multiple dimension

username: VikasRajashekar
cooked: <p>Hello,</p>
<p>My logits are of dimention: torch.Size([64, 26, 7900])<br>
My target is of dimension: torch.Size([64, 26])</p>
<p>It is so because the output is from LSTM for some NLP task. 7900 is the size of the vocabulary.</p>
<p>How do I formulate the loss for this scenario?</p>
<pre><code class="lang-auto">loss = nn.CrossEntropyLoss()
input = torch.randn(64, 26, 7900, requires_grad=True)
target = torch.empty(64,26, dtype=torch.long).random_(5)
output = loss(input, target)
output.backward()
</code></pre>
<p>throws the error:</p>
<blockquote>
<p>ValueError: Expected target size (64, 7900), got torch.Size([64, 26])</p>
</blockquote>

username: chetan_patil
cooked: <p>The size of <code>logits</code> must be <code>[64,7900,26]</code>.</p>

username: ptrblck
cooked: <p>As Chetan explained, the model output tensor should contain the class indices in dim1 and additional dimensions afterwards.<br>
Generally, <code>nn.CrossEntropyLoss</code> expects these shapes:</p>
<ul>
<li>output: <code>[batch_size, nb_classes, *]</code>
</li>
<li>target <code>[batch_size, *]</code>
</li>
</ul>
Lift the test image size limit

username: kungkookie
cooked: <p>The code as :<br>
def validate(val_loader, net, criterion, optimizer, curr_epoch, writer):<br>
‘’’<br>
Runs the validation loop after each training epoch<br>
val_loader: Data loader for validation<br>
net: thet network<br>
criterion: loss fn<br>
optimizer: optimizer<br>
curr_epoch: current epoch<br>
writer: tensorboard writer<br>
return:<br>
‘’’<br>
net.eval()<br>
val_loss = AverageMeter()<br>
mf_score = AverageMeter()<br>
IOU_acc = 0<br>
dump_images = []<br>
heatmap_images = []<br>
for vi, data in enumerate(val_loader):<br>
input, mask, edge, img_names = data<br>
assert len(input.size()) == 4 and len(mask.size()) == 3<br>
assert input.size()[2:] == mask.size()[1:]<br>
h, w = mask.size()[1:]</p>
<pre><code>    batch_pixel_size = input.size(0) * input.size(2) * input.size(3)
    input, mask_cuda, edge_cuda = input.cuda(), mask.cuda(), edge.cuda()

    with torch.no_grad():
        seg_out, edge_out = net(input)    # output = (1, 19, 713, 713)

    if args.joint_edgeseg_loss:
        loss_dict = criterion((seg_out, edge_out), (mask_cuda, edge_cuda))
        val_loss.update(sum(loss_dict.values()).item(), batch_pixel_size)
    else:
        val_loss.update(criterion(seg_out, mask_cuda).item(), batch_pixel_size)

    # Collect data from different GPU to a single GPU since
    # encoding.parallel.criterionparallel function calculates distributed loss
    # functions

    seg_predictions = seg_out.data.max(1)[1].cpu()
    edge_predictions = edge_out.max(1)[0].cpu()

    #Logging
    if vi % 20 == 0:
        if args.local_rank == 0:
            logging.info('validating: %d / %d' % (vi + 1, len(val_loader)))
    if vi &gt; 10 and args.test_mode:
        break
    _edge = edge.max(1)[0]

    #Image Dumps
    if vi &lt; 10:
        dump_images.append([mask, seg_predictions, img_names])
        heatmap_images.append([_edge, edge_predictions, img_names])

    IOU_acc += fast_hist(seg_predictions.numpy().flatten(), mask.numpy().flatten(),
                               args.dataset_cls.num_classes)

    del seg_out, edge_out, vi, data

if args.local_rank == 0:
    evaluate_eval(args, net, optimizer, val_loss, mf_score, IOU_acc, dump_images, heatmap_images,
            writer, curr_epoch, args.dataset_cls)

return val_loss.avg, mf_score.avg
</code></pre>
<p>Large images cannot be tested, and the test image and the data set verification set share the same folder. Each time you need to test the image after training, you must cut the large image and provide the corresponding label. How to improve the test program so that you can enter a large image at will Can test results be produced?</p>

username: ptrblck
cooked: <p>If you are running out of memory during evaluation, you could wrap the evaluation code block into a <code>with torch.no_grad()</code> block, which would avoid storing the unnecessary intermediate tensors, which would be needed to calculate the gradients.</p>
<p>If that doesn’t help you could lower the batch size (if it’s not already 1) use model sharding (if more than a single GPU is available) or move the evaluation to the CPU.</p>
How to dynamically scale input of unknown magnitude?

username: Florian_Dietz
cooked: <p>I am building a RL agent that operates in a set of unknown environments.</p>
<p>The sensor inputs given by these environments are dynamically generated, so it is very hard for me to tell in advance how large they are going to be. Some inputs might be in the range [0;1], others in [10000; 100000000]. Some might have low variance, but others could have very high variance.</p>
<p>This make sit difficult to normalize the inputs.</p>
<p>Sampling from the environments is expensive, so it would be wasteful to wait for many iterations of sampling before I know how to scale the inputs. Worse, some inputs might only return high values in special circumstances that are only encountered at runtime.</p>
<p>Is there any mechanism in pytorch I can use as a preprocessor to normalize this completely unknown input data effectively? Preferably, this should not be too expensive to compute, because most of the inputs should be sane already. There are only a few inputs with crazy scaling.</p>
<p>There is also a second problem related to this one: Some inputs are categorical strings, and the categories are not known in advance. Currently, I just hash each string into a fixed-size vector, since this also works for data that hasn’t been seen before. I wonder if there is a better way to do it, or if pytorch has a function that already does this, so I don’t have to write the code for it myself, since it seems like a standard feature.</p>
CUDA manual startup

username: pmeier
cooked: <p>Consider the following snippet:</p>
<pre><code class="lang-python">from time import time
import torch

for run in range(5):
    start = time()
    torch.empty(1, device="cuda")
    end = time()
    print(f"{run}: {end - start:.2g} seconds")
</code></pre>
<p>On my machine this results in the following output:</p>
<pre><code class="lang-auto">0: 1.6 seconds
1: 3e-05 seconds
2: 7.9e-06 seconds
3: 7.4e-06 seconds
4: 7.2e-06 seconds
</code></pre>
<p>It is pretty clear that the first time a tensor a moved to the GPU some startup or initialization is happening. Is it possible to manually do this upfront?</p>
<hr>
<p>I’m asking this since I have multiple tests in my test suite that require CUDA. Depending on which test is run first, it indirectly performs the startup is therefore slow. If have a flag to skip slow tests, but the test in itself is not slow.</p>

username: ptrblck
cooked: <p>The first CUDA call initializes the CUDA context, which is this slow.<br>
You might want to add an init method, which calls into <code>torch.cuda.init()</code> or any random CUDA tensor creation.</p>
<p>Also note, that you are not timing the tensor creation in this current code snippet, since you are not synchronizing the code. The 1e-6 times might thus profile the Python overhead from the kernel launch.</p>

username: pmeier
cooked: <p><em>facpalm</em>. Sorry, I’m not sure how I missed <code>torch.cuda.init()</code> in the docs.</p>
<p>About the timing: this was just my quick and dirty method to show the difference between the initial startup and the following runs. I do not use this to actually profile my code.</p>

username: pmeier
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Are you sure <code>torch.cuda.init()</code> is the way to go here? I’ve added it before the loop and this has a negligible effect on the first run.</p>

username: ptrblck
cooked: <p>You are right. I only see the smaller ~3MB allocation, but not the creation of the CUDA context.<br>
In that case you might need to create a dummy tensor instead in the init method. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>