akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Decreasing validation accuracy in Quantization-aware training?

username: icyhearts
cooked: <p>I see a decreasing validation accuracy in this tutorial:<a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training</a></p>
<p>in the article, validation accuracy decreases from<br>
<code>Epoch 0 :Evaluation accuracy on 300 images, 77.67</code><br>
to<br>
<code>Epoch 7 :Evaluation accuracy on 300 images, 75.67</code></p>

username: dskhudia
cooked: <p>Is it continuously decreasing or just for one Epoch? Does it recover?</p>

username: Vasiliy_Kuznetsov
cooked: <p>hi <a class="mention" href="/u/icyhearts">@icyhearts</a>, this is expected for the tutorial as written, as it is using a toy dataset for speed (a tiny subsample of ImageNet, without enough images in each class).  You would need to run it on a real dataset to get better numerics on the validation set.</p>
How to properly implement LSTM layer

username: Keith72
cooked: <p>I am having difficulty getting my LSTM based model to train properly, the loss stays fairly high. I have an implementation of the same model in Keras and I’m trying to convert it to PyTorch. In Keras, my LSTM layer looks like this (as a Sequential model):</p>
<pre><code class="lang-auto">model.add(LSTM(36)) 
</code></pre>
<p>and in PyTorch:</p>
<pre><code class="lang-auto">self.lstm = nn.LSTM( input_size = 1,hidden_size = 36,num_layers = 1)
</code></pre>
<p>In Keras, the default is not to return the hidden state, so in PyTorch I’m not sure how to handle that in the forward function, or when to initialize the hidden states. Right now I’m using this to initialize the hidden state:</p>
<pre><code class="lang-auto">def init_hidden(self):
        return (
            torch.zeros(1, 1, 36, device='cuda'),
            torch.zeros(1, 1, 36, device='cuda'),
        )
</code></pre>
<p>And I call this at the beginning of the forward() function. When I use the lstm layer it looks like this:</p>
<pre><code class="lang-auto">out, self.hidden = self.lstm(out, self.hidden)
</code></pre>
<p>So maybe the hidden states are being initialized too often? I’m using a batch_size of 4096 if that matters.</p>
<p>Thank you!</p>
Why does gradcheck expect at least one input tensor to require gradient?

username: wasabi
cooked: <p>Say I want to check the gradient of f_\theta(x) wrt \theta, we should call gradcheck(f, x.requires_grad_()) rather than just gradcheck(f, x), why is this the case? The quantity we care about is df/d\theta, to do this, I guess Pytorch numerically check whether |f(x+h)-f(x)-grad(f)(x)h|/|h|-&gt;0? (or some other numerically efficient version). It seems redundant to let x require grad?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Indeed the gradients are only checked for the Tensors that are explicit inputs to the function. So if none of them requires gradients, that means that no gradient needs to be checked.<br>
You will need to give theta as an input to your function to be able to use gradcheck I’m afraid.</p>

username: wasabi
cooked: <p>Hi albanD. Thanks for your reply. Here I use \theta to represent the parameters of the network f. So say if f=nn.Linear(a,b), then we can write f(x)=Wx, and here \theta will be W. It doesn’t make sense to pass W as an input to gradcheck(), no? Because the input to gradcheck() should be the input to the forward() function of f. Are you suggesting that gradcheck() calculates df/dx rather than df/d\theta?</p>

username: albanD
cooked: <p>Yes gradcheck only checks gradient for inputs to the function that are Tensor that require gradients.</p>
<p>Note that in this case, you can “cheat” by doing: <code>gradcheck(lambda inp, *ignore: model(inp), (inp, *model.parameters())</code>.<br>
This will make all the parameters look like inputs for gradcheck. And all inplace changes to them must properly update the forward of the model (this will work with pytorch vanilla Module at least).</p>
How to calculate Laplacian for multiple batches in parallel

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I just have a quick question regarding parallelism and calculation of the Laplacian within PyTorch. As noted with a previous question I asked, I managed to get some code to calculate the Laplacian of a feed-forward like model which is defined as R^N -&gt; R^1 function. The original question is <a href="https://discuss.pytorch.org/t/implementing-calculation-of-the-laplacian/99644">here</a> if anyone is curious.</p>
<p>In short, I have a feedforward like model which takes in an input Tensor <code>x</code> of size <code>[S,N]</code> where <code>S</code> is the size of the batch and <code>N</code> is the number of nodes in the input layer. The code that calculated it was,</p>
<pre><code class="lang-auto">y = model(x) #where model is an R^N to R^1 function

laplacian = torch.zeros(x.shape[0]) #array to store values of laplacian

for i, xi in enumerate(x):
    hess = torch.autograd.functional.hessian(model, xi.unsqueeze(0), create_graph=True)
    laplacian[i] = torch.diagonal(hess.view(N, N) offset=0).sum()
</code></pre>
<p>For my particular problem, I need to compute multi batches in parallel pass them through my network and take a mean of all batches. The way I’ve chosen to represent this is by having an input Tensor of size <code>[S,B,N]</code> where <code>S</code>, <code>B</code>, <code>N</code> are the number of samples, number of batches, and number of inputs respectively.</p>
<pre><code class="lang-auto">x = get_inputs() #returns Tensor of shape [S,B,N]

laplacian_per_batch = torch.zeros(x.shape[0:2]) #array to store laplacian for each batch

for j in range(x.shape[1]): #cycle over each batch
  xj = x[:,j,:]     #input of batch j
  yj = model(xj)    #output of batch j
  laplacianj = torch.zeros(xj.shape[0]) #laplacian array for batch j
  for i, xi in enumerate(xj): #cycle over each element within batch j
    hess = torch.autograd.hessian(model, xi.unsqueeze(0), create_graph=True)  #calculate
    laplacian[i] = torch.diagonal(hess.view(N, N), offset=0).sum()            #laplacian
  laplacian_per_batch[:,j] = laplacian #store laplacian of batch j in total array
</code></pre>
<p>I know from previous questions that batch-vectorization of calculating the hessian is currently a work in progress, but I was wondering if it were possible to run each “batch loop” (the outer loop, enumerated by j) in parallel? This is because a single pass through loop “i” takes around 10s, but given I usually need anywhere from 8 to 32 batches the runtime for a single pass becomes somewhat impractical.</p>
<p>I assume this is possible as the individual batches aren’t connected to one another so, in theory, I could reshape <code>x</code> to <code>[S*B,N]</code>  but that would just compress the 2 nested loops in an equal-sized single loop, rather than doing each batch in parallel.</p>
<p>Thank you in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
DataLoader loading hidden image files

username: Wilan
cooked: <p>I’m trying to load images using a DataLoader with a specific batch size to feed into a CNN.</p>
<pre><code class="lang-auto">mean = torch.tensor([0.5], dtype=torch.float32)
std = torch.tensor([0.5], dtype=torch.float32)

train_transforms = transforms.Compose([transforms.Resize(size=(700, 700)),
                                       transforms.RandomHorizontalFlip(p=0.5),
                                       transforms.RandomPerspective(distortion_scale=0.1, p=1),
                                       transforms.Grayscale(1),
                                       transforms.ToTensor(),
                                       transforms.Normalize(mean, std)]) 

test_transforms = transforms.Compose([transforms.Resize((700, 700)),
                                      transforms.ToTensor(),
                                      transforms.Normalize(mean, std)]) 

train_data = ImageFolder(train_dir, transform=train_transforms)
test_data = ImageFolder(test_dir, transform=test_transforms)

train_loader = DataLoader(train_data, batch_size=64, shuffle=True)
test_loader = DataLoader(test_data, batch_size=32, shuffle=True)
</code></pre>
<p>When loading the (images, labels) pair vis next:</p>
<pre><code class="lang-auto">data_iter = iter(data_loader)
images, labels = next(data_iter)
</code></pre>
<p>I get this error:</p>
<pre><code class="lang-auto">UnidentifiedImageError: cannot identify image file &lt;_io.BufferedReader name='./data/train/1. angle-high/._Screen Shot 2020-11-29 at 8.23.59 AM.png'&gt;
</code></pre>
<p>Checking for hidden files using command+shift+. in MacOS, I can’t find any hidden files (starting with “._” in the directory).</p>

username: Alexey_Demyanchuk
cooked: <p>Hi. Yep, mac makes this kind of files in their file system.<br>
You have couple of options here:</p>
<ol>
<li>Make sure there are no files like this in your folder. You can check in mac terminal with <code>ls -la</code> which will show files starting with dots and remove lets say files started with “._” manually. Don’t like this option, though.</li>
<li>You can make you <code>ImageFolder</code> dataset robust to erroneus data. According to <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#id32" rel="noopener nofollow ugc">documentation</a> you can use argument <code>is_valid_file</code> and provide with callable to evaluate the file validity. After that your dataset will ignore invalid files.</li>
</ol>
<p>Minimal example:</p>
<pre><code class="lang-auto">import torchvision
from pathlib import Path

def check_valid(path):
    path = Path(path)
    return not path.stem.startswith('._')

ds = torchvision.datasets.ImageFolder('your_root_folder_here',  is_valid_file=check_valid)
</code></pre>
<p>Hope, it helps!</p>

username: Wilan
cooked: <p><a class="mention" href="/u/alexey_demyanchuk">@Alexey_Demyanchuk</a> Thank you so much! This is a really clean solution, and it works - also, didn’t know about <code>ls -la</code> thanks for the tip!</p>
Torch deg power produces error?

username: vymao
cooked: <p>I am getting a confusing error that I’m not sure why is occurring. This is the error I am getting:</p>
<pre><code class="lang-auto">torch.Size([24047])
tensor([1315, 1318, 1472, 1454, 1318, 1337, 1453, 1454, 1317, 1310])

Traceback (most recent call last):
    ...
    deg_inv = deg.pow(-1)
RuntimeError: Integers to negative integer powers are not allowed.
</code></pre>
<p>And this is part of the code:</p>
<pre><code class="lang-auto">from torch_scatter import scatter_add
from torch_geometric.utils import add_remaining_self_loops, add_self_loops, remove_self_loops
...
    if edge_weight ==None:
        edge_weight = torch.ones((edge_index.size(1), ), dtype=dtype,
                                     device=edge_index.device)
    fill_value = 1
    edge_index, edge_weight = add_self_loops(
        edge_index, edge_weight, fill_value, num_nodes)  
    row, col = edge_index
    deg = scatter_add(edge_weight, row, dim=0, dim_size=num_nodes)

    print(deg.size())
    print(deg[:10])

    deg_inv = deg.pow(-1) 
</code></pre>
<p>I have not found anything related to this online. I’m not sure why this would be occurring?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think this means that the pow function does not support raising integer values to negative integer powers?<br>
In particular, taking the inverse of an integer is not well defined because you don’t get an integer back.<br>
I think you wnat to change your input to be floating point numbers for this to work fine: <code>deg.float().pow(-1)</code> (and the result will be a floating point number as well).</p>
Vector subscript out of range and other errors with Intel C++

username: torchok
cooked: <p>Hello,</p>
<p>I’m new in pytorch c++ and I try to run any project. I use VS2017 under windows 7 x 64 and intel compilers v19.0.</p>
<p>I downloaded precompiled libtorch for release and debug and creating a VS project.</p>
<p>there are some problems with creating any simple test project.</p>
<hr>
<ol>
<li>with VS C++ toolchain compilers in Debug mode:<br>
runtime error : “vector subscript out of range”</li>
</ol>
<hr>
<ol start="2">
<li>with Intel C++ toolchain compilers in any mode:</li>
</ol>
<p>(2.a) in ArrayRef.h file, error : attribute does not apply to any entity<br>
I have to comment it in order to compile with intel. (is it ok ?)</p>
<p>(2.b) the same error 1 with VS runtime is always present</p>
<p>(2.c) LNK2001	unresolved external symbol "const c10::intrusive_ptr_target::`vftable’</p>
<hr>
<p>What am I doing wrong ?<br>
Could somebody give a working VS2017 with Intel C++ compilers project, please ?</p>
<p>Thank you for your help</p>

username: jakjak
cooked: <p>Hello,<br>
I got the same issue here with the exact same configuration, VS2017, windows 7 x64 and Intel compilers v19.0<br>
Did you find a solution ?<br>
Thanks!</p>
Pytorch JVP slow

username: pomonam
cooked: <p>Using functional.jvp (<a href="https://pytorch.org/docs/stable/autograd.html" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html</a>) and taking gradient with respect to it is 2 times slow in general and for larger networks (ResNet-20), it is 4 times slower. Are there any known issues on using JVP? Are there any methods to make it more efficient (maybe using other packages)?</p>

username: albanD
cooked: <p>Hi,</p>
<p>As mentioned in the note on the function definition, the function uses the “double backward trick” to compute the jvp and is thus expected to be slower.<br>
We are working on changing that for 1.7.</p>

username: pomonam
cooked: <p>Thank you for the reply! That is great to hear <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> . When do you expect 1.7 to be released?</p>

username: albanD
cooked: <p>It should be early November IIRC.</p>

username: InfoMax
cooked: <p>Are these changes already available in pytorch-nightly?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am afraid this got delayed and won’t be in 1.7.<br>
But they should get into master soon after we are done working on the release (mid/end November)</p>

username: InfoMax
cooked: <p>Hi,</p>
<p>Thank you for your answer. Is there an issue I can follow to know when this is merged into master?</p>

username: albanD
cooked: <p>Sure, you can follow <a href="https://github.com/pytorch/pytorch/issues/10223">https://github.com/pytorch/pytorch/issues/10223</a> that contains most of the discussions around this.</p>
VGG16 error: output size is too small

username: Tomash
cooked: <p>Hello everyone<br>
I try to train a little bit modified VGG16 network but I stuck with an following error:<br>
<code>RuntimeError: Given input size: (128x1x1). Calculated output size: (128x0x0). Output size is too small.</code><br>
I suspect that it’s something with chosen size of kernel or padding but I’m not sure what and how to choose these values better.<br>
As an input I take tensors of shape <code>[N, 256, 3, 3]</code>. As an output I just wish to have a single value.<br>
My modified code is here:</p>
<blockquote>
<pre><code>    net = models.vgg16(pretrained=True)  
    first_conv_layer = [nn.Conv2d(256, 3, kernel_size=3, stride=1, padding=1, dilation=1, groups=1, bias=True)]
    first_conv_layer.extend(list(net.features))
    net.features = nn.Sequential(*first_conv_layer)
    last_linear = [nn.Linear(1000, 1)]
    last_linear.extend(list(net.classifier))
    last_linear = last_linear[1:] + [last_linear[0]]
    net.classifier = nn.Sequential(*last_linear)
</code></pre>
</blockquote>
<p>I’ll be glad for any help! Cheers.</p>

username: PistonY
cooked: <p>I think this may caused by your input is too small, both vgg models have 2^5 downsample then pass by a <code>nn.AdaptiveAvgPool2d((7, 7))</code> that’s mean your input (h,w) at least is <code>7*2^5 = 224</code>.</p>

username: Tomash
cooked: <p>Thanks! That was a solution <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to find the index of values in a given tensor

username: 111429
cooked: <p>a = torch.IntTensor([1,3,2,1,4,2])<br>
b=[2,1,6]<br>
I want to find index of values in list b, with the result index sorted<br>
like output as  tensor([0, 2, 3, 5])</p>
<p>I know how to do it separately:<br>
torch.nonzero(a == 1).squeeze_(1)<br>
–&gt;tensor([0, 3])<br>
torch.nonzero(a == 2).squeeze_(1)<br>
–&gt;tensor([2, 5])<br>
torch.nonzero(a == 6).squeeze_(1)<br>
–&gt;tensor([], dtype=torch.int64)</p>
<p>but how can I do it at once? or in a better way.<br>
Thanks.</p>

username: 111429
cooked: <p>According to <a href="https://stackoverflow.com/questions/64300830/pytorch-tensor-get-the-index-of-the-element-with-specific-values" rel="noopener nofollow ugc">https://stackoverflow.com/questions/64300830/pytorch-tensor-get-the-index-of-the-element-with-specific-values</a>, this is a way to do it:<br>
a = torch.IntTensor([1,3,2,1,4,2])<br>
b=[2,1,6]<br>
mask = torch.zeros(a.shape).type(torch.bool)<br>
for e in b:<br>
…mask = mask + (a == e)</p>
<p>torch.nonzero(mask).squeeze_(1) # tensor([0, 2, 3, 5])</p>
TypeError: forward() takes 2 positional arguments but 3 were given in JIT trace (but works fine otherwise)

username: James_W
cooked: <p>I have this model:</p>
<pre><code class="lang-auto">class PriorBox4JIT(torch.nn.Module):
    def __init__(self, cfg):
        super(PriorBox4JIT, self).__init__()
        self.cfg=cfg
        self.idf = torch.nn.Identity()

    def forward(self,image_size):
        self_min_sizes = self.cfg['min_sizes']
        self_steps = self.cfg['steps']
        self_clip = self.cfg['clip']
        self_image_size = image_size
        self_feature_maps = self_image_size[1].to(dtype=torch.int64).tolist()
        self_name = "s"
        anchors = []
        for k, f in enumerate(self_feature_maps):
            min_sizes = self_min_sizes[k]
            for i, j in product(range(f[0]), range(f[1])):
                for min_size in min_sizes:
                    s_kx = min_size / self_image_size[0][1].item()
                    s_ky = min_size / self_image_size[0][0].item()
                    dense_cx = [x * self_steps[k] / self_image_size[0][1].item() for x in [j + 0.5]]
                    dense_cy = [y * self_steps[k] / self_image_size[0][0].item() for y in [i + 0.5]]
                    for cy, cx in product(dense_cy, dense_cx):
                        anchors += [cx, cy, s_kx, s_ky]

        # back to torch land
        output = torch.Tensor(anchors).view(-1, 4)
        if self_clip:
            output.clamp_(max=1, min=0)
        return output
</code></pre>
<p>I pass inputs in vanilla mode (i.e no tracing) like so:</p>
<pre><code class="lang-auto">        priorbox_jit = PriorBox4JIT(cfg)
        inp0=torch.Tensor([im_height, im_width])
        inp1=torch.Tensor(   [  [ceil(im_height/step), ceil(im_width/step)] for step in cfg['steps']    ]  )
        inputs=(inp0, inp1)

        priors_jit = priorbox_jit(  inputs )
        prior_data_jit = priors_jit.data
        print("prior_data_jit.shape", prior_data_jit.shape)
</code></pre>
<p>and this returns me what I expect: <code>prior_data_jit.shape torch.Size([8142, 4])</code></p>
<p>However, when I try to trace the same model like so:</p>
<p><code>traced_script_module2 = torch.jit.trace(priorbox_jit.eval(), inputs )</code></p>
<p>I get thrown an error:</p>
<p><code>TypeError: forward() takes 2 positional arguments but 3 were given</code></p>
<p>I am confused why this is - it seems to do the forward() correctly when I do not trace…</p>
<p>Any pointers would be great <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Save and load model issue

username: Aiman_Mutasem-bellh
cooked: <p>Hello, dears <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m trying to implement the code of <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="noopener nofollow ugc">The Annotated Transformer</a> from Harvard NLP, everything is working well but I need to save and load my own mode instead of their (already) trained one, which is <code>iwslt.pt</code>.</p>
<pre><code class="lang-auto">model_opt = NoamOpt(model.src_embed[0].d_model, 1, 2000,
            torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))

N_EPOCHS = 2
CLIP = 1

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):
    model_par.train()
    run_epoch((rebatch(pad_idx, b) for b in train_iter), 
                  model_par, 
                  MultiGPULossCompute(model.generator, criterion, 
                                      devices=devices, opt=model_opt))
    model_par.eval()
    valid_loss = run_epoch((rebatch(pad_idx, b) for b in valid_iter), 
                          model_par, 
                          MultiGPULossCompute(model.generator, criterion, 
                          devices=devices, opt=None))
        if valid_loss &lt; best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model_par.state_dict(), 'model.pt')
        
    print(valid_loss)
</code></pre>
<p>For loading model I use :</p>
<p><code>model = torch.load("model.pt")</code></p>
<p>I got an error of<br>
<code>AttributeError: 'collections.OrderedDict' object has no attribute 'encode'</code></p>
<p>Also, I used :<br>
<code>model.load_state_dict(torch.load('en-de-model.pt'))</code></p>
<p>I got an error : <code>AttributeError: 'collections.OrderedDict' object has no attribute 'load_state_dict'</code></p>
<p>Could you help to find the optimal way to save and load my won model?</p>

username: Alexey_Demyanchuk
cooked: <p>After saving the model state dict like this: <code>torch.save(model_par.state_dict(), 'model.pt')</code><br>
In general, you need to do these steps:</p>
<ol>
<li>Initialize the new instance of the model the same way you initialize it before training and saving, like: <code>model_par= MyFancyModelClass(some_args_here)</code>
</li>
<li>Load state dict into model: <code>model_par.load_state_dict(torch.load('model.pt'))</code>
</li>
</ol>
What is the correct way to assign new value to buffer?

username: MariosOreo
cooked: <p>Hello,</p>
<p>I want to save some intermediate feature values during training. However, I failed to assign new values to existing buffers by <code>getattr()</code> and <code>setattr()</code>, the example is as follows:</p>
<pre><code class="lang-auto">class MyNetwork(nn.Module):
    def __init__(self):
        self.conv1 = nn.Conv2d(3, 3, 3, 1, 1)
        self.conv2 = nn.Conv2d(3, 3, 3, 1, 1)
        self.register_buffer('inter_feature', torch.Tensor())
        self.register_buffer('count', torch.tensor(0))
        self.register_buffer('weight', torch.ones(10))
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        
        # case1: does not work for updating self.count
        count = getattr(self, 'count')
        count = count + 1
        setattr(self, 'count', count)
        
        # case2: work
        count = getattr(self, 'count')
        count += 1
        setattr(self, 'count', count)
        
        # case3: does not work
        feature = getattr(self, 'feature')
        feature = x
        setattr(self, 'feature', feature)
        
        # case4: work
        feature = getattr(self, 'feature')
        feature.data = x.data
        
        return x
</code></pre>
<p>Could you tell me how can I correctly use <code>getattr</code> and <code>setattr</code> to update buffers during training?</p>
<p>Thanks in advance!</p>

username: PistonY
cooked: <p>I think use <code>self.xxx = aaa</code> may a good way for change a buffer.</p>

username: MariosOreo
cooked: <p>Thanks for reply!<br>
Absolutely, using <code>self.xxx = aaa</code> is a convenient way to update buffers. But, in my code, I need to update buffer like this:</p>
<pre><code class="lang-auto">for i in range(stages):
    self.register_buffer('feature_stage{}'.format(i))
</code></pre>
<p>In this case, I need to access buffers by <code>getattr(self,  feature_stage{}.format(i))</code>, and I cannot use <code>self.feature_stage{}.format(i) = aaa</code> to update a buffer.</p>
<p>Does there exist more efficient way like <code>self.xxx = aaa</code> for my case?</p>

username: PistonY
cooked: <p>Why not use a matric like a <code>NxM</code> tensor, the num <code>N</code> is <code>stages</code>.<br>
And then you could change by <code>self.xxx[i] = aaa</code>.</p>

username: MariosOreo
cooked: <p>Thanks a lot, I will have a try <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Pytorch matmul in GPU is slower than CPU

username: kinwai_cheuk
cooked: <p>I am comparing how much faster is the matmul on GPU, surprisingly, my test result shows that running on  a GPU is slower than running on a CPU.</p>
<pre><code class="lang-auto">import torch
import numpy as np

'''
Tensor shape = (batch,
                attention heads,
                features per head,
                height,
                weight,
                attention window
                )
Goal: We want to apply dot product
to only the last dimension
'''
# softmax score for the Query and Key
QK = torch.randn([64, 8, 4, 28, 28, 9])
# Value Tensor 
V = torch.randn([64, 8, 4, 28, 28, 9])

def method1(QK, V):
    """matmul way"""
    
    # Preparing the right shape for dot product in the lastt dimension
    out1 = torch.matmul(QK.unsqueeze(-2), V.unsqueeze(-1))
    # Reshape it back to the original shape
    out1 = out1.squeeze(-1).squeeze(-1)
    return out1

def method2(QK,V):
    """Einstein summation"""
    return torch.einsum('bnchwk,bnchwk -&gt; bnchw', QK, V)

# torch CPU

%timeit -n 500 method1(QK,V)
%timeit -n 500 method2(QK,V)

# torch GPU

QK = QK.cuda()
V = V.cuda()

%timeit -n 500 method1(QK,V)
%timeit -n 500 method2(QK,V)
</code></pre>
<p><strong>Torch CPU</strong><br>
Method1: 2.7 ms ± 140 µs per loop (mean ± std. dev. of 7 runs, 500 loops each)<br>
Method2: 2.64 ms ± 92.1 µs per loop (mean ± std. dev. of 7 runs, 500 loops each)</p>
<p><strong>Torch GPU</strong><br>
Method1: 3.34 ms ± 206 µs per loop (mean ± std. dev. of 7 runs, 500 loops each)<br>
Method1: 3.43 ms ± 123 µs per loop (mean ± std. dev. of 7 runs, 500 loops each)</p>
<p>Am I doing anything wrong? The result doesn’t make sense to me.</p>

username: ptrblck
cooked: <p>CUDA operations are asynchronously executed so you would need to synchronize your code before starting and stopping the timers via <code>torch.cuda.synchronize()</code> to get proper timings. Also, the operations could be executed in a loop to calculate the average time and reduce the noise a bit.</p>
<p>That being said, if your workload is small you won’t be able to saturate the GPU (and would also have the kernel launch overheads) and the CPU might be indeed faster.</p>
Where is "SysML gym" mentioned in developer day 2020?

username: ugurkanates
cooked: <div class="youtube-onebox lazy-video-container" data-video-id="jaPVoObpdO0" data-video-title="PyTorch Developer Day 2020 | Full Livestream" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=jaPVoObpdO0" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/jaPVoObpdO0/maxresdefault.jpg" title="PyTorch Developer Day 2020 | Full Livestream" width="690" height="388">
  </a>
</div>

<p>Here they mention a gym environment based on compilers. But there is literally no mention of this in entire internet</p>
Resampling the 3D Image to required shape

username: nithesh5
cooked: <p>Hi,</p>
<p>I want to resample the 3D image to (256, 256, 166) if their shapes are different.</p>
<p>Example: (192, 192, 160) to (256, 256, 166)<br>
and (256, 256, 170) to (256, 256, 166) etc</p>
<p>I used the below code to do that(code referred from <a href="https://www.kaggle.com/mechaman/resizing-reshaping-and-resampling-nifti-files" rel="noopener nofollow ugc">https://www.kaggle.com/mechaman/resizing-reshaping-and-resampling-nifti-files</a>)</p>
<pre><code>    target_img_shape = (256, 256, 166)

    if img_shape != target_img_shape:
        resampled_nii = resample_img(image, target_affine=np.eye(4) * 2, target_shape=target_img_shape,
                                     interpolation='nearest')
        resampled_img_data = resampled_nii.get_fdata()
    else:
        resampled_img_data = image.get_fdata()
</code></pre>
<p>I am getting the required shape but the resultant image is too small. I have the following questions:</p>
<ol>
<li>Is the above code is correct</li>
<li>Is there any other ways to resample the 3D image to the required size ie., (256, 256, 166)</li>
</ol>
<p>Thank you.</p>

username: Naruto-Sasuke
cooked: <p>Doesn’t the original <code>interpolate</code> function work fine for your case ?<br>
<code>The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width.</code></p>
<p><code>The modes available for resizing are: nearest, linear (3D-only), bilinear, bicubic (4D-only), trilinear (5D-only)</code>, however, you can only adopt <code>trilinear</code> mode.</p>
Is there a way to get entire data metrics with batch loading data sets too large for memory?

username: localh
cooked: <p>I am wondering if there is a way to get statistics like mean and standard deviation when using the Torch dataset class to batch load very large data sets.</p>
<p>I need to normalize my features, which means I need the standard deviation and mean for each feature column. My data has millions of rows, so it cannot fit into memory. How might I accomplish this task? Some example code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, Subset, DataLoader
from sklearn import datasets
import pandas as pd
import numpy as np

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

X, y = datasets.make_classification(n_samples=1000,
                                         n_features=2,
                                         n_informative=2,
                                         n_redundant=0,
                                         n_classes=2,
                                         random_state=15)
# place data into df
df = pd.DataFrame({'x1': X[:, 0], 'x2': X[:, 1], 'y': y})
df.to_csv('classification_demo.csv', index=False)

class CSVDataset(torch.utils.data.Dataset):
    def __init__(self, path, chunksize, nb_samples, transform=None):
        self.path = path
        self.chunksize = chunksize
        self.len = nb_samples // self.chunksize
        self.transform = transform

    def __len__(self):
        return self.len

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        x = next(
            pd.read_csv(
                self.path,
                skiprows=idx * self.chunksize,  #+1 to skip the header
                chunksize=self.chunksize))

        target = torch.as_tensor(x.values)[0][-1]  # last col
        features = torch.as_tensor(x.values)[0][0:-1]  # all but last

        # pull a sample in a dict
        sample = {'features': features,
                  'target': target,
                  'idx': torch.as_tensor(idx)}

        if self.transform:
            sample = self.transform(sample)
        return sample

# instantiate the lazy data set
csv_dataset = CSVDataset('classification_demo.csv', chunksize=1, nb_samples=1000, transform=None)
</code></pre>

username: PistonY
cooked: <p>I think you could calculate them before training. If you data is changing you could write a func to do it when dataset init.</p>
How to modify and rewrite the activation output of a layer before applying the output to next layer?

username: mHr029
cooked: <p>LeNet architecture is like this:<br>
CONV1-&gt;RELU1-&gt;POOL1-&gt; CONV2-&gt;RELU2-&gt;POOL2)-&gt; FC1-&gt;RELU-&gt;…</p>
<p>Suppose, in forward pass, I want to modify the output of RELU1 and then rewrite the modified version before going to POOL1 layer so that it would reflect on backward pass.</p>
<p>In pytorch, how can I achieve this?</p>

username: ptrblck
cooked: <p>You could use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> and manipulate the <code>output</code> inplace in the hook.</p>

username: mHr029
cooked: <p>Thank you for your help.<br>
But I fixed this by declaring custom ReLU class to make an effect on backward pass.</p>
Pytorch model deployment in mediapipe

username: amitkayal
cooked: <p>Hello Experts,</p>
<p>I am trying to deploy a pytorch model into mediapipe. Was wondering if anyone has done such deployment into mediapipe and used it? As of now, i have mediapipe installed in my ubuntu…</p>
<p>Thanks<br>
Amit</p>

username: dsuess
cooked: <p>If you can convert your model to a format supported by mediapipe, that’s the easiest solution. I’ve toyed around with writing a calculator for mediapipe supporting pytorch here: <a href="https://github.com/dsuess/mediapipe-pytorch" rel="noopener nofollow ugc">https://github.com/dsuess/mediapipe-pytorch</a>. It’s just a very basic setup, but it can easily be extended and refactored into separate modules for tensor conversion, inference, etc.</p>
<p>The biggest challenge I ran into is the build environment: pytorch has a bazel environment for linux, so that’s not too hard to get working. However, as far as I can tell, you need to use cmake for the Android/iOS builds, so that’s more challenging. What finally stopped me is the lack of GPU support on mobile in pytorch ATM.</p>
CUDA runtime error out of memory while trying to build the model

username: albat3ross
cooked: <p>I’m trying to build up a simple NN model and use GPU to run it. Then I have the memory error as trying to upload the model to Cuda. I have tried it locally on my computer, uploaded my code and run it on Kaggle notebook as well as Google colab, and the same issue happens for all the cases. So that I’m sure it’s caused by reasons other than PyTorch version, Cuda version, running environment, or batch size (since it’s not caused by allocating the input into GPU)</p>
<p>the code for building up the model is:<br>
device = torch.device(‘cuda’)<br>
des = model.DenseNet(num_init_features=32,num_classes=10)<br>
cnn = nn.Sequential(<br>
nn.Conv2d(1,3,5,1,2,bias=True),<br>
des,<br>
)<br>
torch.cuda.empty_cache()<br>
cnn.to(device)</p>
<h2>The error detail on Google colab is like:</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
6 )<br>
7 torch.cuda.empty_cache()<br>
----&gt; 8 cnn.to(device)<br>
9</p>
<p>5 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in convert(t)<br>
422<br>
423         def convert(t):<br>
–&gt; 424             return t.to(device, dtype if t.is_floating_point() else None, non_blocking)<br>
425<br>
426         return self._apply(convert)</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 11.17 GiB total capacity; 10.52 GiB already allocated; 1.81 MiB free; 349.51 MiB cached</p>
<p>So as it shows it’s trying to allocate 2MB from 350MB space and failed, restarting kernel isn’t helping, using empty cache right in front of the code isn’t helping, everything is installed properly on local compute and the online running environment should be all set. I’ve searched almost all the cases from the internet and still can’t find the reason it doesn’t work, please help me.</p>

username: vainaijr
cooked: <p>did you go to runtime -&gt; manage sessions -&gt; get a list of active sessions, and terminate ones which you are not using in colab?</p>

username: albat3ross
cooked: <p>I did and there’s only one active session there.</p>

username: christofferaakre
cooked: <p>I was having a similar issue and this fixed it for me in google colab! thanks!</p>
Fastest way to collect intermediate feature maps

username: JamesDickens
cooked: <p>I have a model mimicking the region proposal network(rpn) of faster r-cnn, and I wish to collect some of the intermediate feature maps before the head of the rpn. At present my code looks like:</p>
<blockquote>
<pre><code> for index, child in enumerate(self.backbone.children()):
        images = child(images)
        if index in self.layer_outputs_to_keep:
            feature_maps.append(images)
</code></pre>
</blockquote>
<p>Since the dataset I am training on is very large, I want to make sure that this is indeed the fastest possible way to do so. Any insights/tips appreciated.</p>
How to create independent blocks of nn pythonically

username: DidierDeschamps
cooked: <p>I simplify the thing but I want to build the following structure:</p>
<p>x -&gt; [Encoder] -&gt; x_encoded<br>
x_encoded -&gt; [predictor 1] -&gt; y_1<br>
x_encoded -&gt; [predictor 2] -&gt; y_2<br>
…<br>
x_encoded -&gt; [predictor n] -&gt; y_n<br>
output = concat(y_1, …, y_n)</p>
<p>All the different predictors have the same structure so I did something like that:</p>
<pre><code class="lang-auto"> self.predictors = nn.ModuleDict()
 for i in range(n):
        self.predictors["predictor_" + str(i)] = nn.Sequential(
                &lt;&lt; STRUCTURE OF THE PREDICTORS &gt;&gt;
            )
</code></pre>
<p>And my forward will be something like:</p>
<pre><code class="lang-auto">def forward(self, x):
        #Encoder
        x =  self.encoder(x)
        #Predictors
        output = torch.cat(tuple(
                      self.predictors["predictor_" + str(i)](x) for i in range(n))
                      )
</code></pre>
<p>Is it the right way to do it ?</p>

username: ptrblck
cooked: <p>Yes, the code looks generally alright.<br>
One minor suggestion: if the first module in <code>predictors</code> might apply an inplace operation, you would need to clone the input via <code>x.clone()</code> before passing it to these predictors. Otherwise it’ll be changed inplace and the following predictors would get the already manipulated input.</p>

username: DidierDeschamps
cooked: <p>Thanks for the suggestion ! A friend one mine coded a multilinear class as well to deal with that kind of problem</p>
<pre><code class="lang-auto">import torch

class MultiLinear(torch.nn.Module):
    def __init__(self, input_size, output_size, nb_estimators, activation=torch.relu):
        super().__init__()
        self.weights = torch.nn.Parameter(torch.randn(nb_estimators, input_size, output_size))
        self.weights.requires_grad = True
        self.activation = activation    def forward(self, x_batch):
        if x_batch.ndim == 2:
            # x_batch: (batch_size, input_size)
            z = torch.einsum("ij,mjl-&gt;ilm", x_batch, self.weights)
        elif x_batch.ndim == 3:
            # x_batch: (batch_size, input_size, nb_estimators)
            z = torch.einsum("ijm,mjl-&gt;ilm", x_batch, self.weights)
        else:
            raise ValueError("input dims must be 2 or 3")
        output = self.activation(z)
        # output: (batch_size, output_size, nb_estimators)
        return output
</code></pre>
Replace Mask R-CNN loss function

username: FiReTiTi
cooked: <p>Hi,<br>
I wanted to test other loss function for Mask R-CNN, so I followed this <a href="https://github.com/pytorch/vision/issues/1882" rel="noopener nofollow ugc">answer here</a>.</p>
<p>This is what I did as a test:</p>
<ul>
<li>I took <code>maskrcnn_loss</code>, changed the name, and added a <code>print</code> to make sure that everything was ok.</li>
<li>I tried to use <code>roi_heads.mask_rcnn_loss = My_Loss</code>
</li>
<li>And I alsoI tried to use <code>mymodel.roi_heads.mask_rcnn_loss = My_Loss</code>
</li>
</ul>
<p>Unfortunately, in both case, <code>MyLoss</code> was never called (print never executed).</p>
<p>I use TorchVision 0.6.1.</p>
<p>What am I doing wrong?</p>

username: ptrblck
cooked: <p>Is the latest suggestion working for you?</p>
<blockquote>
<p>For what I proposed, you would need to do instead<br>
<code>torchvision.models.detection.roi_heads.mask_rcnn_loss = My_Loss</code><br>
i.e., directly modify what torchvision is using.<br>
Is that what you tried doing?</p>
</blockquote>

username: FiReTiTi
cooked: <p>Unfortunately no <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
See my <a href="https://github.com/pytorch/vision/issues/1882" rel="noopener nofollow ugc">answer here</a>.</p>

username: FiReTiTi
cooked: <p>As pointed <a href="https://github.com/pytorch/vision/issues/1882#issuecomment-736724557" rel="noopener nofollow ugc">here</a>, it should not be:</p>
<aside class="quote no-group" data-username="FiReTiTi" data-post="1" data-topic="104085">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/firetiti/48/28042_2.png" class="avatar"> FiReTiTi:</div>
<blockquote>
<p>roi_heads.mask_rcnn_loss = My_Loss</p>
</blockquote>
</aside>
<p>but <code>roi_heads.maskrcnn_loss = My_Loss</code></p>
Training loss jumps up suddenly from nearly 1 to 39?

username: grizzlycoder
cooked: <p>I’m trying to train a self-supervised model to detect human interactions. After creating the positive and negative pairs, when I run my training module I get a strange training loss.</p>
<p>For the first half of the batches in each epoch I get a loss between 0.5-1 however in the middle it suddenly jumps to values like 39.8 which is very weird. I am using a contrastive loss function. Is there any reason as to why this could be happening?</p>

username: ptrblck
cooked: <p>If you are not shuffling the dataset, you might encounter a “difficult” example in the same iteration, which would increase the loss. Could you shuffle it and check, if the behavior changes?</p>

username: grizzlycoder
cooked: <p>Hi, thank you for this! In fact, the reason for my loss being so high all of a sudden was because my loss function was contrastive loss for which I had set the default margin to 10 (really high in my use case!) but now I have another issue, my loss does not converge. I’ll try reshuffling the dataset and test out some other things.</p>
ResNet, time series classification and input tensor dimension mapping

username: sobek
cooked: <p>I’m trying to reimplement the ResNet from this <a href="https://arxiv.org/pdf/1611.06455.pdf" rel="noopener nofollow ugc">paper</a> for use in time series classification. Since it’s my first time working with convolutional layers, I’m a bit confused about how to arrange the input tensor for the convolution.</p>
<p>The <a href="https://github.com/cauchyturing/UCR_Time_Series_Classification_Deep_Learning_Baseline/blob/master/ResNet.py" rel="noopener nofollow ugc">original implementation</a> in Keras uses 2d layers. I’m given to understand that the convention in torch for a 2d layer is that the tensor should look like <code>[batch_size, channels, height, width]</code>. For image input, this makes sense to me but I’m not sure how multivariate time series data maps to this schema, as in my mind, a 1d layer would make much more sense. Could somebody please point out to me how <code>[sample_axis, time_axis, feature_axis]</code> could be mapped to this schema and why the paper uses a 2d layer?</p>
<p>Am I correct in assuming that in my case, <code>channels</code> would be 1? Is this the same as using Conv1d or am I overlooking something?</p>

username: sobek
cooked: <p>Turns out I had a thorough misunderstanding of the original paper, as all the datasets are image series, so it makes perfect sense that they used 2d layers.</p>
How to batch normalization in LSTM

username: Mingyu_Kim
cooked: <p>How to batch normalization in LSTM<br>
My code is here…</p>
<pre><code>class LSTM(nn.Module):
    def __init__(self, **model_config):
        super(LSTM, self).__init__()
    
        if model_config['emb_type'] == 'glove' or 'fasttext':
            self.emb = nn.Embedding(model_config['vocab_size'],
                                   model_config['emb_dim'],
                                   _weight = TEXT.vocab.vectors)
        else:
            self.emb = nn.Embedding(model_config['vocab_size'],
                                   model_config['emb_dim'])
        
        self.bidirectional = model_config['bidirectional']
        self.num_direction = 2 if model_config['bidirectional'] else 1
        self.model_type = model_config['model_type']
    
        self.LSTM = nn.LSTM(input_size = model_config['emb_dim'],
                           hidden_size = model_config['hidden_dim'],
                           dropout = model_config['dropout'],
                           bidirectional = model_config['bidirectional'],
                           batch_first = model_config['batch_first'])
    
        self.fc = nn.Linear(model_config['hidden_dim'] * self.num_direction,
                          model_config['output_dim'])
    
        self.drop = nn.Dropout(model_config['dropout'])
    
    def forward(self, x):
        emb = self.emb(x)
        output, (hidden, cell) = self.LSTM(emb)
        last_output = output[:,-1,:]
    
        return self.fc(self.drop(last_output))</code></pre>
2D matrix multiplication with scalar

username: John_Price
cooked: <p>I would like to multiply a 2D matrix by a scalar (say 0.016) and at the end have a 2D matrix as the result. What is the best way to do this? Thanks in advance!</p>

username: vaisakh_m
cooked: <p><strong>k * X</strong> is the operation you are looking for, where, <strong>X</strong> is your 2D matrix and <strong>k</strong> is the scalar value.</p>

username: John_Price
cooked: <p>thank you very much!</p>
<p>I still have one question:</p>
<p>if <strong>k</strong> is a scalar value and <strong>X</strong> is a 2D matrix.<br>
Can I also use <strong>X * k</strong> (which means does the * operator work in both directions?)</p>
Torch.cat() different variable issue?

username: krishna511
cooked: <p>I am contenanting three fetures  who have same no of rows,</p>
<p>A=torch.cat(chroma_feats.T,mel_feats.T,mfcc_feats.T)</p>
<p>getting an error</p>
<p>cat() received an invalid combination of arguments - got (numpy.ndarray, numpy.ndarray, numpy.ndarray), but expected one of:</p>
<ul>
<li>(tuple of Tensors tensors, name dim, *, Tensor out)</li>
<li>(tuple of Tensors tensors, int dim, *, Tensor out)</li>
</ul>
<p>what to do?</p>

username: 4158ndfkvBHJ1
cooked: <p>You need to provide a tuple of tensors instead of just 3 tensors :<br>
tuple = (tensor_1, tensor_2)<br>
new_tensor = torch.cat(tuple)</p>