akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The train accuracy value has a sudden rise at end of every epoch

username: yiqiang_wu
cooked: <p>Hello, I meet a problem during training. The train accuracy value has a sudden drop at end of every epoch. It is stranger，how to explain this issue?<br>
(ps: I have shuffled the dataloader option: shuffle=True)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7f1785534c1987dcaa6ac2c71336d315808bbdc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b7f1785534c1987dcaa6ac2c71336d315808bbdc" title="捕获"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7f1785534c1987dcaa6ac2c71336d315808bbdc_2_690x138.jpeg" alt="捕获" data-base62-sha1="qfeIUag5xcvI1mHgDrGkPclED3S" width="690" height="138" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b7f1785534c1987dcaa6ac2c71336d315808bbdc_2_690x138.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7f1785534c1987dcaa6ac2c71336d315808bbdc.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7f1785534c1987dcaa6ac2c71336d315808bbdc.jpeg 2x" data-dominant-color="FAF6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获</span><span class="informations">776×156 14.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you check or post the code you are using to calculate the loss or accuracy?<br>
I guess the last batch might be smaller, while you might be using the batch size to calculate the metric.</p>

username: yiqiang_wu
cooked: <p>Oh, you are right. Thanks for your reply.</p>
Partial model parameter updates due to missing input data

username: Eiphodos
cooked: <p>Below is a simplified version of a model I am developing</p>
<pre><code class="lang-auto">class ExampleModel(nn.Module):
    def __init__(self, model_1, model_2):
        super(ExampleModel, self).__init__()

        self.model_1_name = 'Model_1'
        self.model_1 = model_1
        self.add_module(self.model_1_name, self.model_1)

        self.model_2_name = 'Model_2'
        self.model_2 = model_2
        self.add_module(self.model_2_name, self.model_2)

        self.fc_name = 'Linear_layer'
        self.fc_linear = nn.Linear(self.2, 1)
        self.add_module(self.fc_name, self.fc_linear)

    def forward(self, input_a, input_b):
        x1 = self.model_1(input_a)
        x2 = self.model_2(input_b)
       
        x = torch.cat((x1, x2), 1)

        x = self.fc_linear(F.relu(x))

        return x
</code></pre>
<p>My end goal is to support cases where data in the input batches might be missing and avoid updating model parameters based on missing data or any “dummy” data that is sent in.</p>
<p>As the data sent to forward is batched, a single batch might have missing data in some places and relevant data in other. Currently the missing data is replaced with a torch.zeros tensor.</p>
<p>Is it possible to create an architecture where the gradients generated from these inputs are excluded from the backwards pass?<br>
Is there a better way to go about it which does not involve creating torch.zeros tensors as dummy data?</p>

username: googlebot
cooked: <p>It is doable if you have one availability state per sample, or can split columns into “core” and “optional” table; then multiplying some entries in either unreduced loss tensor or “optional” residual layer output by zeros does the trick. With varying missing column combinations it is much harder to do, without hindering performance.</p>
How to get gradient of loss w.r.t. parameters?

username: Hdk
cooked: <p>I have to implement a loss in backward of convolution layer as illustrated in below code. But I don’t know how can I get <code> grad_weight =  grad_weight + cont_loss_weight</code> such that the shape of cont_loss_weight should be same as that of grad_weight and <code>grad_bias = grad_bias + cont_loss_bias</code> such that the shape of cont_loss_bias should be same as that of grad_bias.</p>
<pre><code class="lang-auto">class Custom_Convolution(torch.autograd.Function):    
    
    @staticmethod
    def forward(ctx, input, weight, bias, stride, padding):  #input's shape = ([batch_size=100, 96, 8, 8])
        output = torch.nn.functional.conv2d(input, weight, bias, stride, padding)  
        ctx.save_for_backward(input, weight, bias, output)
        return output    #output's shape = ([[batch_size= 100,128, 4, 4])
    
    @staticmethod
    def backward(ctx, grad_output):  #grad_output size = ([batch_size= 100, 128, 4, 4])
    
        input, weight, bias, output = ctx.saved_tensors      #input's size = ([batch_size=100 , 96,8,8])
              
        features = output
        features = features.view(features.shape[0], features.shape[1], -1)
        features = torch.nn.functional.normalize(features,dim=1)
        #Total_features=  features.shape[0]* features.shape[1]
                 
        cont_loss = torch.tensor([0.]).requires_grad_(requires_grad=True).to(dev)  # shape: ([1])
        for ..... :
                # My code for loss...   includes some operations like torch.div,exp,sum...
                # Calculation of loss for each feature 'i' (Li) out of Total_features
                # Total loss = sum of all Li (Number of Li values = features.shape[0]* features.shape[1])
                 
                cont_loss = cont_loss.add(cont_loss_img)   # Total contrastive loss of the batch
         
        grad_input = grad_weight = grad_bias = None

        if ctx.needs_input_grad[0]:
            grad_input = torch.nn.grad.conv2d_input( input.shape, weight, grad_output) #shape = ([batch_size,96,8,8])
                    
        if ctx.needs_input_grad[1]:
            grad_weight = torch.nn.grad.conv2d_weight(input, weight.shape, grad_output)  #shape = ([128,96,5,5])
            # cont_loss_weight = torch.nn.grad.conv2d_weight(cont_loss, weight.shape, grad_output)                                              
            # grad_weight =  grad_weight + cont_loss_weight
            
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum((0,2,3))        #shape = ([128])
            #grad_bias = grad_bias + cont_loss_bias          
            
        if bias is not None:
            return grad_input, grad_weight, grad_bias, None, None
        else:
            return grad_input, grad_weight, None, None

</code></pre>
Non-negative embeddings [root question: enforcing non-negativity in learnt params]

username: Nipun_Batra
cooked: <p>Is there a way we can enforce non-negativity constrains in embeddings?</p>
<p>In Keras, one could do like using the <code>keras.constraints.non_neg</code></p>
<pre><code class="lang-auto">from keras.constraints import non_neg
movie_input = keras.layers.Input(shape=[1],name='Item')
movie_embedding = keras.layers.Embedding(n_movies + 1, n_latent_factors, name='NonNegMovie-Embedding', embeddings_constraint=non_neg())(movie_input)
</code></pre>
<p>I’m trying to rewrite in PyTorch, a <a href="https://nipunbatra.github.io/blog/2017/recommend-keras.html" rel="nofollow noopener">blogpost I wrote on non-neg. matrix factorization in Keras.</a></p>

username: SimonW
cooked: <p>How is the keras one implemented? Is it just simply clamping below at 0 after each gradient update? If so you can just also do so after each optim update in pytorch.</p>

username: Nipun_Batra
cooked: <p>In <a href="https://github.com/keras-team/keras/blob/master/keras/constraints.py" rel="nofollow noopener">https://github.com/keras-team/keras/blob/master/keras/constraints.py</a> we have:</p>
<pre><code class="lang-auto">class NonNeg(Constraint):
    """Constrains the weights to be non-negative.
    """

    def __call__(self, w):
        w *= K.cast(K.greater_equal(w, 0.), K.floatx())
        return w
</code></pre>
<p>I guess this is just clamping it to minimum of 0.</p>

username: Kyeongpil
cooked: <p><a class="mention" href="/u/nipun_batra">@Nipun_Batra</a><br>
Clamping all embedding vectors per every itersion is so time consuming.<br>
How to clamp only updated embedding vectors with cuda variable?</p>

username: SimonW
cooked: <p>After <code>.backward()</code>, look at the embeddings <code>.weight.grad</code> attribute to get the updated indices. After <code>optim.step()</code>, clamp weights at those indices.</p>

username: jroberayalas
cooked: <p>Would the same effect be achieved if we add a ReLU layer after the embedding layer?</p>

username: Richard_S
cooked: <aside class="quote no-group" data-username="SimonW" data-post="2" data-topic="12287">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/48/1702_2.png" class="avatar"> SimonW:</div>
<blockquote>
<p>clamping</p>
</blockquote>
</aside>
<p>How did you manage to implement this in pytorch? would you mind sharing your code?</p>
Biaffine module in pytorch

username: Elidor
cooked: <p>Hi everyone,</p>
<p>I have been reading articles on parse Biaffine for several days, but I just can’t understand how the biaffine module should be implemented. My parser uses an affine MLP and I just can’t find how to change it to Biaffine.</p>
<p>The structure of my parser is as follows:</p>
<pre><code class="lang-auto">Parser(
  (dropout): Dropout(p=0.6, inplace=False)
  (word_embedding): Embedding(381, 100, padding_idx=0)
  (tag_embedding): Embedding(17, 40, padding_idx=0)
  (bilstm): LSTM(908, 600, num_layers=3, batch_first=True, dropout=0.3, bidirectional=True)
  (bilstm_to_hidden1): Linear(in_features=1200, out_features=500, bias=True)
  (hidden1_to_hidden2): Linear(in_features=500, out_features=150, bias=True)
  (hidden2_to_pos): Linear(in_features=150, out_features=45, bias=True)
  (hidden2_to_dep): Linear(in_features=150, out_features=41, bias=True)
)
</code></pre>
<p>where <code>Embedding = nn.Embedding</code> and <code>Linear = nn.Linear</code>.<br>
Now, what I need to change to implement BiAffine layer?<br>
I think that this two layers:</p>
<pre><code class="lang-auto">self.bilstm_to_hidden1 = nn.Linear(...)
self.hidden1_to_hidden2 = nn.Linear(...)
</code></pre>
<p>are simply MLPs in which there is only one Linear layer.<br>
So I think the right thing to do is replace</p>
<pre><code class="lang-auto">self.hidden2_to_pos = nn.Linear(...)
self.hidden2_to_dep = nn.Linear(...)
</code></pre>
<p>with something a <code>BiAffine</code> module. What I found as a ““standard”” implementation in pytorch is the following:</p>
<pre><code class="lang-auto">class BiAffine(nn.Module):
    """Biaffine attention layer."""
    def __init__(self, input_dim, output_dim):
        super(BiAffine, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.U = nn.Parameter(torch.FloatTensor(output_dim, input_dim, input_dim))
        nn.init.xavier_uniform(self.U)

    def forward(self, Rh, Rd):
        Rh = Rh.unsqueeze(1)
        Rd = Rd.unsqueeze(1)
        S = Rh @ self.U @ Rd.transpose(-1, -2)
        return S.squeeze(1)
</code></pre>
<p>but it doesn’t seem to work.</p>
<p>What am I doing wrong?<br>
Do you have any ideas / suggestions?<br>
Do you know if there is any code / module that I can use as a baseline to implement this module?</p>
<p>I no longer know where to turn.<br>
Thanks a lot to everyone.</p>
Dynamic data iterator value change during training

username: Hasan_Sait_ARSLAN1
cooked: <p>Hi!</p>
<p>My Data Loader :</p>
<pre><code class="lang-auto">LoadData(Dataset):
    def __init__(self, ..., ...., ...):
        self.ns = (640, 640)
        .....
        .....

   def __getitem__(index):
       img = resize(img[index], self.ns)
       .....
       .....
       return img

   def set_size(self, ns):
       self.ns = ns
</code></pre>
<p>And the training loop</p>
<pre><code class="lang-auto">for img in dataiterator:
    forward(img)
    backward()
    datatiterator.dataset.set_size(new_ns)
</code></pre>
<p>I would like to resize images dynamically after each iteration. When I do it as above, it does not work. Is there any simple trick to do that? Please help.</p>
<p>Best,</p>

username: ptrblck
cooked: <p>You won’t be able to manipulate the underlying <code>.dataset</code> through the <code>DataLoader</code>, if you are using multiple workers during the epoch, since each worker will use a separate copy of the <code>Dataset</code>.<br>
You could manipulate the <code>.dataset</code> after each epoch and before the start of a new one (if <code>persistent_workers=False</code>) or you could iterate the <code>Dataset</code> and create the batches manually.<br>
Alternatively, you could also try to forward specific arguments to the <code>__getitem__</code> through a custom <code>sampler</code> to switch between different behaviors.</p>

username: Hasan_Sait_ARSLAN1
cooked: <p>Thank you for your answer. Let me see how I can create a custom sampler and control the image size through sending argument to getitem.</p>
RCNN object detection

username: AlexeyG
cooked: <p>Hello<br>
I am beginner and I try to learn RCNN from TorchVision. I have 8 classes of objects, and i have wrote dataset class with this def__getitem  and here labels is numpy array with classes for boxes on image.(for example ([3, 5]))</p>
<p>def <strong>getitem</strong>(self, index):<br>
path_image = self.paths[index]<br>
image, boxes , labels, area = self.load_image_and_boxes(index)<br>
target = {}<br>
target[‘boxes’] = boxes<br>
target[‘labels’] = labels<br>
target[‘image_id’] = torch.tensor([index])<br>
target[‘area’] = area<br>
if self.transforms:<br>
sample = self.transforms(**{<br>
‘image’: image,<br>
‘bboxes’: target[‘boxes’],<br>
‘labels’: labels<br>
})<br>
image = sample[“image”]<br>
target[‘bboxes’] = torch.as_tensor(sample[‘bboxes’], dtype=torch.float32)<br>
target[‘bboxes’] =  target[‘bboxes’].reshape(-1, 4)<br>
target[“boxes”] = target[“bboxes”]<br>
del target[‘bboxes’]<br>
else:<br>
target[‘boxes’] = torch.as_tensor(target[‘boxes’], dtype=torch.float32)<br>
target[‘boxes’] =  target[‘boxes’].reshape(-1, 4)<br>
image = np.transpose(image,(2,0,1))<br>
image = torch.from_numpy(image)<br>
return image, target, path_image</p>
<p>After lerniing  RCNN return outputs, with 8 items. This is one of this<br>
outputs[7]</p>
<p>{‘boxes’: tensor([[  0.0000,  64.0552, 500.5320, 477.4697],<br>
[ 22.2902, 121.1948, 360.7867, 475.3104],<br>
[  6.6014, 123.5036, 282.5276, 460.8509],<br>
[206.0831,  77.8670, 505.2312, 443.2711],<br>
[ 10.5778,  97.3200, 489.6339, 433.8982],<br>
[ 16.5061,  66.0151, 495.6974, 367.8644],<br>
[  0.0000, 264.9781, 240.3872, 496.7976],<br>
[141.8789,  51.5878, 508.8718, 501.8432],<br>
[244.2009,  45.8100, 512.0000, 479.3023],<br>
[243.4653,  92.4951, 512.0000, 474.3302],<br>
[287.1657, 273.5756, 483.0078, 463.3998],<br>
[ 14.1116, 115.1430, 275.1363, 383.5063],<br>
[  0.0000, 118.9819, 428.0742, 509.8760],<br>
[336.0939, 133.6007, 501.2955, 395.3423],<br>
[331.1056, 110.4328, 497.5609, 425.7774],<br>
[259.7857, 140.4948, 505.6891, 472.5172],<br>
[ 36.9824, 208.6160, 237.9554, 512.0000],<br>
[ 19.1136, 186.5290, 151.8781, 499.3608],<br>
[213.0657, 102.6356, 474.4019, 215.7374],<br>
[  0.0000, 266.3952, 198.0168, 487.8173],<br>
[ 23.2956, 125.9192, 169.1113, 444.2588],<br>
[ 64.2475,  91.6519, 478.7527, 428.1402],<br>
[330.7071, 115.4189, 491.4273, 420.2543],<br>
[  0.0000,  96.2847, 219.0983, 512.0000],<br>
[  0.0000, 478.2526, 189.6664, 512.0000],<br>
[ 11.6199, 131.2440, 319.1119, 458.8158]], grad_fn=),<br>
‘labels’: tensor([2, 7, 2, 7, 6, 7, 2, 4, 2, 1, 4, 6, 4, 1, 7, 4, 4, 2, 4, 7, 6, 1, 2, 7,<br>
3, 1]),<br>
‘scores’: tensor([0.4158, 0.2980, 0.2887, 0.2255, 0.1818, 0.1811, 0.1805, 0.1661, 0.1560,<br>
0.1539, 0.1526, 0.1276, 0.1204, 0.1092, 0.0959, 0.0930, 0.0845, 0.0823,<br>
0.0796, 0.0788, 0.0780, 0.0750, 0.0691, 0.0663, 0.0524, 0.0515],<br>
grad_fn=)}</p>
<p>So, i thing i made wrong in __getitem, and in outputs in each item should have boxes belongs only one class. But i dont know how to rewrite it properly</p>
Backpropagation with for loops

username: snowe
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a conceptual question regarding <code>for loops</code> in the <code>forward method </code> of a Convolutional Neural Network and the corresponding <code>backpropagation</code>.</p>
<p>I read that when you want to loop over modules in the <code>forward method</code> you can make use of the <code>nn.ModuleList</code> class. But what if I would like to use a <code>for loop</code> for something else? How does this influence the backpropagation? Say I would like to loop over the channels of an image and process them individually, does that influence the <code>backpropagation</code>?</p>
<p>Any help is very much appreciated!</p>
<p>All the best<br>
snowe</p>

username: ptrblck
cooked: <p>It “influences” the computation graph in the sense that each operation would be added to the graph and the backpropagation should just work. It shouldn’t be necessary to take extra steps and could just use for loops etc. without a problem.</p>

username: snowe
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I was worrying that it might only use the last loop or so or overwrite them but nice to hear otherwise!</p>
Equivelent to keras NonNeg weight constraint in pytorch?

username: Richard_S
cooked: <p>Keras has an option that can cause the weights of the model to be non negative :</p>
<pre><code class="lang-auto">tf.keras.constraints.NonNeg()
</code></pre>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://keras.io/img/favicon.ico" class="site-icon" width="64" height="64">
      <a href="https://keras.io/api/layers/constraints/" target="_blank" rel="noopener nofollow ugc">keras.io</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/239;"><img src="https://keras.io/img/logo-k-keras-wb.png" class="thumbnail" width="690" height="239"></div>

<h3><a href="https://keras.io/api/layers/constraints/" target="_blank" rel="noopener nofollow ugc">Keras documentation: Layer weight constraints</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>What is the equivelent of this in pytorch? lets say my model is the one below, how should i change it to force the weights to be non negative ?</p>
<pre><code>class LogisticRegression(torch.nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LogisticRegression, self).__init__()
        self.linear = torch.nn.Linear(input_dim, output_dim)

    def forward(self, x):
        outputs = self.linear(x)
        return outputs</code></pre>
What does the at::gt_out function do?

username: SamuelMosesA
cooked: <p>I came accross the function <strong>at::gt_out</strong>  in some code but couldn’t find the details of it in the documentation. Can someone tell me what it does?</p>
<p>There are only doxygen warnings in the c++ documentation site. Is it an error or has it been like this always?</p>

username: ptrblck
cooked: <p><code>at::gt_out</code> is the “greater than” operation. I don’t think you would need to use it directly, as the dispatcher should call into it based on the usage of <code>at::gt</code>.</p>

username: SamuelMosesA
cooked: <p>Thanks a lot for the help.<br>
Do all the c++ and python api functions share the same name? The c++ documentation does not have any description for the functions. So I’m not able to use it.</p>

username: ptrblck
cooked: <p>They should use the same name as the Python API, but some C++ operations might still be missing.<br>
You can search for the functions <a href="https://pytorch.org/cppdocs/index.html">here</a> and would find at least the entry for <code>gt</code>. Unfortunately the docs don’t seem to be complete yet.</p>
Models with different paramater amount have simililar CPU usage

username: Sining_Sun
cooked: <p>Hi all,</p>
<p>Recently, I deployed my models to android using libtorch (1.6). Models are quantized and exported using torch.jit.script. When I do inference using C++, I find that model with 0.8M parameters has the same cpu usage as the model with 2M parameters.  I have set thread number to 1 using the flowing code:</p>
<p>at::set_num_threads(1);<br>
at::set_num_interop_threads(1);</p>
<p>How can I know if I’m using qnnpack when I do inference?</p>
Converting Keras implementation to Pytorch results in worse results

username: Sara_Al-Rawi
cooked: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/converting-multi-output-model-from-keras-to-pytorch-results-in-worse-results/102443">Converting Multi-output model from Keras to Pytorch results in worse results</a>:</p>

username: ptrblck
cooked: <p>You’ve unfortunately deleted the old topic.<br>
Would you like to repost the original question here? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Sara_Al-Rawi
cooked: <p>I found the answer in another post. My problem was with loading the data. I was basically training on a subset of the data.</p>
Video Dataset Loading for Everyone

username: raivokoot
cooked: <p>I’ve created a standalone implementation with comprehensive documentation of a generic Video Dataset PyTorch class for data loading. It is very user-friendly, fast, and effective. Torchvision’s transforms pair very well with it for video preprocessing and augmentation as well.</p>
<p>When I was recently looking for a way to load video datasets in PyTorch, I couldn’t find anything that was not slow or smelted into a messy codebase. So, to anyone looking for loading video data, I strongly recommend checking this <a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" rel="noopener nofollow ugc">Github Repository</a> out.</p>
<p>With my two 3090s, it reduced excess data loading time from 800ms to 20ms, making my arbitrary model train 3x faster than when using torchvision’s native video loaders and giving me much better frame selection from videos.</p>
Can we do autoregressive using pad_packed_sequence?

username: XezXey
cooked: <p>I’m curious about if we want to do the autoregressive manner. Is it possible to do with implemented using pad_packed_sequence and pack_padded_sequence input to some recurrent network? Because we need to wait for the output of timestep t. Given this scenario we cannot gather all prediction and packed them. Then what should I do with this. Is there any way to do this without looping to each time step? Also If i want to do it in batch processing is it possible?</p>
<p>Thank you</p>
How to load a large video dataset?

username: dersu
cooked: <p>Hello everyone,</p>
<p>I have a dataset of multiple videos, consisting of ~40,000 frames. I am trying to feed every video as one batch (batch_size=1) to a recurrent network for a regression task. In the custom dataloader function, I read all the preprocessed frames of one video at once, and expectedly, GPU memory cannot handle it and besides, data loading can take a long time.</p>
<ol>
<li>Any suggestions on how to load this type of datasets into GPU, without having memory and time problems?</li>
</ol>
<p>I thought about feeding a part of the frames and saving the hidden outputs, then restart the network with the hidden outputs as hidden inputs now and a new set of the frames. For example if the GPU is capable of processing 5k frames at a time, a 40k frame long network will be started for 8 times, and at the end the prediction will be generated.</p>
<ol start="2">
<li>I’m confused about how I can do this, regarding backprop and again, dataloader. Is there a way like this?</li>
</ol>
<p>Thanks in advance for answering one or both of my questions! (first has higher priority <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">)</p>

username: raivokoot
cooked: <p>This might be helpful to you <a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" rel="noopener nofollow ugc">Video Dataset Loading in PyTorch</a>.</p>
<p>Its a custom dataset implementation that goes very well with any video dataset and is very fast. It also allows you to specify the amount of frames to load from each video, taking the frames evenly from start to finish of a video. This way you might be able to make your loaded frame tensor a bit sparser in terms of the amount of frames it contains from the video, and allow yourself to not overrun memory</p>
Error installing dependencies for pytorch geometric

username: agt
cooked: <p>I am trying to install pytorch-geometric in a conda environment on 64-bit Windows 10, using python 3.7.9.</p>
<p>When I try to install one of the dependencies <code>pip install torch-sparse</code>, I get a big error. Same thing for the other dependencies, torch-scatter and torch-cluster. How can I install them?</p>
<p>The error: <a href="https://dpaste.org/O5Uq" rel="noopener nofollow ugc">https://dpaste.org/O5Uq</a></p>

username: ptrblck
cooked: <p>I would try to make sure your current setup is able to build custom PyTorch extensions e.g. by building the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">tutorial extension</a>.<br>
The issue is also tracked <a href="https://github.com/rusty1s/pytorch_geometric/issues/1826">here</a>.</p>
How to measure two classes segmentation results?

username: hao_guo
cooked: <p>I am working on the segmentation task with 2 classes :vessel and background.<br>
It seems to have two options.<br>
1 .output is 1 channel mask,grayscale image,using binary cross entropy as loss function.I am confused with how the metric like FP TP caculated,should I set the threshold to turn the grayscale picture to black and white picture then caculate the metric?<br>
2 .output is 2 channel mask,using cross entropy, get the mask by output.argmax(dim=1),and it looks like easier to caculate the FP and TP<br>
Is there a best option between them?</p>

username: ptrblck
cooked: <p>Both options are valid and your explanations are correct.</p>
<p>The difference is in treating the segmentation use case as a binary segmentation (first option) with <code>nn.BCEWithLogitsLoss</code> and a single output channel or as a 2-class multi-class segmentation (second option) with two output channels.<br>
From what I saw in other posts the first option is often preferred, but the second one should also work.<br>
<a class="mention" href="/u/kfrank">@KFrank</a> is an expert in these topics and might want to add something. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
ONNX opset version 9, the Upsample operator and ONNX opset version 11

username: Ilias_machinery
cooked: <p><strong>Situation</strong>: I am trying to implement a Convolutional Text Binarizer, a CNN which accepts as inputs RGB images with superimposed text and returns as outputs a map F which is (after some more processing) its corresponding black and white binary image, with the text in black and the backround in white. After the training and validation of the network I am trying to export it to ONNX so I can use it with the openCV’s dnn.module.</p>
<p><strong>Problem <span class="hashtag">#1</span></strong>: In my architecture I use one Upsample layer, which results in this warning after the export to ONNX finishes:</p>
<pre><code class="lang-auto">UserWarning: You are trying to export the model with onnx:Upsample for ONNX opset version 9. This operator might cause results to not match the expected results by PyTorch.
ONNX's Upsample/Resize operator did not match Pytorch's Interpolation until opset 11. Attributes to determine how to transform the input were added in onnx:Resize in opset 11 to support Pytorch's behavior (like coordinate_transformation_mode and nearest_mode).
We recommend using opset 11 and above for models using this operator. 
  "" + str(_export_onnx_opset_version) + ". "
</code></pre>
<p>I tried to use my network anyway, without re-exporting it to opset version 11 and it works pretty good, there is some room for improvments but it’s a satisfacting first attemp. I use the blob method from openCV to perform a forward pass of an image with text and I do have its corresponding binary as output. But still that warning itches my brain a bit.</p>
<p><strong>Problem 2</strong>: Naturally the next thing I did, was to try to re-export the model using opset 11 and seemed to workout fine, no UserWarning was raised. Then i tried the exact same thing as before and perform a forward pass for the same image and see if there is any obvious differences between the two versions, but then this error was raised:</p>
<pre><code class="lang-auto">error: OpenCV(4.3.0) C:\projects\opencv-python\opencv\modules\dnn\src\layers\padding_layer.cpp:41: error: (-215:Assertion failed) params.has("paddings") in function 'cv::dnn::PaddingLayerImpl::PaddingLayerImpl'
</code></pre>
<p><strong>My two main questions</strong>:</p>
<ol>
<li>Is the miss-match between the way that PyTorch and ONNX handles the Upsample operator<br>
significant enough to cause me trouble, or can I just ignore the warning?</li>
<li>Why openCV’s dnn.module is loading the opset 9 version model whitout any trouble but it raises an error for the exact same model but in opset 11 version?</li>
</ol>
<p>PS: I know thats not a forum for openCV users, so I don’t expect any answers for my second question, but if by any change somebody happens to think of a reason , I 'd really appreciate to know.</p>
Where is the `grad` defined?

username: HaFred
cooked: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/how-can-i-get-the-gradients-of-the-weights-of-each-layer/28502/2">How can I get the gradients of the weights of each layer?</a>:</p>
<p>Hi ptrblck <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I was trying to find the attribute definition of <code>grad</code> in both <code>nn.Linear</code> and <code>nn.Conv2d</code> class, but all I find is the PairwiseDistance function defined in the <code>PariwiseDistance.py</code> as below:</p>
<pre><code class="lang-auto">class PairwiseDistance(Module):

    def __init__(self, p):
        super(PairwiseDistance, self).__init__()
        assert p % 1 == 0
        self.gradInput = []
        self.diff = torch.Tensor()
        self.norm = p

        self.outExpand = None
        self.grad = None
        self.ones = None
...
</code></pre>
<p>I wonder is it where the <code>grad</code> in the <code>model.my_layer.weight.grad</code> defined? But it seems to me neither of them inherits <code>PairwiseDistance</code>? Thank you.</p>

username: ptrblck
cooked: <p>Modules don’t have a <code>grad</code> definition, but parameters or tensors.<br>
You can find the attribute/property definition <a href="https://github.com/pytorch/pytorch/blob/0652d755d3a5d6c21a695ff9b57d8be972496a4f/torch/tensor.py#L968-L988">here</a>. <code>tensor.grad</code> will be populated e.g. after a <code>backward</code> operation or <code>torch.autograd.grad</code> call and will give you the gradient of X w.r.t. this tensor.</p>
Extract Weight Vector Estimate

username: marathan
cooked: <p>Hi I am pretty new to PyTorch and neural networks in general and I am wondering about the weight vector estimate of linear networks. Suppose I have built a very simple model with 3 input nodes and 2 output nodes (no hidden layers) to use on a 2D dateset I generated  (I know what the weight vector should look like). I only have two different classes in this set and I would like to extract the weight vector (linear decision boundary) from my network once training is done. I was wondering what is the best way to do that.<br>
So far I have tried something like this :</p>
<p><code>weights_out = list(model.fc1.weight.data)</code></p>
<p>which gives me a pair of 2D vectors, one for each output node I presume.</p>
<p>Thanks in advance for any help.</p>
<p>My network actually looks like this :</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self,dimensions,model_type):
        super(Net,self).__init__()
        self.fc1 = nn.Linear(3,2)
    def forward(self, x):
        x = self.fc1(x)        
        return x
</code></pre>
Shift elements of a tensor (and not roll) independently in different rows

username: PyScorch
cooked: <p>Suppose there is an array A = tensor([[0.4869, 0.5144, 0.9086, 0.6139],<br>
[0.5103, 0.8270, 0.4832, 0.8980],<br>
[0.5234, 0.1135, 0.1037, 0.7451]])</p>
<p>And I want to shift the elements in each row and replace the shifted places by zeros, depending on another tensor t = tensor([0, 1, 3])</p>
<p>The output should be like out = tensor([[0.4869, 0.5144, 0.9086, 0.6139],<br>
[0, 0.5103, 0.8270, 0.4832],<br>
[0, 0, 0, 0.5234]])</p>
<p>I have already tried an implementation that uses the torch.gather function but that operation seems to consume a lot of memory and it runs into memory overflow when dealing with huge tensors.</p>
RNN loop uses previous time step's hidden state to predict current output in the PyTorch tutorial

username: devpramod
cooked: <pre><code>class RNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
    super(RNN, self).__init__()

    self.hidden_size = hidden_size

    self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
    self.i2o = nn.Linear(input_size + hidden_size, output_size)
    self.softmax = nn.LogSoftmax(dim=1)

def forward(self, input, hidden):
    combined = torch.cat((input, hidden), 1)
    hidden = self.i2h(combined)
    output = self.i2o(combined)
    output = self.softmax(output)
    return output, hidden
</code></pre>
<p>Hi, I’ve just begun using Pytorch and was going through the RNN example in <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial</a>. As per my understanding, the current time step’s output is predicted using the current hidden state. But here, the previous time step’s hidden state seems to be used. Can I get an explanation? Thanks.</p>

username: googlebot
cooked: <p>It may be a bit confusing, but it is shifted. The reason is, before the first step, you already have one <em>a priori</em> hidden state, and first step looke like rnn(input[0], prior_hidden) -&gt;hidden[0]. This way, tensor sizes match in time dimension.</p>

username: Abhilash_Srivastava
cooked: <p>There are several variables used in a typical RNN.</p>
<ol>
<li>Current hidden state: <code>h_t</code>
</li>
<li>Last hidden state: <code>h_t-1</code>
</li>
<li>Current input: <code>x_t</code>
</li>
<li>Current output: <code>o_t</code>
</li>
</ol>
<p>In this example, <code>h_t</code> is computed using <code>h_t-1</code> and <code>x_t</code>.  <code>o_t</code> is computed using the <code>h_t-1</code> and <code>x_t</code>.</p>

username: devpramod
cooked: <pre><code>def rnn_cell_forward(xt, a_prev, parameters):

  a_next = np.tanh(np.dot(Waa,a_prev)+np.dot(Wax,xt)+ba)
  yt_pred = softmax(np.dot(Wya,a_next)+by)   

  cache = (a_next, a_prev, xt, parameters)
  return a_next, yt_pred, cache
</code></pre>
<p>These are the forward pass equations I’m most familiar with. At the very first time step, the initial hidden state a_prev (h[0] initialized to 0) is never used to make output predictions. Whereas in the above PyTorch tutorial, a_prev gets to make the first output prediction. Are these changes in architecture/implementation? sorry, I’m still sort of confused.</p>

username: Abhilash_Srivastava
cooked: <p>The implementation you posted above is different from the implementation in the Pytorch tutorial. I think your understanding is correct in both cases.</p>

username: devpramod
cooked: <p>Thanks!! I’ll continue to think it’s just a difference in implementation.</p>
One of the variables needed for gradient computation has been modified by an inplace-operation

username: mhong94
cooked: <p>I am running into this error ‘One of the variables needed for gradient computation has been modified by an inplace-operation’ when back propagating on my loss function.  My loss function is a simple</p>
<pre><code>def loss_fn(x):

    total_loss = []
    for i in range(k)
        loss = ...
        total_loss.append(loss)

    final_loss = torch.sum(torch.stack(total_loss))
</code></pre>
<p>I can call loss.backward() at any given iteration of the loop, but cannot call final_loss.backward() indicating the for loop is breaking it.  I do not see an inplace operation, and I have tried solutions from other questions asked.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is quite surprising indeed.<br>
Can you share a small code sample that reproduces this?</p>

username: mhong94
cooked: <p>Hi,</p>
<p>I found my error, I had a variable assignment in on of my helper functions for calculating the loss.  Thanks!</p>
How to set gradients of some elements inside a convolution kernel always equal to 0?

username: TANKUN_LI
cooked: <p>Hello everyone,</p>
<p>I am now trying to implement a custom convolution kernel.<br>
Let me take dilated convolution as example. (I know there are dilated conv in Pytorch. Just an example to state my question).<br>
The kernel shold be something like:</p>
<pre><code class="lang-auto">[a, 0, b, 0, c ],
[0, 0, 0, 0, 0 ],
[d, 0, e, 0, f  ],
[0, 0, 0, 0, 0 ],
[g, 0, h, 0, i  ],
</code></pre>
<p>Browsing others discusions, I know the way to construct such kernel, and then use <code>F.conv2d</code>.</p>
<p>But when calculating the gradients, those <code>0</code> values in the kernel still have gradients. I just want them to be ‘0’ all the time and never update. Only <code>[a, b, c, d, e, f, g, h, i]</code> can be updated.</p>
<p>How can I do this?</p>
<p>I know one way to do it is that manually set <code>weights.grad[index of </code>0<code>] = 0</code>. But suppose the model has a lot of kernels of this kind, it will not be efficient.</p>
<p>Regards,<br>
LI</p>

username: ptrblck
cooked: <p>I think zeroing out the gradients is a valid approach and you could register hooks to perform this operation, if this would be more convenient, via <code>param.register_hook</code>.<br>
Alternatively, you could also create a trainable <code>nn.Parameter</code> containing the values of <code>[a-i]</code> and recreate the kernel in each iteration e.g. by using <code>torch.cat</code>, <code>torch.scatter</code>, or <code>torch.gather</code> (depending on the actual shape). However, I would prefer the first approach as it sounds simpler and cleaner.</p>

username: TANKUN_LI
cooked: <p>That makes sense. Thx!</p>
.item() gives different value than the tensor itself

username: rouniuyizu
cooked: <p>Not sure if that’s mixed-precision related but here is what happened:</p>
<p>a = torch.tensor([1.1])<br>
a.item() &lt;-- 1.100000023841858<br>
a &lt;= 1.1  &lt;-- True<br>
a.item() &lt;= 1.1 &lt;-- False</p>
<p>I’m a bit confused here…</p>
<p>my env ==&gt;<br>
pytorch = 1.6.0<br>
python = 3.7.6</p>

username: ptrblck
cooked: <p>The outputs differ in the used print precision and you will see the same output values using:</p>
<pre><code class="lang-python">a = torch.tensor([1.1])
print(a.item())
&gt; 
1.100000023841858
print(a)
&gt; tensor([1.1000])

torch.set_printoptions(precision=15)
print(a)
&gt; tensor([1.100000023841858])
</code></pre>
<p>so it’s unrelated to mixed-precision training.</p>
<p>I think the comparison using the tensor should yield <code>True</code>, as the right-hand side would be transformed to a tensor first and would thus be equal to the left-hand side.</p>
<p>The second comparison uses Python directly, so I guess it might be using <code>float64</code> values (or even a higher precision?) and might be able to represent the <code>1.1</code> value in a different way, such that the comparison fails.</p>

username: rouniuyizu
cooked: <p>Thank you for the comment.</p>
<p>I agree on the print thing, while why tensor internally stores the value differently as it should be still confuses me. As you mentioned in your code, when print the tensor <em>a</em>, it shows tensor([1.100000023841858]) instead of tensor([1.1]).</p>
<p>Also I’m curious why 1.1 gets stored as exactly 1.100000023841858 as it happens in your environment as well.</p>

username: ptrblck
cooked: <p>That’s due to the limited floating point representation using 32bits.<br>
The <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">Single-precision floating-point format</a> Wikipedia article explains it quite well.</p>

username: rouniuyizu
cooked: <p>Thank you! All clear now.</p>
How to code attention map equation in Pytorch

username: rperera
cooked: <p>I am trying to code a model which works with a feature map to end up with an attention map. I think I got some progress in coding it but I am stuck in getting the final attention (attn) calculation.</p>
<p>Some definitions:</p>
<pre><code class="lang-auto">Xk is the kth channel of the input feature maps X after going through a ResNet18. X shape is C x H x W
W is the weights of dilated convolution blocks after X is sent through 2 of them.
M is output of compressing features by computing the sum of all channels
S is generated by applying a spatial softmax layer that performs the softmax operation over all feature points in the aggregated map M
XT is the original convolutional feature maps with the shape of c × hw
</code></pre>
<p>I am stuck trying to figure out how to code the following equation :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08a1e620f20f586a46545de87939f955ba2719e2.png" alt="image" data-base62-sha1="1emGzsjuc43ByDVxT5vA8orMgLg" width="619" height="120"></p>
<p>The rest of the equations are:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/dfc295bdecc640f8593b928de5fb857eb0ee0ce5.png" alt="image" data-base62-sha1="vVtn4JiFfcwkRwMfaKpjIWZYGBn" width="233" height="45"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/6832ec050c6c9c4048e7cf0558c0def8cb22cdd7.png" alt="image" data-base62-sha1="eRMKUMaKrWmBGVxgvuEAuwnyoU7" width="505" height="110"></p>
<p>The code I have upto now</p>
<pre><code class="lang-auto">def forward(self, x):
        # x.shape - torch.Size([1, 3, 500, 500])
        X = self.resnet18(x) # torch.Size([1, 512, 32, 32])
        W = self.dialation_network(X) # torch.Size([1, 512, 32, 32])
        M = torch.einsum('bchw,bchw-&gt;bhw', W, X) # torch.Size([1, 32, 32])
        S = F.softmax(M.reshape(1, -1), dim=1) #torch.Size([1, 1024])
        XT = Xk.reshape(1, 512, 32*32) # torch.Size([1, 512, 1024])
        attn = # TODO
        
        return out
</code></pre>
<p>I would like to know whether,</p>
<ol>
<li>What I have done upto now is correct</li>
<li>How to get the final equation in pytorch</li>
</ol>
<p>Any help is appreciated!</p>
<p>TIA!</p>
Same transform for a paired image?

username: Johnson_Mark
cooked: <p>Hello all, I have a paired image such as <code>img1, img2</code>. I want to apply the same transform during training for these images as</p>
<pre><code class="lang-auto">transform = transforms.Compose([
           transforms.RandomHorizontalFlip(),
           transforms.ToTensor(),
]
img1 = transform(img1)
img2 = transform(img2)
</code></pre>
<p>Is it possible to do it in the data loader of pytorch?</p>

username: Kushaj
cooked: <p>You can concatenate the images along the channel dim and then apply the transform. You can check <a href="https://kornia.readthedocs.io/" rel="noopener nofollow ugc">kornia</a>, it does exactly what you want.</p>

username: ptrblck
cooked: <p>Alternatively, you could also use the functional API from <code>torchvision</code> as given in <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">this example</a>.</p>

username: Johnson_Mark
cooked: <p>Does the approach work on distributed training?</p>

username: ptrblck
cooked: <p>If you are using multiple processes, each process would still apply the same transformations on the data and target. The transformations will be different for each sample (in a single process and in a distributed setup).<br>
Could you explain a bit more, what your use case is and what you would like to do?</p>

username: Johnson_Mark
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. We use 16 workers and the current dataloader is</p>
<pre><code class="lang-auto"> train_transform = transforms.Compose([
           transforms.RandomHorizontalFlip(),
           transforms.ToTensor(),
           transforms.Normalize(mean = RGB_MEAN,
                            std = RGB_STD),])
    dataset_train = FaceDataset(DATA_ROOT, RECORD_DIR, train_transform)
    train_sampler = torch.utils.data.distributed.DistributedSampler(dataset_train)
    train_loader = torch.utils.data.DataLoader(dataset_train, batch_size=batch_size, shuffle = (train_sampler is None), num_workers=workers, pin_memory=True, sampler=train_sampler, drop_last=True)
    SAMPLE_NUMS = dataset_train.get_sample_num_of_each_class()
    NUM_CLASS = len(train_loader.dataset.classes)
 
</code></pre>
<p>And the current dataloader is refered from <a href="https://github.com/HuangYG123/CurricularFace/blob/8b2f47318117995aa05490c05b455b113489917e/dataset/datasets.py#L92" rel="noopener nofollow ugc">https://github.com/HuangYG123/CurricularFace/blob/8b2f47318117995aa05490c05b455b113489917e/dataset/datasets.py#L92</a></p>
<pre><code class="lang-auto">path, target = self.imgs[index]
sample = Image.open(path)
sample = sample.convert("RGB")
if self.transform is not None:
     sample = self.transform(sample)
</code></pre>
<p>Is it fine to replace your customized transformation for the above code?</p>

username: ptrblck
cooked: <p>Yes, it should be alright.<br>
Note that using multiple workers is not a distributed setup, which we define as using multiple devices or nodes.</p>
Distributed data sampler for training iterable datasets on TPUs

username: Rabeeh_Karimi
cooked: <p>Hi<br>
I have an iterable dataset and I need to define distributed datasampler for it to train efficiently on TPUs, here is the example distributed sampler for TPUs in case of non-iterable datasets, could you assist me please with providing an example with an iterable dataset (like tf.data.Dataset) which gets convert to an iterable datasets in pytorch and a distributed sampler which can be used on TPUs. thank you.</p>
<pre><code class="lang-auto">def get_tpu_sampler(dataset: torch.utils.data.dataset.Dataset):
    if xm.xrt_world_size() &lt;= 1:
        return RandomSampler(dataset)
    return DistributedSampler(dataset, num_replicas=xm.xrt_world_size(), rank=xm.get_ordinal())
</code></pre>

username: mrshenli
cooked: <p>cc <a class="mention" href="/u/ailzhang">@ailzhang</a> for XLA<br>
cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for data loader</p>

username: ailzhang
cooked: <p>Hi <a class="mention" href="/u/rabeeh_karimi">@Rabeeh_Karimi</a>, for all pytorch/xla(TPU) related questions, please open an issue in pytorch/xla github repo instead. <a href="https://github.com/pytorch/xla" rel="noopener nofollow ugc">https://github.com/pytorch/xla</a> Thanks!</p>