akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
BERT positional encoding

username: kevs
cooked: <p>How is the positional encoding for the BERT model implemented with an embedding layer? As I understand sin and cos waves are used to return information on what position a certain word has in a sentence - Is this what the lookup in weight is doing?</p>
<pre><code class="lang-auto">model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased')

model.embeddings.position_embeddings # this will return the layer I am trying to figure out

model.embeddings.position_embeddings.weight.shape # what do these dimensions mean?
</code></pre>
<p>The shape of the embedding is <code>torch.Size([512, 768])</code>. I would guess that 768 is the embedding dimension, but what is 512? Is that the max length of the sentence?</p>
Stop backward() at some intermediate tensor

username: lucasegidio1
cooked: <p>Hi everyone,</p>
<p>I’m implementing a problem in which I have to calculate gradients with respect to intermediate tensors, use these gradients in further calculations to get a final value and then backpropagate again from this final value. I know it sounds confusing so I made a dummy example for what I’m doing:</p>
<pre><code class="lang-auto">import torch

# initialize tensor a and do some dummy operations
a = torch.tensor(2.4, requires_grad=True)
b = a * 7
c = 4 * b ** 2
c.retain_grad()  # I want to use a gradient with respect to this variable

# more dummy operations
d = 2 * torch.log(c) * a
e = d ** 1.2

# backpropagating to get the intermediate grad
e.backward(retain_graph=True)
f = c.grad  # I got the gradient I wanted f = de/dc
print(f)

g = e * d * f * f  # using the f in more dummy calculation

# Finally backpropagating to get dg/da
g.backward()
print(a.grad)

</code></pre>
<p>The thing is: when I call e.backward(retain_graph=True) gradients are computed all the way back to the tensor a. In this specific problem, it’s no big deal but in my original problem It takes too much time for unnecessary computing. Is there any way to stop backward() as soon as I have a gradient on the tensor c?</p>
<p>Thank you for reading all this and I hope someone can help me.</p>

username: lucasegidio1
cooked: <p>I did some research and it seems that</p>
<pre><code class="lang-auto">f, = torch.autograd.grad(e, c, retain_graph=True)
</code></pre>
<p>does the job! Sorry, I’m new in pytorch.</p>

username: Tan
cooked: <p>Note for anybody in the future;</p>
<pre><code class="lang-auto">e.backward(inputs=(c,), retain_graph=True)
</code></pre>
<p>works for cases where c is a Variable/Parameter and populates c.grad with the gradient output</p>
<hr>
<p>This (should) let you do things like;</p>
<pre><code class="lang-auto">loss.backward(inputs=tuple(model.parameters()), retain_graph=True)
</code></pre>
<p>… which lets you backprop a model or two using <code>retrain_graph</code> without autograd doing any unnecessary compute. It looks like autograd will automatically figure out which latents to backprop and skip the rest.</p>
Combining learnt policies (from different leanring algoritms)

username: Kallinteris-Andreas
cooked: <p>Are you guys aware of any techniques of combining learnt policies obtained from using different learning algorithms</p>
<p>Ie image you have a domain and 2, and you apply 2 different learning algorithms (algo A &amp; algo B) and algo A finds some policy trajectories while algo B tents to find other (but not mutually exclusive) is there a way combining them to try to get the best of both worlds</p>
<p>if possible please attach a paper to you apply</p>
Loading pretrained weights for BERT base

username: kevs
cooked: <p>How many parameters are there for the BERT base which has 12 encoder blocks? I figure it’s 192 but I am not sure. Is there a convenient way to load these parameters without depending on the actual BERT model? I had a look a huggingface transformers library but I found it very complicated and needed to download a lot of dependencies. Ideally, I just want to download the pretrained weights as numpy arrays.</p>
<p>Another question related to this: When training the BERT model with the pretrained weights - do we change all parameters in the network or only the parameters inside FFNN which come after the transformer encoder blocks?</p>
Normalising images in Cityscapes

username: 887574002
cooked: <p>Hi, I am working on Cityscapes dataset and I looked for some <code>mean</code> and <code>std</code> for images in <code>Cityscapes</code> dataset in order to normalise them. But when I tried those values for normalising the images and I checked the <code>mean</code> and <code>std</code> of images after normalising it was not close to 0, and 1. So I skipped normalisation, since I did not want to use a pertained model but a Unet from scratch.</p>
<p>without normalisation, each epochs take roughly 145 seconds to be trained on images resized to 256x256 and batch_size =32 and learning rate = 10^{-4}. I am wondering the speed of training on this dataset with almost 2600 images for segmentation is normal or it is too high. I increased the num workers to 8 for one gpu and it was the maximum speed for training that I obtained. I also tried the DataPrallel    with 3 GPUs and increased the num of workers to 12 (4 times as number of gpus) but I did not observe significant speed up.</p>
<p>I should add this point for a fixed learning rate the time for training each epoch is about <code>50</code> seconds that I think is reasonable but with a fixed learning rate I have the overfitting problem and the usual techniques for dealing with overfitting problem did not work. Since I do not want to make the model simpler I used a cyclic step learning where I set learning rate at beginning of each cycle as large as <code>10^-1</code> and then I reduce it to <code>10^{-4}</code> during 30-50 epochs based on cousin annealing. I do not use the pytorch    implementation but the implementation based on this paper <code>https://arxiv.org/pdf/1902.03932.pdf</code> that is compatible with model that I am working on it. When I use this learning rate schedule, the loss decreases very smoothly but the speed of training of each epoch increases from <code>50</code> seconds to <code>145</code> seconds. While I expect it gets faster since I increase the learning rate at starting of each cycle. I am wondering if this slow training is because of the lack of normalisation or it is totally natural with this dataset. I do not use any augmentation except resizing and converting to tensor. If normalisation is the issue, can anyone tell me how the images of this dataset are normalised?</p>
<p>Thank you in advance.</p>
What is torch.fx?

username: Andrea_Rosasco
cooked: <p>I just read about some of the new features of PyTorch 1.8 and noticed the introduction of the torch.fx module. I read the documentation and I kind of get what it does but I have two questions:</p>
<ul>
<li>What are the use cases?</li>
<li>Does it allow to do something that was previously impossible on PyTorch or is it just about improving performances?</li>
<li>Is there a particular area of machine learning where it could be useful?</li>
</ul>
<p>Thank you!</p>

username: nscotto
cooked: <aside class="quote no-group" data-username="Andrea_Rosasco" data-post="1" data-topic="118595">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrea_rosasco/48/14939_2.png" class="avatar"> Andrea_Rosasco:</div>
<blockquote>
<p>of the new features of PyTorch 1.8 and</p>
</blockquote>
</aside>
<p>It can be useful when writing a framework that does automatic model modifications, whether it is for improving performance accuracy or just hardware support. That wasn’t possible before.</p>

username: Andrea_Rosasco
cooked: <p>Thank you! Can you link me any GitHub project that exemplifies it well?</p>
Which layers can be used multiple times in one network?

username: PySimon
cooked: <p>Hello,</p>
<p>when I have a network with 5 layers that should have dropout, do I need one separate nn.Dropout instance for each layer or can I just reuse one? (assuming all have the same dropout rate)</p>
<p>In general:<br>
Is there an overview or simple rule how I can decide which layers can be safely reused and which not? I mean, layers as ReLU or Tanh have no internal state, so they can be reused. Layers like Conv2D or Linear have weights, so they should not be reused (except I want weight sharing?)</p>
<p>But with Dropout I am not sure. Does this layer need to remember which connections were removed in the forward() pass in order to do the backward pass correctly?</p>
<p>Sorry if this is a stupid question, I could not find an answer and I want to be sure that my networks are as simple as possible.</p>

username: melike
cooked: <p><a href="https://discuss.pytorch.org/t/using-same-dropout-object-for-multiple-drop-out-layers/39027/6">Here</a> is an advice.</p>
Is it advisable to use the same Dataset class for training and predicting?

username: jakes
cooked: <p>I have recently started using PyTorch and I liked it for its object-oriented style. However, I wonder what’s the best and advised workflow when predicting the model. I wanted to use a custom Dataset class I wrote and which I use for training and validating my model. This class is a map-style dataset, therefore I implement <strong>getitem</strong> method to return image and target:</p>
<pre><code class="lang-auto">class CDiscountDataset:

    def __init__(self, ...):
        ...

    def __getitem__(self, image_id):
        ....
        return (
            torch.tensor(image, dtype=torch.float),
            torch.tensor(target, dtype=torch.long),
       )
</code></pre>
<p>However, when I’m using this class for predicting I don’t have any targets to return. My current workaround is something like</p>
<pre><code class="lang-auto">    def __getitem__(self, image_id):
        ....
        if predict:
            return (
                torch.tensor(image, dtype=torch.float),
                np.nan,
           )
       else:
            return (
                 torch.tensor(image, dtype=torch.float),
                 torch.tensor(target, dtype=torch.long),
           )
</code></pre>
<p>However, I wonder if there’s a better way to do it. And at the same time, as it feels a bit unnatural, I started wondering if it is even advisable to use the same class for training and predicting (it should be, but the clunkiness of my solutions makes me wonder). Of course, I could not return a tuple at all, but only a first element, but this still needs if-else.</p>
Implement faults during training

username: soepblik
cooked: <p>Hi,</p>
<p>I was wondering to make my neural network more robust.</p>
<p>I tried to change the model after the feedforward stage but then I get Runtime errors.<br>
What is the best way to inject faults to a model during training?</p>

username: soepblik
cooked: <p>someone who can help me?</p>
Installing anaconda on mac OS Big Sur

username: 887574002
cooked: <p>Hi I am installing <code>anaconda on macOs Big Sur</code> on a remote server using the terminal version. But when I try to install it I got the following error <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p><code>PREFIX=/home/anaconda3 Anaconda3-2020.11-MacOSX-x86_64.sh: line 377: md5: command not found WARNING: md5sum mismatch of tar archive expected: ffe6c79627482cbbc073b4d7509bacb2</code></p>
<p>When I was installing it I got this error that the archive is for <code>mac OSX</code> not <code>max OS Big Sur</code>. How can I fix this issue?</p>

username: ptrblck
cooked: <p>This issue doesn’t seem to be PyTorch-related, so you might get a better and faster answer on an Anaconda discussion board, mailing list, or StackOverflow.</p>
<p>However, based on the warning it seems that your current setup doesn’t have <code>md5</code> installed and thus cannot verify the downloaded archive.</p>

username: 887574002
cooked: <p>Yes, sorry for posting a non-related pytorch question. I couldn’t find the answer in any other community that you mentioned. I thought perhaps someone has experience with this macOS Big Sur, here.<br>
Thank you for your answer.</p>
How to make torch.nonzero faster

username: Nathan_Drake
cooked: <p>I have a big tensor X whose shape is [?, 18900, 85] and another big tensor Y whose shape is [18900,].<br>
Data of X is float and data of Y is boolean.</p>
<p>The last broadcast operation below takes about 25ms if X and Y both in GPU:</p>
<pre><code class="lang-auto">X = YoloModel(image)
# ...
# non max suppresion preprocess
Y = X[..., 4] &gt; conf_thres
for i, x in enumerate(X):
    # x shape is [18900, 85]
    x = x[Y[i]]  # 25ms!!
# X shape now is [47, 85] for example
# ...
</code></pre>
<p>Althought it takes less than 1ms when X and Y both reside in CPU, calling X.cpu() would take roughly 25ms.  So I want to know how to fix this problem.</p>
<p>Thanks for your help!</p>

username: Nathan_Drake
cooked: <p>I found an equivalent operation:</p>
<pre><code class="lang-auto">for i, x in enumerate(X):
    x = torch.masked_select(x, Y[i].repeat(x.shape[1], 1).T).reshape(-1, x.shape[1])
</code></pre>
<p>The operation is still slow though. It seems that GPU is not good at such kind of task…</p>

username: Nathan_Drake
cooked: <p>I got more clear code below after I’ve read the doc. And I find that the performance problem comes from ‘nonzero’ function</p>
<pre><code class="lang-auto">mask = Y[i].nonzero().squeeze()
x = torch.index_select(x, 0, mask)
</code></pre>
<p>Then I found the same issue on google:<br>
<a href="https://github.com/pytorch/pytorch/issues/14848" rel="noopener nofollow ugc">torch.nonzero slower than np.nonzero · Issue #14848 · pytorch/pytorch (github.com)</a></p>
<p>GPU version ‘nonzero’ function is still much slower than that of CPU version</p>

username: Nathan_Drake
cooked: <p>It seems quite hard to parallel nonzero since it involves ‘index’.</p>

username: Nathan_Drake
cooked: <p>I think I’ve got the right idea: slice the long tensor and call nonzero on multiprocessing.</p>

username: Nathan_Drake
cooked: <p>found right timing method here: <a href="https://discuss.pytorch.org/t/measuring-gpu-tensor-operation-speed/2513/4" class="inline-onebox">Measuring GPU tensor operation speed - #4 by apaszke</a></p>
Way to avoid for loop consisting of tensor roll and norm computation to achieve faster computation

username: trinity
cooked: <p>I need to roll a tensor and compute a norm for a definite number of times. This is pretty straightforward to do by using a for loop shown as shown bellow -</p>
<pre><code class="lang-auto">for i in range(k):
        y = torch.roll(y, shifts, dims)
        x = torch.norm(y - x, dim)
</code></pre>
<p>But for my use case, this tensor is huge in size and the entire set of operations needs to be done large number of times. Also I need to do this computation inside a loss function. So using a loop is taking quite some time in terms of computation and that is becoming a considerable bottleneck. Is there any way to avoid using the for loop to perform this task, perhaps using PyTorch’s operations? Any suggestions would be highly appreciated.</p>

username: mshuaibi
cooked: <p>Running into a similar issue, did you ever resolve this?</p>

username: trinity
cooked: <p>Hi, apologies for the late reply! I did not exactly resolve that issue. I actually reformulated that use-case problem to avoid that.</p>
Higher order derivative - differentiated tensor has not been used in graph

username: se3
cooked: <p>Hi,</p>
<p>I have this toy example and trying to take the second order derivative. I was able to use .grad to compute the first order derivative, but I was having trouble with the second order derivative. The error was something about one of the differentiate tensors has not been used in the graph. I tried to set allow_unused to True and it returned None. Can someone help explain what is wrong in this toy example? Thank you!</p>
<pre><code class="lang-auto">class TOY_NN(torch.nn.Module):

    def __init__(self):
        super().__init__()
        self.layer1 = torch.nn.Linear(1,3)
        self.layer2 = torch.nn.Linear(3,2)

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        return x


nn = TOY_NN()
x = torch.tensor([3.]).requires_grad_(True)
out = nn(x)
d_out_d_in = torch.autograd.grad(outputs = out, inputs = x, retain_graph = True,
                                     create_graph = True, grad_outputs=torch.ones(out.size()[0]))[0]
d_out_d_in_2 = torch.autograd.grad(outputs = d_out_d_in, inputs = x, retain_graph = True,
                                 create_graph = False, allow_unused=False, grad_outputs=torch.ones(d_out_d_in.size()[0]))[0]

</code></pre>
How to use tensor image in Lime?

username: Geoffrey_Payne
cooked: <p>I want to use the Lime library to evaluate an image for explainability.<br>
I have these lines of code;</p>
<pre><code class="lang-auto">    iter = 3500
    top_labels = 5
    num_super_pixels = 10
    fasterrcnn_model.eval()
    explainer = lime_image.LimeImageExplainer()
    explanation = explainer.explain_instance(img, fasterrcnn_model, top_labels = top_labels, hide_color = 0, num_samples = iter)
</code></pre>
<p>It seems line where I execute the explain_instance method, it does not like the image being a tensor.<br>
When I convert it from a PILImage to a nump array, I have a problem because I need this image to have a dim attribute, which it does not have.</p>
<pre><code class="lang-auto">def get_image(file):
    path = os.path.join(Constants.dir_individual_image, file)
    p_img = PILImage.open(path)
    img = np.array(p_img)
    t_img = transform_img(img)
    return t_img, img, path
</code></pre>
<p>I am wondering if I can use LIME for object detection and Pytorch?</p>
Trasforming a tensor to different size

username: Hdk
cooked: <p>Hello, I have a 4D tensor of size e.g. (batch_size=10, channels=10, height=6, width=6). I want to resize each (6x6) channel to (30x30). I want to conserve the pixel values as much as possible during resizing, maybe by interpolation or so.</p>
<p>How can I do it for all the channels and images at once instead of one by one with torchvision.transforms?</p>

username: cskarthik7
cooked: <p>This might work</p>
<pre><code class="lang-auto">import torch
from torchvision import transforms
transform = transforms.Compose([
  transforms.Resize((30,30)),
])
x = torch.ones(10, 10, 6, 6)
y = transform(x)
print(y.shape)
</code></pre>
<p>Output -</p>
<pre><code class="lang-auto">torch.Size([10, 10, 30, 30])
</code></pre>

username: Hdk
cooked: <p>Thanks <a class="mention" href="/u/cskarthik7">@cskarthik7</a> for reply.</p>
<p>I tried it already, but the problem is that the pixels are a bit dull as you can see in the image which is understandable.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2f5e51f9a199fb6dc969ff2ac3889ad7575e5b06.png" alt="a" data-base62-sha1="6L2wjEbYnSyqMbN2exMINW3ntWe" width="414" height="254"><br>
So, any ideas to sharpen the image?</p>

username: cskarthik7
cooked: <p>Image resize generally gives an image after resampling. I guess the default resampling method is Nearest Interpolation. You can explore more.</p>

username: Hdk
cooked: <p>Found rather a better solution through <a href="https://discuss.pytorch.org/t/resize-tensor-without-converting-to-pil-image/52401/4">this thread.</a></p>
Inductive vs Transductive transfer learning

username: Sahil_malik
cooked: <p>Hi members,</p>
<p>I am bit confused b/w inductive and transductive TL; Transffering knowledge from VGG16 pre-trained model to calssify Fashion-MNIST would be an inductive or transductive transfer learning?</p>
<p>Thank you!</p>

username: cskarthik7
cooked: <p>So there’s a difference in inductive and transductive learning.</p>
<p>Inductive Learning - Goes through only the training data during the training phase<br>
Transductive Learning - Goes through training and testing data during the training phase and if a new data point is added to the testing data the training needs to be done again from scratch.</p>
<p>So this difference clearly indicates that transfer learning is a inductive learning. Transfer Learning is simply using a model which was created for a prior task.</p>
Pytorch operation

username: cbd
cooked: <p>Let “a” is array of [1,1,3,3] tensor having value from 1 to 50. and “b” is torch.zero tensor of same size [1,1,3,3]. “c” is array of [5] tensor.</p>
<p>If 1st value in tensor “c” is non zero then i want to find out value in range (0,10) from tensor “a” and replace the value of c[0] in tensor “b” of size [1,1,3,3] where tensor “a” of size [1,1,3,3] have value between (0,10).</p>
<p>If 2nd value in tensor “c” is non zero then i want to find out value in range (11,20) from tensor “a” and replace the value of c[1] in tensor “b” of size [1,1,3,3] where tensor “a” of size [1,1,3,3] have value between (11,20).</p>
<p>For example, let c=[0.1, 0.3, 0, 0.5, 1] and if a[0,0,0,0] = 9 (its between 0 to 10) then b[0,0,0,0] = 0.1 (first value of c). If a[0,0,0,0] = 39 (its between 31 to 40) then b[0,0,0,0] = 0.5 (fourth value of c).</p>
<p>Is there any simplest way using pytorch functions?</p>
What kind of Neural Network should I build to classify each instance of a time series sequence?

username: bbasaran
cooked: <p>Let’s say, I have the time-series dataset below-left. I would like to train a model in such a way that if I feed the model with an input like below-right, it should be able to classify each sample with the correct class label.</p>
<pre><code class="lang-auto">    Training Sequence:             Test Sequence:

Time,   Bitrate,   Class           Time,   Bitrate      Predicted Class
 0,       312,       1              0,       234      -----&gt; 0
 0.3,     319,       1              0.2,     261      -----&gt; 0
 0.5,     227,       0              0.4,     277      -----&gt; 0
 0.6,     229,       0              0.7,     301      -----&gt; 1 
 0.7,     219,       0              0.8,     305      -----&gt; 1
 0.8,     341,       1              0.9,     343      -----&gt; 1 
 0.9,     281,       0              1.0,     299      -----&gt; 0 
          ...                                ...
</code></pre>
<p>Any help is appreciated. Thanks!</p>

username: ptrblck
cooked: <p>Are the columns (time, bitrate) the input features? If so, I guess you could try to train a small classifier or it seems even a threshold such as <code>return bitrate &gt; 300</code> could perfectly classify the data.</p>

username: bbasaran
cooked: <p>This is a network experiment, therefore the only feature column is the bitrate. However, the actual data is not like this, I mean that one simple threshold wouldn’t work. I just wrote those values to demonstrate.</p>

username: vdw
cooked: <p>That sounds like a <a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" rel="noopener nofollow ugc">sequence labeling task</a>, which is common in NLP, e.g., to assign each word in a sentence a Part-of-Speech tag. You should be able to easily adopt the linked example if it fits the bill.</p>

username: bbasaran
cooked: <p>Thank you Chris! I will definitely have a look at this.</p>
Big matrix multiplication on GPU eats up my CPU memory

username: marcel1991
cooked: <p>I am doing a correspondance calculation between two matrices. One of these vectors has a fixed shape of [256, 5000] and the other one is variable of shape [256, n]. Here n is usually smaller than 100.</p>
<p>Both matrices are on the gpu and I do the calculation as follows:</p>
<pre><code class="lang-auto"> # mat1, mat2 are on cuda
correspondance_mat = mat1.unsqueeze(-1) * mat2.unsqueeze(1) 
</code></pre>
<p>This results in a matrix of shape [256, 5000, n].</p>
<p>The strange problem I am having now is that every time n increases to a larger value than ever before, a huge increase in CPU RAM occures. This used memory also does not seem to be realeased anymore, and it looks like its leaking. This even gets my computer to freeze when it reaches 100% memory occupation.</p>
<p>My question now is what is happening here?<br>
1.) Since both matrices are on GPU why does it even affect my CPU RAM???<br>
2.) Why does it not release the memory anymore after it is finished with this multiplication?<br>
3.) Could this be a bug/memory leak in pytorch?</p>
<p>My Pytorch version is 1.4 and I am using it with Python 2.7. Unfortunately there is no way I can upgrade this version.</p>

username: ptrblck
cooked: <p>I cannot reproduce this issue using:</p>
<pre><code class="lang-python">import torch
import psutil

a = torch.randn(256, 5000, 1, device='cuda')

for epoch in range(10):
    for n in range(1, 100):
        b = torch.randn(256, 1, n, device='cuda')
        out = a * b
        print('Epoch {} Iter {}'.format(epoch, n))
        print(psutil.virtual_memory().used)
        print('GPU allocated mem {:.3f}MB'.format(torch.cuda.memory_allocated()/1024**2))
</code></pre>
<p>and get:</p>
<pre><code class="lang-auto">Epoch 0 Iter 1
16522018816
GPU allocated mem 9.767MB
Epoch 0 Iter 2
16522534912
GPU allocated mem 15.119MB
[...]
Epoch 0 Iter 98
16523194368
GPU allocated mem 483.494MB
Epoch 0 Iter 99
16523194368
Epoch 1 Iter 1
16523194368
GPU allocated mem 9.767MB
Epoch 1 Iter 2
16523194368
GPU allocated mem 15.119MB
[...]
Epoch 9 Iter 98
16523194368
GPU allocated mem 483.494MB
Epoch 9 Iter 99
16523194368
</code></pre>
<p>Which shows an overall host memory increase of <code>(16523194368 - 16522018816) / 1024**2 ~= 1.12MB</code>.</p>
<p>EDIT: I would generally recommend to update PyTorch to the latest release especially since you are also using the deprecated Python2.7 version.</p>

username: marcel1991
cooked: <p>Thank you for the quick reply.</p>
<p>I use a Jetson AGX Xavier with CUDA 10.0 in combination with ROS which unfortunately limits me to this configuration including Python 2.7. I will maybe build a newer version with Pytorch from source and try to setup ROS with Python 3 if I have time. However, I will first investigate if this is due to the Pytorch version or maybe Python 2.7 itself. I will investigate this and report back.</p>
<p>For now I fixed this leak(?) by splitting up the matrix multiplication in a for loop and using a max size for n of 30 in each step. This does not result in RAM memory building up until overflow.</p>
How to convert this 3d matrix to 2d efficiently?

username: ADONAI_TZEVAOT
cooked: <p>I have a matrix A which is 3d and I want to convert so that it is equal to B which is a 2d matrix</p>
<pre><code class="lang-auto">&gt; A = torch.tensor( [ [[1,1,1,1,1],
&gt;          [2,2,2,2,2],
&gt;          [3,3,3,3,3],
&gt;          [4,4,4,4,4],
&gt;          [ 5,5,5,5,5]],
&gt; 
&gt;         [[6,6,6,6,6]
&gt;          [7,7,7,7,7],
&gt;          [8,8,8,8,8],
&gt;          [9,9,9,9,9],
&gt;          [ 10,10,10,10,10]],
&gt; 
&gt;         [[11,11,11,11,11],
&gt;          [12,12,12,12,12],
&gt;          [13,13,13,13,13],
&gt;          [14,14,14,14,14],
&gt;          [ 15,15,15,15,15]]])
&gt; 
&gt; B = torch.tensor( [[1,1,1,1,1, 6,6,6,6,6, 11,11,11,11,11],
&gt;          [2,2,2,2,2, 7,7,7,7,7, 12,12,12,12,12],
&gt;          [3,3,3,3,3, 8,8,8,8,8, 13,13,13,13,13],
&gt;          [4,4,4,4,4, 9,9,9,9,9, 14,14,14,14,14],
&gt;          [ 5,5,5,5,5, 10,10,10,10,10, 15,15,15,15,15]])
</code></pre>
<p>Please help</p>

username: Mole_Turner
cooked: <p>I’m really awful at working out dimensions from printing out arrays. But if I’m right <code>A.shape = (5,5,3) and B.shape = (5,15)?</code> If this is the case could you not just do <code>A = torch.reshape(A,B.shape())</code></p>

username: ptrblck
cooked: <p>Directly reshaping <code>A</code> will unfortunately not work, as it would result in:</p>
<pre><code class="lang-python">print(A.view(B.size()))
&gt; tensor([[ 1,  1,  1,  1,  1,  2,  2,  2,  2,  2,  3,  3,  3,  3,  3],
          [ 4,  4,  4,  4,  4,  5,  5,  5,  5,  5,  6,  6,  6,  6,  6],
          [ 7,  7,  7,  7,  7,  8,  8,  8,  8,  8,  9,  9,  9,  9,  9],
          [10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12],
          [13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15]])
</code></pre>
<p>To get the desired output you would need to <code>permute</code> the first two dimensions first:</p>
<pre><code class="lang-python">print(A.permute(1, 0, 2).reshape(B.size()))
&gt; tensor([[ 1,  1,  1,  1,  1,  6,  6,  6,  6,  6, 11, 11, 11, 11, 11],
          [ 2,  2,  2,  2,  2,  7,  7,  7,  7,  7, 12, 12, 12, 12, 12],
          [ 3,  3,  3,  3,  3,  8,  8,  8,  8,  8, 13, 13, 13, 13, 13],
          [ 4,  4,  4,  4,  4,  9,  9,  9,  9,  9, 14, 14, 14, 14, 14],
          [ 5,  5,  5,  5,  5, 10, 10, 10, 10, 10, 15, 15, 15, 15, 15]])
</code></pre>
Google Colaboratory

username: Rajat_Singh1
cooked: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210429-060047-RC00_371090719" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/1f32gj5IYIyFipoINiC8P3DvKat-WWLUK" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1f32gj5IYIyFipoINiC8P3DvKat-WWLUK" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Code snippet–</p>
<h1>
<a name="specify-max_len-1" class="anchor" href="#specify-max_len-1"></a>Specify <code>MAX_LEN</code>
</h1>
<p>MAX_LEN = 6</p>
<h1>
<a name="print-sentence-0-and-its-encoded-token-ids-2" class="anchor" href="#print-sentence-0-and-its-encoded-token-ids-2"></a>Print sentence 0 and its encoded token ids</h1>
<p>token_ids = list(preprocessing_for_bert([X[0]])[0].squeeze().numpy())</p>
<p>print('Original: ', X[0])</p>
<p>print('Token IDs: ', token_ids)</p>
<h1>
<a name="run-function-preprocessing_for_bert-on-the-train-set-and-the-validation-set-3" class="anchor" href="#run-function-preprocessing_for_bert-on-the-train-set-and-the-validation-set-3"></a>Run function <code>preprocessing_for_bert</code> on the train set and the validation set</h1>
<p>print(‘Tokenizing data…’)</p>
<p>train_inputs, train_masks = preprocessing_for_bert(X_train)</p>
<p>val_inputs, val_masks = preprocessing_for_bert(X_val)</p>
<h2>
<a name="error-4" class="anchor" href="#error-4"></a>error–</h2>
<p>TypeError                                 Traceback (most recent call last)</p>
<p> in ()<br>
3<br>
4 # Print sentence 0 and its encoded token ids<br>
----&gt; 5 token_ids = list(preprocessing_for_bert([X[0]])[0].squeeze().numpy())<br>
6 print('Original: ', X[0])<br>
7 print('Token IDs: ', token_ids)</p>
<p>TypeError: can’t convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.</p>

username: ptrblck
cooked: <p>As the error points out, you cannot directly transform a GPU tensor to a numpy array and would need to transfer it to the CPU first via the <code>tensor.cpu()</code> operation before calling <code>.numpy()</code>.</p>
RuntimeError: expanded size

username: jpj
cooked: <pre><code class="lang-auto">self.hidden_layer_1 = torch.nn.Linear(50,78)
self.output = torch.nn.Linear(78, 2)
</code></pre>
<pre><code class="lang-auto">def forward(self, input):
        H1 = self.hidden_layer_1(input)
        H1 = self.ReLU(H1)
        final_inputs = self.output(H1)
        # not applying activation on final_inputs because CrossEntropyLoss does that
        return final_inputs
</code></pre>
<p>input shape = (2425727, 50)</p>
<pre><code class="lang-auto">        weights = np.load('./w.npy')
        bias = np.load('./b.npy')
        print(weights.shape,bias.shape)# (50, 78) (78,)
        self.hidden_layer_1.weight = torch.nn.Parameter(torch.from_numpy(weights))
        self.hidden_layer_1.bias = torch.nn.Parameter(torch.from_numpy(bias))
</code></pre>
<p>Error - RuntimeError: The expanded size of the tensor (50) must match the existing size (78) at non-singleton dimension 1.  Target sizes: [1024, 50].  Tensor sizes: [78]</p>

username: ptrblck
cooked: <p>Your <code>weights</code> and <code>bias</code> seem to have a wrong shape, since the default linear layer using your setup would use:</p>
<pre><code class="lang-python">print(nn.Linear(50, 78).weight.shape)
&gt; torch.Size([78, 50])
print(nn.Linear(50, 78).bias.shape)
&gt; torch.Size([78])
</code></pre>
<p>so you migth need to fix it (<code>permute</code> the <code>weights</code> and make sure the <code>bias</code> has a single dimension).</p>
batchNorm1d input N,C,L vs N,L

username: Mohamad_Bairakdar
cooked: <p>I expected the following code to have <code>True</code> for all entries of <code>a</code> but it doesn’t. Why is that? (I’m trying to understand the difference between an input shape N,C,L and N,L for batch norm 1D)</p>
<pre><code class="lang-auto">  self.bn1 = torch.nn.BatchNorm1d(hidden_channels)
  def bn(self, x):
        batch_size, num_nodes, num_channels = x.size()
        y = x.clone()
        x = x.view(-1, num_channels)
        x = getattr(self, "bn1")(x)
        y = y.transpose(2,1)
        y = getattr(self,"bn1")(y)
        y = y.reshape(-1, num_channels)
        a = torch.eq(x,y)
</code></pre>

username: ptrblck
cooked: <p>The last <code>reshape</code> operation will interleave the output values and you would need to <code>permute</code> <code>y</code> before calling <code>reshape</code>:</p>
<pre><code class="lang-python">
hidden_channels = 4
bn1 = torch.nn.BatchNorm1d(hidden_channels)

x = torch.randn(2, 3, 4)
batch_size, num_nodes, num_channels = x.size()
y = x.clone()
x = x.view(-1, num_channels)
x = bn1(x)

y = y.transpose(2,1)
y = bn1(y)
y = y.permute(0, 2, 1).reshape(-1, num_channels)
print((x - y).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p><code>y</code> will have the shape <code>[batch_size, channels, seq_len]</code> after the call into <code>bn1</code>.<br>
If you want to flatten the temporal dimension into the batch dimension, you would have to <code>permute</code> it first to <code>[batch_size, seq_len, channels]</code> before calling <code>reshape(-1, num_channels)</code>.</p>
How to assign values to specific blocks within a 2D tensor？

username: Qizhou
cooked: <p>Hi! I want to realize this operation:<br>
Here is a 2D tensor:<br>
[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]]<br>
I want to set 1 to specific square blocks, such like this:<br>
[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,1,1,1,0,0,0,0,1,1,1,0,0,0],<br>
[0,0,1,1,1,0,0,0,0,1,1,1,0,0,0],<br>
[0,0,1,1,1,0,0,0,0,1,1,1,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],<br>
[0,0,1,1,1,0,0,0,0,1,1,1,0,0,0],<br>
[0,0,1,1,1,0,0,0,0,1,1,1,0,0,0],<br>
[0,0,1,1,1,0,0,0,0,1,1,1,0,0,0],<br>
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]]<br>
I’ve tried this code before:</p>
<pre><code class="lang-auto">import torch 
import numpy as np 
import matplotlib.pyplot as plt

def random_patch_slice(h, w, p_size, sqrt_n):
  i_h = np.random.choice(range(0,h-p_size),size=sqrt_n,replace=False)
  i_w = np.random.choice(range(0,w-p_size),size=sqrt_n,replace=False)
  i_h = np.linspace(i_h, i_h + p_size-1,p_size,dtype=int,axis=1).flatten()
  i_w = np.linspace(i_w, i_w + p_size-1,p_size,dtype=int,axis=1).flatten()  
  return i_h, i_w

imgs = torch.zeros([10,10]) 
i_h,i_w = random_patch_slice(10, 10, 5, 2) 
imgs[i_h,i_w] = 1  
print(imgs)
plt.imshow(imgs)
plt.show()   
</code></pre>
<p>But torch maps the indexes of the two dimensions one by one instead of the form of Cartesian product.<br>
Then I tried this code:</p>
<pre><code class="lang-auto">import torch 
import numpy as np 
import matplotlib.pyplot as plt

def random_patch_slice(h, w, p_size, sqrt_n):
  i_h = np.random.choice(range(0,h-p_size),size=sqrt_n,replace=False)
  i_w = np.random.choice(range(0,w-p_size),size=sqrt_n,replace=False)
  i_h = np.linspace(i_h, i_h + p_size-1,p_size,dtype=int,axis=1).flatten()
  i_w = np.linspace(i_w, i_w + p_size-1,p_size,dtype=int,axis=1).flatten()  
  return i_h, i_w

imgs = torch.zeros([10,10]) 
i_h,i_w = random_patch_slice(10, 10, 5, 2) 
imgs[i_h][:,i_w] = 1   # modify here
print(imgs)
plt.imshow(imgs)
plt.show()   
</code></pre>
<p>The value was just not assigned to the tensor. How can I realize this operation?<br>
I would appreciate it if you can help me. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
Reset Optimizer for retraining

username: aRI0U
cooked: <p>Hi all,<br>
I am currently implementing a method that needs a model to be trained multiple times on different datasets but while keeping identical architecture, optimizer, etc. I therefore need to reset model weights, optimizer stats and so on multiple times.<br>
I am aware that I can reset the model weights with <code>for _, module in model.named_children(): module.reset_parameters()</code> but is there some similar method for optimizers?</p>
<p>As of now the only trick I found is to recreate it, so I wonder if I can just reset its parameters directly. Here is how I recreate it, please tell me if it’s wrong:</p>
<pre><code class="lang-auto">new_optimizer = old_optimizer.__class__(model, **old_optimizer.defaults)
</code></pre>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I think recreating the optimizer is the right approach.<br>
Besides calling <code>reset_parameters()</code> on each module, you could also recreate both, the model and optimizer, in case this would fit your use case.</p>
Constant offset with NN regression

username: costelljoe
cooked: <p>I’m using a simple 4-layer feedforward network to predict velocities, which can be positive and negative. The first 3 layers use ReLU and the final layer is linear to be able to output negative and positive values. I’m using Adam with MSE loss and I stop training when validation loss stops improving.</p>
<p>The network trains and converges to a solution relatively fast, however, <strong>the final network predictions often have a constant offset/bias relative to the true data</strong>. I would expect the MSE loss to minimize this constant offset. Could this be related to learning rate, or should I try a different loss that has a larger penalty on small offsets?</p>
<p>Here is an example output on test-data, where blue is the true velocity and orange is the NN predicted velocity. The orange has a constant negative bias.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cfdc5d9c88c8b4d839862af4ee53744cfb577fa.png" alt="predictions with offset" data-base62-sha1="fybgU25fB9QfJiYaSoz2VaShqQy" width="613" height="325"></p>
Long indexing with further indexing on Tensor gives a copy

username: marcel1991
cooked: <p>I see that when we index a Tensor with another LongTensor and afterwards normally index it via [:] a copy is created.</p>
<p>For example</p>
<pre><code class="lang-auto">a = torch.Tensor([10, 20, 30, 40, 50])
ind = torch.LongTensor([1,3])

a[ind][:] = 0
print(a)
# prints [10, 20, 30, 40, 50]

a[ind] = 0
print(a)
# prints [10, 0, 30, 0, 50]
</code></pre>
<p>No I am wondering how I can do the same without getting a copy of the Tensor so that I am able to overwrite the indexed values?<br>
E.g. if I want to set the first two values of an indexed tensor to zero</p>
<pre><code class="lang-auto">a[ind][:2] = 0
</code></pre>
<p>How would I do this so the actual Tensor is eventually overwritten?</p>

username: googlebot
cooked: <p>the last statement is same as:<br>
setitem(<strong>getitem(a,ind)</strong>, slice(0,2), 0)</p>
<p>so it is staged with early gathering, and thus not composable as inplace operation</p>
<p>but you can do things like: a[ind[:2]] = 0</p>

username: marcel1991
cooked: <p>Thank you very much. That was exactly what I was looking for!</p>
Exponential of sparse tensor

username: Gabriele_Orlando
cooked: <p>hello,</p>
<p>is there a way to get the exp of a sparse tensor? It seems exp is not supported, and</p>
<p>torch.pow(torch.tensor([math.e]),sparse)</p>
<p>where sparse is a sparse coalesced tensor.</p>
<p>returns</p>
<p>RuntimeError: Could not run ‘aten::pow.Tensor_Tensor’ with arguments from the ‘SparseCPU’ backend. ‘aten::pow.Tensor_Tensor’ is only available for these backends: [CPU, CUDA, Named, Autograd, Profiler, Tracer, Autocast].</p>
<p>Since there is a Softmax implemented for sparse tensors I assume there is an exponential as well.<br>
Thanks in advance for the help.</p>
<p>Best,<br>
Gabriele</p>

username: tom
cooked: <p>Clumsy but working is to create the tensor manually using the values:</p>
<pre><code class="lang-python">csp = sp.coalesce()
torch.sparse_coo_tensor(csp.indices(), csp.values().exp(), csp.shape)
</code></pre>
ParameterDict of ParameterList throws an error?

username: Boltzmachine
cooked: <p>I can implement ModuleDict of ModuleList. But I got an error when I write a ParameterDict of ParameterList</p>
<pre><code class="lang-python">nn.ParameterDict({'a': nn.ParameterList([nn.Parameter(torch.Tensor(1,2))])})
</code></pre>
<pre><code class="lang-python">/home/qiuweikang/miniconda3/lib/python3.8/site-packages/torch/nn/modules/container.py:434: UserWarning: Setting attributes on ParameterList is not supported.
  warnings.warn("Setting attributes on ParameterList is not supported.")
/home/qiuweikang/miniconda3/lib/python3.8/site-packages/torch/nn/modules/container.py:550: UserWarning: Setting attributes on ParameterDict is not supported.
  warnings.warn("Setting attributes on ParameterDict is not supported.")
Traceback (most recent call last):
  File "/home/qiuweikang/UnifiedNLP/train.py", line 164, in &lt;module&gt;
    main()
  File "/home/qiuweikang/UnifiedNLP/train.py", line 92, in main
    model = Simple(config).to(device)
  File "/home/qiuweikang/UnifiedNLP/models/simple.py", line 23, in __init__
    self.query_weight = nn.ParameterDict({
  File "/home/qiuweikang/miniconda3/lib/python3.8/site-packages/torch/nn/modules/container.py", line 537, in __init__
    self.update(parameters)
  File "/home/qiuweikang/miniconda3/lib/python3.8/site-packages/torch/nn/modules/container.py", line 615, in update
    self[key] = parameter
  File "/home/qiuweikang/miniconda3/lib/python3.8/site-packages/torch/nn/modules/container.py", line 543, in __setitem__
    self.register_parameter(key, parameter)
  File "/home/qiuweikang/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 321, in register_parameter
    raise TypeError("cannot assign '{}' object to parameter '{}' "
TypeError: cannot assign 'torch.nn.modules.container.ParameterList' object to parameter 'dep' (torch.nn.Parameter or None required)

</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Yes, as of right now, the ParameterDict can only contain Parameters, not other nn.Modules.<br>
Note that nn.ModuleDict already exist and that is their exact use case <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: Boltzmachine
cooked: <p>Thanks. Maybe I should find another way to do what I want</p>
Building Pytorch with rocm4.0(beta)

username: Shre
cooked: <p>I am using acer nitro 5 ryzen 5 AN515-42 (2018) model, which has ryzen 2500u, vega8 igpu, rx560x as dgpu on ubuntu 20.04.2</p>
<p>I tried to install pytorch with rocm4.0(beta), with no rocm installation ever, (should i try to download rocm, from  laptop has anaconda installed. There are 2 commands as given in the getting started document- <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>command1 - was successful</p>
<p>command 2 - pip3 install ninja &amp;&amp; pip3 install ‘git+https://github.com/pytorch/vision.git@v0.9.1’<br>
which got error-ed out  ( ninja got installed properly)</p>
<p>(adding ss, as more than 2 links for new accounts is not allowed, F<br>
Also, i dont know how to get logs too)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c46d75ba31f7e0b85e173d4a5742512c94f9340b.png" data-download-href="https://discuss.pytorch.org/uploads/default/c46d75ba31f7e0b85e173d4a5742512c94f9340b" title="Screenshot from 2021-04-30 20-54-58"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c46d75ba31f7e0b85e173d4a5742512c94f9340b_2_690x387.png" alt="Screenshot from 2021-04-30 20-54-58" data-base62-sha1="s1G6fS0elnleO00c0rSVAUytFYD" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c46d75ba31f7e0b85e173d4a5742512c94f9340b_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c46d75ba31f7e0b85e173d4a5742512c94f9340b_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/4/c46d75ba31f7e0b85e173d4a5742512c94f9340b.png 2x" data-dominant-color="381328"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-04-30 20-54-58</span><span class="informations">1366×768 137 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: tom
cooked: <p>On Debian/Ubuntu you want to install the package libtinfo5 . I don’t know how it works for conda.<br>
Note that I also don’t know whether your setup is supported by ROCm (see eg <a href="https://github.com/RadeonOpenCompute/ROCm/issues/750" class="inline-onebox" rel="noopener nofollow ugc">make rocm work with "ASUS TUF505DY" · Issue #750 · RadeonOpenCompute/ROCm · GitHub</a> mentioning rx560x).</p>
<p>Best regards</p>
<p>Thomas</p>