akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
transforms.Normalize(): expected type Double but got type Float

patrick-han: <p>I see this question has been previously asked, but the solutions posted don’t seem to be working for me/the traceback message is different. For some reason my model is failing during an application of <code>transforms.Normalize()</code>:</p>
<pre><code class="lang-auto">transforms_ =  [transforms.Normalize((0.5,), (0.5,))]
self.transform = transforms.Compose(transforms_)

raw_arr_A = np.load(self.files_A[index % len(self.files_A)])
raw_arr_A_res = raw_arr_A[:,:,0]
tensor_A = torch.unsqueeze(torch.from_numpy(raw_arr_A_res), 0).double()
item_A = self.transform(tensor_A)
</code></pre>
<p>Printing the <code>.type()</code> of <code>tensor_A</code> it’s definitely <code>torch.DoubleTensor</code></p>
<p>Bottom of the traceback: (It starts at a for loop over an <code>enumerate()</code> on my dataloader</p>
<pre><code class="lang-auto">File "/Users/user/Documents/env/lib/python3.7/site-packages/torchvision/transforms/functional.py", line 208, in normalize
    tensor.sub_(mean[:, None, None]).div_(std[:, None, None])
RuntimeError: expected backend CPU and dtype Double but got backend CPU and dtype Float
</code></pre>
<p>I’ve tried casting my networks to <code>.double()</code> (even though I think that is unrelated). Does <code>transforms.Normalize()</code> not accept Double type tensors or something?</p>

ptrblck: <p>This was a known bug and was recently fixed <a href="https://github.com/pytorch/vision/pull/1021" rel="nofollow noopener">here</a>.<br>
Since the PR was merged 4 days ago, you might need to build from source.</p>
PyTorch module (nn.Module) questions

blackbirdbarber: <p>Hi, I printed the alexnet model…</p>
<pre><code>model = models.alexnet(pretrained=True)
</code></pre>
<p>What is the idea behind features, classifier, and avgpool inside:</p>
<pre><code>model._models
</code></pre>
<p>Just need little explain.</p>

barthelemymp: <aside class="quote no-group" data-username="blackbirdbarber" data-post="1" data-topic="48275">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>features, classifier, and avgpool</p>
</blockquote>
</aside>
<p>The Neural Networks has just been devided in 3 parts (you could also just see it as one big NN)</p>
<p>the features part is just applying the following layers:</p>
<pre><code class="lang-auto">nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(64, 192, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
</code></pre>
<p>Then perform an averagepooling,</p>
<p>and finally applying the following group of layers (here gathered under the name classifier):</p>
<pre><code class="lang-auto">nn.Sequential(
            nn.Dropout(),
            nn.Linear(256 * 6 * 6, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )
</code></pre>

blackbirdbarber: <p>Thanks should look into the source code myself. <img src="https://discuss.pytorch.org/images/emoji/apple/sunglasses.png?v=9" title=":sunglasses:" class="emoji" alt=":sunglasses:"></p>
Jit.trace() question about SiamMask network

Erissonleo: <p>My Siammask network is used for object tracking. There are two input arguments in my network, one is target object and the other one is search domain. I hope to use libtorch, but i dont’t know how can I use jit.trace() funciton in Pytorch to save it . I referred some blog and docs but could not find a method for double input args about jit.trace() function. Cound anyone meet the similar situation?</p>

driazati: <p><code>torch.jit.trace()</code> takes in the function to trace and a tuple of example inputs, so you can pass as many inputs as your model needs.</p>
<p>See <a href="https://pytorch.org/docs/stable/jit.html#torch.jit.ScriptModule" rel="nofollow noopener">Tracing</a> for more details, does this fix your issue?</p>
Using sklearn preprocessing in utils.data.dataset

barthelemymp: <p>Hello,</p>
<p>I’d like to add some pre-processing on my data. My data are just a list of features. For example I would like to do a polynomial expansion. sklearn propose <code>PolynomialFeatures</code>.<br>
However I’m wondering which way would be the best to apply if.</p>
<p>Either in the <code>__init__</code> of my dataset:</p>
<pre><code class="lang-auto">class MyDataset(data.Dataset):


    def __init__(self, csv_file, transform=None):

        self.data = pd.read_csv(csv_file)
        ...
        self.Xs = self.data.iloc[:,1:-1]
        poly = PolynomialFeatures(2)
        self.Xs = poly.fit_transform(self.Xs)



        self.labels = self.data.iloc[:,-1]
        self.transform = transform
</code></pre>
<p>Or as a transform in my getitem, which means I neead to define a transform:</p>
<pre><code class="lang-auto">class Polynomial_exp(object):


     def __init__(self, degree):
        self.degree = degree
        self.poly = PolynomialFeatures(2)
        


     def __call__(self, sample):
        
        
        
        
        sample = {'X': X, 'label': Y}
        X, Y = sample['X'], sample['label']
        ... (change to numpy, reshape as a 2dim matrice)
        X = self.poly.fit_transform(X)

        return {'X': X, 'label': Y}
</code></pre>
<p>Indeed it the sklearn function can work on the entire dataset, and in the second option I’m calling it for each sample. However this remark can be done for all transform we add in our custome dataset. So I’m wondering the point of it and when we should use each solution.</p>
<p>Regards</p>
<p>Barthélémy</p>

Nikronic: <p>Hi,</p>
<p>As far as I know based on model’s I have implemented or source codes I have read in github ,etc, developers usually use their preprocessing step using custom defined class and in <code>__getitem__</code> method.</p>
<p>Actually, I have implemented 2 different classes for different tasks, one using Native python and other using third-party libraries and both works fine.</p>
<p>So I go for the second method.</p>
Passing back variables through the graph

zwang-asapp: <p>I understand that in Pytorch, a node in the computation graph can communicate with its downstream and upstream neighbor nodes through passing gradients. However, I wonder if a node in the graph can directly communicate with a non-neighboring downstream node. For example, I would like all the nodes in the graph to communicate directly with the loss computation. (aka broadcast a variable from the loss node to the entire graph) E.g. Say based on the softmax computations of the loss, I’d like to change the gradient computations of all the convolution layers.</p>
<p>What is the least-effort approach to do this? Thank you!</p>
Pytorch model not utilizing all the cores of cpu

rajamohan_reddy: <p>Hi, I trained model on GPU and testing model on CPU. I found pytorch is not fully utilizing all the cores of CPU. I found most of the threads on my mac are not being utilized. In Tensorflow/keras it happens without updating any settings.</p>
<p>So, How to make pytorch model utilize all the cores of CPU while doing inference/prediction?</p>
How to calculate the FLOPs of a BatchNorm layer?

jih332: <p>Hi, I’m trying to calculate the FLOPs of a BatchNorm2d layer, can anyone tell me how it is implemented in Pytorch? Is it always automatically fused into previous conv layer or not?<br>
Thanks!</p>

tymokvo: <p>I’m not sure if a “true” FLOPS number can be calculated, as it depends on many vaiables (CUDA/MKL version, Driver version, model params, etc.) But the <a href="https://pytorch.org/docs/stable/autograd.html?highlight=emit_nvtx#torch.autograd.profiler.emit_nvtx" rel="nofollow noopener">profiling tools</a> might give you a good idea of computational cost.</p>

jih332: <p>Thanks for your reply! Will take a look at the tools you mentioned.</p>

AtenaNguyen: <p>How about this one? They did support batchnorm as well<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/sovrasov/flops-counter.pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/5373517?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/sovrasov/flops-counter.pytorch" target="_blank" rel="nofollow noopener">sovrasov/flops-counter.pytorch</a></h3>

<p>Flops counter for convolutional networks in pytorch framework - sovrasov/flops-counter.pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

jih332: <p>Thanks for the reply. I’ve checked their code but it seems that they only implemented for inference, while I was trying to get a full estimation of FLOPs on training. But I guess just as tymokvo said, without knowing how the batchnorm is implemented in cuDNN, I can’t make any reasonable estimation.</p>
Load text data from file

slavavs: <p>I have a file with text data (dataset for predict).<br>
eg. such (many rows, three columns)<br>
34, 56, 76<br>
44, 55, 79<br>
45, 79, 87<br>
…<br>
The file is large, about 700mb.<br>
How best can I convert this file to pytorch?</p>

Pranavan_Theivendira: <p>Hi,</p>
<p>You can do the following. Replace file.csv with your file name.</p>
<pre><code class="lang-auto">from numpy import genfromtxt
my_data = genfromtxt('file.csv', delimiter=',')
</code></pre>
<p>You will get a numpy array(Nx3). There may be other ways in pytorch. This might help.</p>
<p>Thanks</p>

slavavs: <p>Thanks, I did just that.<br>
But I was embarrassed by the data output.</p>
<p>if i do like this:</p>
<pre><code class="lang-auto">b = "1.0;0.0;512.50;507.00;508.25;122.0;20.0;0.0;20.0"
c = b.split(';')
r = [float(item) for item in c]
print(r)
</code></pre>
<blockquote>
<blockquote>
<blockquote>
<p>[1.0, 0.0, 512.5, 507.0, 508.25, 122.0, 20.0, 0.0, 20.0]</p>
</blockquote>
</blockquote>
</blockquote>
<p>well, if i do like this:</p>
<pre><code class="lang-auto">st = np.genfromtxt('123.txt',delimiter=';')
print(st)
</code></pre>
<blockquote>
<blockquote>
<blockquote>
<p>[1.00000e+00 0.00000e+00 5.12500e+02 … 2.00000e+01 0.00000e+00<br>
2.00000e+01]</p>
</blockquote>
</blockquote>
</blockquote>
<p>Missing comma between numbers.</p>

slavavs: <p>I did<br>
<code>t = torch.as_tensor(st, dtype=torch.float32)</code><br>
now everything is fine</p>

slavavs: <p>if i do the distribution of numbers so -&gt;<br>
<code>st = np.interp(3.4, [0,10], [-1,1])</code> = -0.31999999999999995</p>
<p>And if I do through the function np.genfromtxt -&gt;<br>
<code>st = np.interp(np.genfromtxt('123.txt',delimiter=';',dtype='d'), [0,10], [-1,1])</code> = [-0.6  -1.   -0.2  -0.6   0.4   0.    0.4  -1.    0.4   0.    0.2   0.9<br>
0.4   0.    0.2   0.95 -1.   -1.    0.2  -1.   -1.   -1.   -1.   -1.<br>
-1.   -1.   -0.6   0.6  -1.   -1.   -0.6  -1.  ]<br>
function round numbers. But for normalization, I need to know the full number. Help return values.</p>
TF LayerNormalization vs PyTorch LayerNorm

tnguyen: <p>In Tensorflow’s implementation of LayerNormalization <a href="https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/LayerNormalization" rel="nofollow noopener">here</a>, we can initialize it within the __init__ function of a module since it doesn’t require an input of the normalized shape already. I might be understanding this incorrectly, but PyTorch’s LayerNorm requires the shape of the input (output) that requires layer normalization, and thus since with each batch, I deal with different sequence lengths, which affects the shape of the output, I suppose I can’t initialize this within my own module?</p>
Problems about Siamese network

Steve_Hu: <p>recently i try to write a basic siamese network, i have finished the ‘training’ part and it works.but now i have a problem ,that is ,how can i get the accuracy.because i can’t get a label from the siamese network, i use contrastive loss and the model output just two vectors and i can only calculate their distance,rather than label.<br>
if there exists a threshold of distance can classify the sample pair  1 or 0, then how can i learn the threshold …or  there exist some methods to get labels?</p>
<p>anybody can help me? ^-^ thanks a lot !!</p>

JuanFMontesinos: <p>You can plot a ROC curve using sklearn library. This plot shows optimal threshold value.</p>
<p>Have a look at this post<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="32" height="32">
      <a href="https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5" target="_blank" title="04:10PM - 26 June 2018" rel="nofollow noopener">Towards Data Science – 26 Jun 18</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://cdn-images-1.medium.com/max/1200/1*q8VdYJGRO9RWbLi1TVeTuQ.jpeg" class="thumbnail"></div>

<h3><a href="https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5" target="_blank" rel="nofollow noopener">Understanding AUC - ROC Curve</a></h3>

<p>In Machine Learning, performance measurement is an essential task. So when it comes to a classification problem, we can count on an AUC …</p>

  <p><span class="label1">Reading time: 5 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

Steve_Hu: <p>Thank you! so you idea is viewing the threshold of distance as hyperparamters and through plotting the ROC  curve to find a best threshold?  i read the original siamese paper , it doesn’t mention the method of getting label , so is it the most popular method in siamese network?</p>

JuanFMontesinos: <p>Well, I don’t know if it’s the most popular method for siamese. It’s just a very traditional way of measuring performance. Basically that curve shows performance when you vary your threshold, then pick optimal point.<br>
You can use other plots like precision-recall.</p>

Steve_Hu: <p>ok , thanks you a lot !<img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:">  i’ll give it a try.</p>
A network has to trained with huge number ofdataset

pytorch_user12: <p>What is the best way to train a network with large database contain billions of samples ??<br>
When the same network is trained with a small dataset in GPU, its taking less time.<br>
But when the datsamples are increased its taking too much time irrespective of same number of batch size, no_worker and same dataloader also.<br>
I am not able to figure out, why the total batch size execution time is not matching with epoch time.<br>
I am just following this link <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a>.<br>
MY network is simple DNN.</p>
What is num_directions of nn.LSTM

kq3rHCt9gK: <p>I see <code>num_directions</code> in the document of nn.LSTM. But the doc does say what it means. Can anybody explain it? Thanks.</p>
<p><a href="http://pytorch.org/docs/master/nn.html" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/master/nn.html</a></p>

smth: <p>if the <code>bidirectional</code> option is set, <code>num_directions</code> is 2.</p>
<aside class="onebox wikipedia">
  <header class="source">
      <a href="https://en.wikipedia.org/wiki/Bidirectional_recurrent_neural_networks" target="_blank" rel="nofollow noopener">en.wikipedia.org</a>
  </header>
  <article class="onebox-body">
    <img src="//upload.wikimedia.org/wikipedia/en/thumb/6/6c/Wiki_letter_w.svg/40px-Wiki_letter_w.svg.png" class="thumbnail onebox-full-image" width="40" height="40">

<h3><a href="https://en.wikipedia.org/wiki/Bidirectional_recurrent_neural_networks" target="_blank" rel="nofollow noopener">Bidirectional recurrent neural networks</a></h3>

<p>Bidirectional Recurrent Neural Networks (BRNN) were invented in 1997 by Schuster and Paliwal. BRNNs were introduced to increase the amount of input information available to the network. For example, multilayer perceptron (MLPs) and time delay neural network (TDNNs) have limitations on the input data flexibility, as they require their input data to be fixed. Standard recurrent neural network (RNNs) also have restrictions as the future input information cannot be reached from the current state.  BR...</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Feng_Zuo: <p>Thanks you so much,I have same the problem too.</p>

Brando_Miranda: <p>what does direction &gt;= 3 mean?</p>

smth: <p>i think it can only be 1 or 2</p>
CyclicLR always between 1e-4 and 1e-2

Noonan: <p>Hi,<br>
I am trying to use the Cyclic on my model. I changed the optimizer from Adam to SGD due to (if I understood well) a bug in the last stable version of pytorch.<br>
The problem now is that no matter what base_lr and max_lr I take to initiate the scheduler, the learning rate fluctuates between 1e-4 and 1e-2.</p>

ptrblck: <p>Could you post a code snippet showing how you are initializing the optimizer and lr_scheduler so that we can have a look?</p>

Noonan: <pre><code class="lang-auto">opti = torch.optim.SGD(self.model_.parameters(), lr=self.lr, momentum=0.9)
scheduler = CyclicLR(optimizer, self.lr, 0.01, step_size_up=10)
</code></pre>
<p>With self.lr = 0.0001.</p>
The bug of torch.norm function?

xuchuan898: <p>Hello, I am using the torch.norm in my implementation (Pytorch 1.0.0a0+35c8f93). I noticed that when the tensor size turns to be huge, torch.norm does not return the answer that I want.</p>
<p>For example, I have a tensor of ones of size 4000000. The 2-norm of this tensor should be exactly sqrt(4000000) = 2000.</p>
<blockquote>
<p>import torch<br>
a = torch.ones([4000000])<br>
torch.norm(a) #<a href="https://pytorch.org/docs/stable/torch.html#torch.norm" class="inline-onebox" rel="noopener nofollow ugc">torch — PyTorch 2.1 documentation</a><br>
However, the function returns 22132.3770</p>
</blockquote>
<p>while</p>
<blockquote>
<p>torch.sum(a**2) did return the correct norm square of this tensor.</p>
</blockquote>
<p>Does anyone have the same issue using the norm function?<br>
Thanks in advance,</p>
<p>Chuan</p>

Pranavan_Theivendira: <aside class="quote no-group" data-username="xuchuan898" data-post="1" data-topic="48257">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xuchuan898/48/13204_2.png" class="avatar"> xuchuan898:</div>
<blockquote>
<p>Hello, I am using the torch.norm in my implementation (Pytorch 1.0.0a0+35c8f93). I noticed that when the tensor size turns to be huge, torch.norm does not return the answer that I want.</p>
</blockquote>
</aside>
<p>Hi,</p>
<p>I ran the same code in my windows machine and pycharm IDE. I am getting 2000. No issues from my side.</p>
<p>Thanks</p>

ptrblck: <p>Which PyTorch version are you using?<br>
It might be related to <a href="https://github.com/pytorch/pytorch/issues/15602" rel="nofollow noopener">this issue</a> which should be fixed by now in the latest stable version.</p>

xuchuan898: <p>I am using the version 1.0.0.   That’s the reason. Thanks a lot.</p>
Merging classes of the output of the semantic segmentation module

Nikronic: <p>Hi all,</p>
<p>My question is about the image segmentation task.<br>
I have a tensor with the size of <code>(batch_size, 150, height, width)</code> and the second number(<code>150</code>) corresponds to the number of classes.</p>
<p>Now I want to merge those classes into <code>25</code> classes by summing their probabilities.<br>
First of all, is this procedure correct? It seems it should be, for example I want to merge <code>river</code> and <code>lake</code> into one class.</p>
<p>Note: I have the source and target indexes to be merged. For example:</p>
<pre><code class="lang-python">target=[1, 2]
source= [[3,4],
         [5,6]]

# here for making things clear, I use a 1-D matrix
input = [1, 3, 4, 2, 5, 7, 6]

expected_output = [1, 1, 1, 2, 2, 7, 2] 
</code></pre>
<p>My question is, what is the optimal way to do this process?</p>
<p>My attempt: First, I tried to make a one hot encoded vector of each source index should be converted to target. Then use these vectors as masks then using linear multiplication to get aggregated value. And this process happens at the same time of all pixels.</p>
<p>By the way, I am still unsure what is the efficient way to achieve the goal using PyTorch available functions.</p>

ptrblck: <p>I’m not sure, if I understand the use case correctly, but would scattering the output logits using a mapping work?<br>
Here is a small example:</p>
<pre><code class="lang-python">batch_size = 10
nb_classes = 2
nb_features = 5
nb_out = 10

x = torch.randn(batch_size, nb_features)
target = torch.randint(0, nb_classes, (batch_size,))

mapping = torch.tensor([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])

model = nn.Linear(nb_features, nb_out)
criterion = nn.CrossEntropyLoss()

output = model(x)
output_small = torch.zeros(batch_size, nb_classes).scatter_add(1, mapping.unsqueeze(0).expand_as(output), output)
loss = criterion(output_small, target)
loss.backward()
print(model.weight.grad)
</code></pre>

Nikronic: <p><code>scatter_add()</code> is exactly the function I was looking for.</p>
<p>By the way, I have pretrained model and I do not need to train it. I just want to merge some different classes if its prediction before using <code>.max()</code> function to get the corresponding class.</p>
<p>Thank you so much.</p>
Implementation of Gram Matrix in Neural Style Tutorial

Nikronic: <p>Hi all,<br>
I want to implement Gram Matrix of a tensor with shape<br>
<code>(batch_size, channel_size, patch_size, height, width)</code></p>
<p>We can calculate it only by this code in numpy <code>A @ A.T</code>.<br>
My question is why in <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html#style-loss" rel="nofollow noopener">neural style transfer tutorial</a>, first we convert our matrix into 2D matrix, then use <code>.mm</code> function to multiply it by its transpose?</p>
<pre><code class="lang-python">def gram_matrix(input):
    a, b, c, d = input.size()  # a=batch size(=1)
    # b=number of feature maps
    # (c,d)=dimensions of a f. map (N=c*d)

    features = input.view(a * b, c * d)  # resise F_XL into \hat F_XL

    G = torch.mm(features, features.t())  # compute the gram product

    # we 'normalize' the values of the gram matrix
    # by dividing by the number of element in each feature maps.
    return G.div(a * b * c * d)
</code></pre>
<p>Is there any optimization consideration or anything else?</p>
<p>Bests</p>

ptrblck: <p>The tutorial explains it as:</p>
<blockquote>
<p>F_XL is reshaped to form F̂_hat_XL, a KxN matrix, where K is the number of feature maps at layer L and N is the length of any vectorized feature map F^k_XL.</p>
</blockquote>
<aside class="quote no-group" data-username="Nikronic" data-post="1" data-topic="46803">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>We can calculate it only by this code in numpy <code>A @ A.T</code> .</p>
</blockquote>
</aside>
<p>It doesn’t seem to be working and I’m not sure, what this operation calculates in your case:</p>
<pre data-code-wrap="python"><code class="lang-python">a, b, c, d = 2, 3, 4, 5
x = torch.randn(a, b, c, d)
x_np = x.numpy()
g_np = x_np @ x_np.T
&gt; ValueError: shapes (2,3,4,5) and (5,4,3,2) not aligned: 5 (dim 3) != 3 (dim 2)
</code></pre>

Nikronic: <p>Actually, It seems I asked my question ambiguously.<br>
By <code>A @ A.T</code>, I mean the definition of Gram matrix which is multiplication of a matrix to its transpose, not a numpy code. My question was why do we have to reshape matrices.Now, I can understand that the only way to multiply two 4-D matrices is to reshape them into 2-D matrices then multiply them (the <code>.t()</code> function only works on 2-D matrix, obviously).</p>
<p>Based on tutorial, <code>batch_size</code> and <code>channel_size</code> has been considered as <em>number of feature maps</em> and <code>height</code> and <code>width</code> of images as <em>length of any vectorized feature map</em></p>
<p>Now, in my problem, I have a 5-D matrix with a dimesion called <code>patch_size</code> and I considered it as <em>number of feature maps</em> that only can cause a difference in output matrix size, but not the values, so I think it will not cause any problem in computing loss process.</p>
<p>Thanks for help</p>
Request: call cudaDeviceReset() after assert

jphoward: <p>Currently after a cuda assert in a REPL such as jupyter notebook, it’s necessary to restart the REPL since cudaDeviceReset() isn’t called after the assert. This makes it rather hard to prototype in such an environment, since every bug requires a complete restart!</p>
<p>Is there any reason not to always reset after a cuda assert? If this isn’t a trivial change, perhaps you at least surface a python wrapper for cudaDeviceReset() we could call from python code?</p>

jphoward: <p>This issue has been coming up on forums.fast.ai a bit too so hopefully it’s OK to bump it since it’s been quite a while. <a class="mention" href="/u/apaszke">@apaszke</a> is this a reasonable idea?</p>

Shubajit_Saha: <p>This is a much needed feature for developing  pytorch models in jupyter. This issue is preventing incremental development which is the essence of notebooks so that there is no need to re execute the entire code.</p>
Getting torch.cuda.device_count() = 1 but still able to pass 2 as device argument in torch.cuda.device()

anubhav1772: <p>Hi,<br>
When I ran <code>torch.cuda.device_count()</code>, I got <code>1</code> in output but still able to pass <code>2</code>(which is greater than device count) as an argument in the <code>torch.cuda.device()</code> method. The documentation <a href="https://pytorch.org/docs/stable/cuda.html" rel="nofollow noopener">here</a> says its a context-manager that changes the selected device but I have just <code>1</code> cuda device available, i.e., ‘cuda:0’. Then, how am I not getting any error when I passed <code>2</code> as an argument in <code>torch.cuda.device()</code>?</p>
<pre><code class="lang-auto">import torch
print(torch.cuda.is_available())
print(torch.cuda.device_count())
x = torch.cuda.device(2)
print(x)
print(x.idx)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">True
1
&lt;torch.cuda.device object at 0x000001873064F2B0&gt;
2
</code></pre>
<p>And, what does <code>torch.cuda.device(device).idx</code> means? It’s value is same as what I am inputting as an argument in <code>torch.cuda.device()</code>.<br>
Please, help me undestand this. Thanks!</p>
Possible bug in seq2seq tutorial?

rajarsheem: <p>Hi, I was following the Seq2Seq tutorial from here:<br>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></p>
<p>While going through the forward of AttnDecoderRNN, I noticed this:<br>
attn_weights = F.softmax(<br>
self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)</p>
<p>Shouldn’t the attn_weights be computed by using the encoder_outputs? The weights depend on how much the current decoder hidden state matches with all the encoder states. Did I miss something?</p>
Gpu memory allocation error and volatility

kirk86: <p>Hi pytorch community. I was wondering if you could clarify a few things regarding gpu memory allocation. I have a particular model and during training I’ve noticed the following things. During training the gpu memory allocation is less than what the same model would allocate using tensorflow yet it always keep giving me an out of memory error. For instance the max gpu memory allocation from tensorflow is 10769MiB and for pytorch is 10011MiB. Any reason for that?</p>
<p>Besides that, the memory allocation during training keeps fluctuating from as less as 45% and as hight as 95% while in the same tensorflow model that is fixed and is not fluctuating.</p>
<p>Another thing is the fact that during training the gpu volatility of pytorch model is fluctuating a lot going from as less as 0%, sometimes even for a minute and then back up to 95%, while in the same model with tensorflow the gpu volatility is constantly at 100%.</p>
<p>Could someone shed some light on this please?</p>
<p>Thank you!</p>

gody7334: <p>Hi,<br>
I had similar issue and spent several days to figure out.<br>
In my case, I turned on torch.backend.cudnn.benchmark and it keeps evaluate on each batch which also causing memory fluctuating.</p><aside class="quote" data-post="1" data-topic="5936">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_zhang/48/45177_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936">What does torch.backends.cudnn.benchmark do?</a> 
  </div>
  <blockquote>
    whats the difference when setting it True or False?
  </blockquote>
</aside>

<p>Hope this will help.</p>
Normalization input

slavavs: <p>I have input - torch.Size([1, 1, 50000, 9])<br>
single line example - 1;0;9512;7507;5508;122;20;0;20<br>
(input values ​​are different, ranging from zero to 10,000)</p>
<p>How do I normalize such data at the network input?</p>

slavavs: <p>I had this idea:<br>
convert large numbers to two numbers. For example 7625.5 -&gt; 7.6; 25.5<br>
Here’s how to do it, I do not know. Do you have any ideas?</p>

ptrblck: <p>I assume the input does somehow consist of a number of samples (50000?) and some features (9?).<br>
If so, you could just calculate the mean and std for each feature and standardize it:</p>
<pre><code class="lang-python">z = (x - mean) / std
</code></pre>

slavavs: <p>I indicated the number 50,000 as an example. There is much more data. I will use a sliding window for data passing data.<br>
If we take the mean and std of all the data, then x will change very little on some data (columns). There is such data where your option will do.</p>
Creating a block diagonal matrix

LorenzoBi: <p>I need to create a block diagonal matrix, where the block are repeated on the diagonal many times. I would like to do something analogous to this numpy code</p>
<pre><code class="lang-auto">import numpy as np

S = np.arange(9).reshape((3, 3))
M = np.kron(np.eye(4), S)
M += np.kron(np.eye(4, 4, 1), S.T)
print(M)
</code></pre>
<p>Is there something similar in Pytorch or it is not yet implemented? Will it ever be implemented if that is the case</p>

yulkang: <p>I made a vectorized implementation on Gist: <code>block_diag.py</code> at <a href="https://gist.github.com/yulkang/2e4fc3061b45403f455d7f4c316ab168" rel="nofollow noopener">https://gist.github.com/yulkang/2e4fc3061b45403f455d7f4c316ab168</a></p>
<p>For an up-to-date version, check <code>block_diag()</code> in <code>numpytorch.py</code> in my <code>pylabyk</code> library: <a href="https://github.com/yulkang/pylabyk/blob/master/numpytorch.py" rel="nofollow noopener">https://github.com/yulkang/pylabyk/blob/master/numpytorch.py</a></p>
For ShuffleNetV2, the speed of inference in cpu is slow

HITerStudy: <p>Sorry, I encountered a problem. When I trained the model “shuffleNetv2” on GPU, I found it that the speed of training was very fast, having obvious advantages than other CNN models. However, the speed of inference on cpu was slower than that of other CNN models obviously, thus, I would know is there some problems in my experiments? Please give me some solutions if possible? Thanks a lot!</p>

AtenaNguyen: <p>Did you experience the same thing with ShuffleNet v1?<br>
ShuffleNet have splitting layers, which split the input to two parallel process.<br>
Additionally, ShuffleNetv2 was design to be GPU efficient and ARM architecture. I wonder it would hurt the CPU speed.</p>

HITerStudy: <p>Thanks a lot! I will perform the same operation with ShuffleNetV1. And, if using ShuffleNets on intel cpu, the speed cannot give any improvements? And try other lightweight models, only?</p>
Filter with probability

khe: <p>Is it possible to filter two same size matrix with probability?<br>
So for example, lets say m1 = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]<br>
and m2 is [[3, 4, 2], [5, 1, 2], [0, 4, 2]]<br>
and I want to combine these two matrix with a probability say 0.3, so element in m2<br>
has a probability of 0.3 to be in the output matrix m3. an example for output would be<br>
[[3, 2, 3], [4, 5, 2], [7, 4, 9]]<br>
is this possible with pytorch?</p>

Pranavan_Theivendira: <p>Hi,</p>
<p>Try the following solution.</p>
<pre><code class="lang-auto">import numpy as np
import torch

p = 0.3

m1 = torch.Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
m2 = torch.Tensor([[3, 4, 2], [5, 1, 2], [0, 4, 2]])

A = torch.Tensor(np.random.choice([0,1], size=[len(m1),len(m1[0])], p=[1-p, p]))

m3 = torch.mul(1-A,m1) + torch.mul(A,m2)

print(m3)
</code></pre>
<p>Thanks</p>
Efficient metrics evaluation in Training and Validation Loop

JimmysCheeseSteak: <p>I am new to PyTorch and want to efficiently evaluate among others F1 during my Training and my Validation Loop.</p>
<p>So far, my approach was to calculate the predictions on GPU, then push them to CPU and append them to a vector for both Training and Validation. After Training and Validation, I would evaluate both for each epoch using sklearn. However, profiling my code it showed, that pushing to cpu is quite a bottleneck.</p>
<pre><code class="lang-python">for epoch in range(n_epochs):
    model.train()
    avg_loss = 0
    avg_val_loss = 0
    train_pred = np.array([])
    val_pred = np.array([])
    # Training loop (transpose X_batch to fit pretrained (features, samples) style)
    for X_batch, y_batch in train_loader:
        scores = model(X_batch)
        y_pred = F.softmax(scores, dim=1)
        train_pred = np.append(train_pred, self.get_vector(y_pred.detach().cpu().numpy()))

        loss = loss_fn(scores, self.get_vector(y_batch))
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        avg_loss += loss.item() / len(train_loader)

    model.eval()
    # Validation loop
    for X_batch, y_batch in val_loader:
        with torch.no_grad():
            scores = model(X_batch)
            y_pred = F.softmax(scores, dim=1)
            val_pred = np.append(val_pred, self.get_vector(y_pred.detach().cpu().numpy()))
            loss = loss_fn(scores, self.get_vector(y_batch))
            avg_val_loss += loss.item() / len(val_loader)

    # Model Checkpoint for best validation f1
    val_f1 = self.calculate_metrics(train_targets[val_index], val_pred, f1_only=True)
    if val_f1 &gt; best_val_f1:
        prev_best_val_f1 = best_val_f1
        best_val_f1 = val_f1
        torch.save(model.state_dict(), self.PATHS['xlm'])
        evaluated_epoch = epoch

    # Calc the metrics
    self.save_metrics(train_targets[train_index], train_pred, avg_loss, 'train')
    self.save_metrics(train_targets[val_index], val_pred, avg_val_loss, 'val')
</code></pre>
<p>I am certain there is a more efficient way to a) store the predictions without having to push them to cpu each batch. b) calculate the metrics on GPU directly?</p>
<p>I also posted this question on <a href="https://stackoverflow.com/questions/56643503/efficient-metrics-evaluation-in-pytorch" rel="nofollow noopener">stackexchange</a>, but thought that this forum is probably even better <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
As I am new to PyTorch, I am very grateful for any hints and feedback <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Releasing GPU memory after deleting model

aronl: <p>I’m experiencing some trouble with the GPU memory not being released after deleting a model.</p>
<p>The pseudo-code looks something like this:</p>
<pre><code class="lang-auto">for _ in range(5):
    data = get_data()
    model = MyModule() ### PyTorch model
    results = model(data)
    del model
    torch.cuda.empty_cache()
</code></pre>
<p>The model occupies around 4 GB of GPU memory and in the second iteration this code crashes on my 8GB GPU with the error</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 3.73 GiB (GPU 0; 7.93 GiB total capacity; 3.73 GiB already allocated; 3.54</code></p>
<p>I’m probably misunderstanding something here, but I thought the <code>del</code> operation together with <code>empty_cache()</code> would free up the memory.</p>
<p>Thanks for any input. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Hi,</p>
<p>The thing is that <code>results</code> most likely requires gradients. And these gradients would require most of the model parameters to be computed. And so the model cannot be freed yet <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

aronl: <p><img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=9" title=":man_facepalming:" class="emoji" alt=":man_facepalming:"> That was indeed it.<br>
Thanks for your prompt reply! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Siamese network not converging

Nilabhra: <p>I have implemented a Siamese network for text similarity. Here is something I observed. When I feed the two sequence batches (one batch of left sequences and another batch of right sequences in separate autograd vars) separately to the LSTM and then compute similarity on the last hidden state of the output, the model works just fine. But If I feed in both the text sequences as a single input batch and then use alternative indexing to separate out the first and second set of sequences (even indices are the left sequence and the odd indices are the right sequence) and then compute similarity, the model doesn’t converge and the outputs are random. Am I somehow losing the gradients when I am slicing the hidden state variable?</p>

apaszke: <p>What version are you running on?</p>

n0obcoder: <aside class="quote no-group" data-username="Nilabhra" data-post="1" data-topic="997">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nilabhra/48/8492_2.png" class="avatar"> Nilabhra:</div>
<blockquote>
<p>indic</p>
</blockquote>
</aside>
<p>Bro , i am also trying to implement Siamses net for face images. Now , Once i get the feature vectors of the images, after passing the images thriough the CNN, do i have to take the sigmoid of the feature vectors, in order to bound the vector? and how do i chose the value of margin in my Triplet Loss? Pls help if u also had to face similar challenges ! Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
Is there any way to compute the Hessian Matrix of a nn.Module w.r.t its (nn.Module's) parameters?

phantom90: <p>For one of my tasks, suppose we have a module M parameters by $\theta$.<br>
$\theta$ denotes a collections of parameters, such as the parameters of a linear layer, within the module M.</p>
<p>Is there a way to calculate:</p>
<p>H_{\theta}</p>
<p>i.e., the Hessian matrix w.r.t $\theta$? Any suggestions or references are welcomed.</p>
<p>Thanks very much ~</p>
Is there any way to convert saved weight data type from torch.FloatTensor to torch.HalfTensor

JetBlock: <p>I trained the model in Dtype torch.cuda.FloatTensor, now I want to load the checkpoint and test in Dtype torch.cuda.HalfTensor.<br>
Must I train the model again? or how can I conver checkpoint weight’s Dtype from float32 to float16</p>

Pranavan_Theivendira: <p>Hi,</p>
<p>Similar question has been asked and discussed previously in these forums. You can have a look here.</p>
<aside class="quote" data-post="1" data-topic="2713">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alan_ayu/48/374_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-cast-a-tensor-to-another-type/2713">How to cast a tensor to another type?</a> 
  </div>
  <blockquote>
    if I got a float tensor,but the model needs a double tensor.what should I do to cast the float tensor to double tensor?
  </blockquote>
</aside>

<p>Thanks</p>

JetBlock: <p>thank you for your reply, I have solved this question by your link.</p>
Why is the shape of hidden num_layers * num_directions?

pinocchio: <p>I was seeing the <a href="https://pytorch.org/docs/stable/nn.html#lstm" rel="nofollow noopener">LSTM</a> and was wondering why the <code>hidden = (h_n,c_n)</code> was <code>(num_layers * num_directions, batch, hidden_size)</code>. In particular why is it not:</p>
<p><code>(num_layers * num_directions, seq_len, batch, hidden_size)</code></p>

pinocchio: <p>Oh I realized that its only the first RNN block that needs a hidden layer…so for each sequence in the batch we need a hidden state. So how many initial hidden states do we need? One for each sequence batch and one for each direction and one for each each layer that we have. Which gives <code>num_layers * num_directions * batch</code></p>
<p>but each has <code>hidden_size</code> dimension. So we get:</p>
<p><code>(num_layers * num_directions, batch, hidden_size)</code></p>
Spline Interpolation on 1D arrays

catubc: <p>Hi everyone. I’m trying to replace a scipy.resample() call in my code with an equivalent in pytorch.  My code just takes a time series trace and can upsamples it using scipy.signal.resample currently.</p>
<pre><code class="lang-auto">upsample_rate = 10
length = 100
trace = np.random.rand(length)
trace_upsampled = signal.resample(trace,length*upsample_rate)
</code></pre>
<p>I need to carry out this operation over multiple traces at a time.  Is there an equivalent in  torch?  I assume it would involve some kind of fast fourier transform, but the nn.interpolate functions all seem linear to me.</p>
<p>Thanks!</p>
CPU Inference time varies depending on checkpoint

Carl: <p>Hi!</p>
<p>I’m confused to see that two weight checkpoints from the same network can have very different inference times on CPU. I’ve seen cases where one checkpoint makes inference consistently 2X slower than another checkpoint. I have not seen this kind of inconsistency on GPU.</p>
<p>I have prepared a minimal working example for which w2.pth makes inference 1.5X slower than w1.pth.<br>
<a href="https://github.com/lemairecarl/paratope-pred/archive/cpu-perf-tests.zip" rel="nofollow noopener">https://github.com/lemairecarl/paratope-pred/archive/cpu-perf-tests.zip</a></p>
<p>From my investigation, it seems that checkpoints taken earlier in training are faster to do inference with. Does that make sense? Is there some magic optimization tricks which I’m not aware of?</p>
<p>All the best,<br>
Carl</p>

ptrblck: <p>Hi Carl,</p>
<p>thanks for the nice script to reproduce this issue.<br>
You are seeing the time difference most likely due to denormal values (float values close to zero), which are apparently slower to process.<br>
If you set <code>torch.set_flush_denormal(True)</code>, you’ll see that both checkpoints are faster to process and have approx. the same time now.</p>

Carl: <p>Thanks for your answer. That’s the kind of obscure quirk I was hoping to learn about. I assume this is not documented. Are there other such undocumented quirks?</p>

ptrblck: <p>I’m not sure if it’s documented and I know of its existence, because I also tried to debug this issue for some time. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="Carl" data-post="3" data-topic="47973">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/carl/48/11468_2.png" class="avatar"> Carl:</div>
<blockquote>
<p>Are there other such undocumented quirks?</p>
</blockquote>
</aside>
<p>That’s hard to tell, but please post anything you find suspicious. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Softmax over H * W

crystal: <p>I want to do softmax over H * W, but the official <code>Softmax2d</code> do softmax with C * H * W.<br>
So I do this by using <code>torch.sum</code> like this,</p>
<pre><code class="lang-python">&gt;&gt;&gt; a = torch.rand(2, 1, 3, 3)
&gt;&gt;&gt; a

(0 ,0 ,.,.) =
  0.2937  0.7227  0.8050
  0.1798  0.3219  0.9238
  0.1541  0.4280  0.3620

(1 ,0 ,.,.) =
  0.1013  0.3864  0.6033
  0.2719  0.9204  0.4946
  0.2204  0.7962  0.8410
[torch.FloatTensor of size 2x1x3x3]

&gt;&gt;&gt; aa = a.div(torch.sum(torch.sum(a, dim=2, keepdim=True), dim=3, keepdim=True))
&gt;&gt;&gt; aa

(0 ,0 ,.,.) =
  0.0701  0.1724  0.1921
  0.0429  0.0768  0.2204
  0.0368  0.1021  0.0864

(1 ,0 ,.,.) =
  0.0219  0.0834  0.1301
  0.0586  0.1986  0.1067
  0.0475  0.1718  0.1814
[torch.FloatTensor of size 2x1x3x3]
</code></pre>
<p>I want to know is there a more efficient or concise way to do this?</p>

tom: <p>How about using .view to feed it into a stock softmax?</p>
<p>Best regards</p>
<p>Thomas</p>

moucheng: <p>could you please elaborate it?</p>

tom: <p><code>torch.softmax(x.reshape(b, c, h * w), -1).view(b, c, h, w)</code></p>