akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pretraining exsiting bert model with new features (continous variables such as time & temp)

rahulrajdevaraja: <p>I need a suggestion of using an existing bert model which is pre-trained for sentence classification.</p>
<p>the existing model accepts text in form of: “ClC1=CC=CC(Cl)=C1C=O.ClC1=CC=CC(Cl)” which is a chemical reaction for example. Now to enhance the model where I can use continuous values such as time and temperature as features to it to retrain the model on my dataset.</p>
<p>my idea is to use an ensemble approach, where I take the last layer output values of the existing bert model, create another model to just accept the continuous variables and concat them in an ensemble approach similar to this link: <a href="https://discuss.pytorch.org/t/combining-trained-models-in-pytorch/28383" class="inline-onebox">Combining Trained Models in PyTorch</a></p>
<p>Any alternate approaches or suggestions? or links to use as a resource for this implementation</p>
<p>Thanks<br>
Rahul Raj Devaraja</p>
Cuda has memory error doing simplest calculation

Tianyi_Zhou: <p>Hello, I am a college student and pretty new to everything. I just got a Ubuntu laptop and set up CUDA for Pytorch, and it seems CUDA has no memory allocation.Showing me “CUDA error: Out of Memory”</p>
<p>Attached is the screenshot for my gpu.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df805cc0a5497d3969d7212e3c4fbabb9dedf7e2.png" data-download-href="https://discuss.pytorch.org/uploads/default/df805cc0a5497d3969d7212e3c4fbabb9dedf7e2" title="Screenshot_20201227_201325"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df805cc0a5497d3969d7212e3c4fbabb9dedf7e2.png" alt="Screenshot_20201227_201325" data-base62-sha1="vTbur1nrKCMRBtV5AaEcyuFhyDw" width="690" height="448" data-dominant-color="646667"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20201227_201325</span><span class="informations">754×490 58.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>  Please help， thanks!!</p>

1chimaruGin: <aside class="quote no-group" data-username="Tianyi_Zhou" data-post="1" data-topic="107294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tianyi_zhou/48/32795_2.png" class="avatar"> Tianyi_Zhou:</div>
<blockquote>
<p>CUDA error: Out of Memory</p>
</blockquote>
</aside>
<p>(common cause) I think your model allocate the Max GPU memory, you’ll need to reduce batch size.<br>
btw, could you please tell me what model did you train and batch size?</p>
<p>else follow this link<br>
<a href="https://pytorch.org/docs/stable/notes/faq.html#:~:text=My%20model%20reports%20%E2%80%9Ccuda%20runtime,these%20cases%20are%20often%20simple" class="inline-onebox" rel="noopener nofollow ugc">Frequently Asked Questions — PyTorch 2.1 documentation</a>.</p>
Question in matlab

randinoo: <p>I tested this code</p>
<pre><code class="lang-auto">g = [ 0.0113  0.0838 0.0113 ; 0.0838  0.6196 0.0838 ; 0.0113  0.0838 0.0113]
     

 
 for i = 1: 400    
   for j=1 : 400
           
       B(i,j) = (abs((((log(b.B1(i,j)))) -((log(a.B2(i,j)))))*g))
          
      end
  end
</code></pre>
<p>error</p>
<pre><code class="lang-auto">Subscripted assignment dimension mismatch.
</code></pre>
<p>help me plz!!</p>

ptrblck: <p>You landed in the PyTorch forum, so I would recommend to post this question in a MATLAB-specific discussion board. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

randinoo: <p>okay thanks ptrblck <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
Unable to install pytorch 1.7.1

mohit117: <p>Hello,<br>
I created a fresh conda environment and ran the following code,</p>
<p><code>conda install pytorch torchvision torchaudio cudatoolkit=10.0 -c pytorch</code> on ubuntu 16.04 LTS. Why did this install pytorch version 1.4.0 and not the latest pytorch 1.7.1?</p>
<p>I want to install the latest version.</p>

InnovArul: <aside class="quote no-group" data-username="mohit117" data-post="1" data-topic="107136">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohit117/48/13447_2.png" class="avatar"> mohit117:</div>
<blockquote>
<p>conda install pytorch torchvision torchaudio cudatoolkit=10.0 -c pytorch</p>
</blockquote>
</aside>
<p>I am not sure if pytorch is available with cuda 10.0.</p>
<p>Can you try with<br>
<code>conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch</code> as given in <a href="https://pytorch.org/">https://pytorch.org/</a>?</p>

mohit117: <p>Yes I tried that immediately after and it was installing the latest one. So may be I have to update cuda drivers. Thankyou</p>

ptrblck: <p>The CUDA drivers are needed to run the CUDA runtime, which ships with the PyTorch binaries. For the latest PyTorch version you can install the CUDA runtime in version 9.2, 10.1, 10.2, and 11.0 as <a class="mention" href="/u/innovarul">@InnovArul</a> described.</p>
Self supervised representation learning - concat tensor to input, sample dependant

enterthevoidf22: <p>how can i concatenate to the input tensor another tensor that is initialized randomly and then updated with back-prop. the purpose is to have some unsupervised byproduct representation.</p>
<p>generally i understand that i can do this with simply adding it to the input before feeding the network and requiring grad.</p>
<p>BUT, how can i initialize it at the beginning and then throughout the training it is only updated and not re-initialized?</p>
<p>ALSO, how can i assign a few samples (being part of some category the network is not aware of initially) to the same concatenated tensor?</p>
<p>a more formal explanation of what i want:</p>
<p>samples = {X1,X2,…Xn-1,Xn}<br>
X1.shape = (1,64,64,32) (channels, width, height,depth)<br>
initial extra channel = torch.rand((1,64,64,32))<br>
extras = {R1,R2,…Rk-1,Rk}<br>
<span class="hashtag">#notice</span> k extras because k&lt;n is the hidden category number</p>
<p>now randomly the loader loads sample X134 (out of n), which belongs to category 6 (out of k) - so the network f gets as input the tensor Z134<br>
Z134 = torch.cat([X134,R6],dim=0)</p>
<p>then during backprop R6 is updated and not initialized randomly until the end of training.</p>
<p>hope anyone can help, thanks in advance!!!</p>
Validation loss and metric not reasonable

moreshud: <p>Hi all,</p>
<p>I ran my code for a segmentation task and the resulting output of the validation loss and score compared to the training is not reasonable. Here’s the output:</p>
<pre><code class="lang-auto">Epoch: 1/10 Training | pathologies - loss: 1.4878 score: 0.7301 lunglobes - loss: 3.6833 score: 0.5535 Validation | pathologies - loss: 2.1480 score: 0.6627 lunglobes - loss: 4.7785 score: 0.3517 |Time: 0:03:06

Training   | pathologies: [0.8714 0.1824 0.2388], Lunglobes: [0.8764 0.122  0.0881 0.1137 0.1    0.118 ]
Validation | pathologies: [0.9067 0.0004 0.0193], Lunglobes: [0.8082 0.0519 0.0288 0.0347 0.0811 0.0519]

Validation loss decreased (inf --&gt; 3.321953).  Saving model ...
Epoch: 2/10 Training | pathologies - loss: 0.9487 score: 0.9429 lunglobes - loss: 2.7038 score: 0.8808 Validation | pathologies - loss: 2.0427 score: 0.9951 lunglobes - loss: 4.6012 score: 0.9285 |Time: 0:02:43

Training   | pathologies: [0.9637 0.3579 0.4035], Lunglobes: [0.9765 0.2197 0.1402 0.222  0.1157 0.257 ]
Validation | pathologies: [0.9872 0.0016 0.0278], Lunglobes: [0.9646 0.0958 0.0952 0.0498 0.0964 0.0754]

Validation loss decreased (3.321953 --&gt; 3.277506).  Saving model ...
Epoch: 3/10 Training | pathologies - loss: 0.7948 score: 0.9630 lunglobes - loss: 2.2768 score: 0.9454 Validation | pathologies - loss: 2.0350 score: 0.9624 lunglobes - loss: 4.5200 score: 0.9001 |Time: 0:02:43

Training   | pathologies: [0.9771 0.4909 0.4688], Lunglobes: [0.9859 0.2769 0.2207 0.2997 0.1757 0.3169]
Validation | pathologies: [0.9839 0.0016 0.0473], Lunglobes: [0.961  0.1586 0.1008 0.0494 0.0877 0.1052]

Epoch: 4/10 Training | pathologies - loss: 0.7234 score: 0.9670 lunglobes - loss: 1.8668 score: 0.9572 Validation | pathologies - loss: 2.1716 score: 0.8341 lunglobes - loss: 4.8151 score: 0.7460 |Time: 0:02:42

Training   | pathologies: [0.9806 0.5481 0.5065], Lunglobes: [0.9886 0.309  0.2769 0.3348 0.239  0.3806]
Validation | pathologies: [0.9201 0.0007 0.017 ], Lunglobes: [0.863  0.0596 0.0854 0.038  0.0362 0.0679]

Epoch: 5/10 Training | pathologies - loss: 0.6755 score: 0.9701 lunglobes - loss: 1.6758 score: 0.9631 Validation | pathologies - loss: 2.0456 score: 0.9831 lunglobes - loss: 4.5550 score: 0.9199 |Time: 0:02:42

Training   | pathologies: [0.983  0.5677 0.536 ], Lunglobes: [0.9894 0.3563 0.3296 0.3845 0.277  0.3667]
Validation | pathologies: [0.9889 0.0019 0.0296], Lunglobes: [0.9713 0.1499 0.1192 0.0417 0.065  0.1274]

</code></pre>
<p>I am using a custom weighted dice + cross-entropy loss with dice metric as the evaluation metric as well as Adam optimizer. Your contributions are well appreciated.</p>
How to concat data to minimize zero padding in data processing? How to do dynamic batch size?

hadaev8: <p>For example, i have data like [1, 2, 3, 4, 5], [1, 2, 3], [1, 2]<br>
Usually, I concat it into one tensor with zero padding.<br>
Like [[1, 2, 3, 4, 5], [1, 2, 3, 0, 0], [1, 2, 0, 0, 0]]</p>
<p>But this should waste gpu utilization.<br>
A better solution should be to contact short samples into one and add some other samples to maintain batch size.<br>
Like [[1, 2, 3, 4, 5], [1, 2, 3, 1, 2], [1, 2, 3, 4, 0]]</p>
<p>Another thing is dynamic batch size for full utilization of gpu memory.<br>
For example with max data len in batch 100 I should have 32 batch size and for max data len in batch 200 I should have 16 batch size.</p>
<p>But I not sure how to approach these problems.</p>
Time and memory complexity

mohit117: <p>Suppose I have 9 images that have to be processed by a neural network <code>NN</code>. I have two choices.</p>
<ul>
<li>Sequentially process 9 images by <code>NN</code> one after the another with batch dim 1.</li>
<li>Stack all images along the batch dimension  i.e batch size becomes 9.</li>
</ul>
<p>Certainly, the output for each  image remains the same in the two cases but which is better in terms of execution time and memory consumption.</p>
<p>Does the answer vary w.r.t to execution on CPU and GPU?</p>
<p>Here I am just concerned about inference statistics i.e NN will run with <code>torch.no_grad()</code> wrapper.</p>

dazzle-me: <p>Usually we prefer to use convenient batch-size from 8 to 32 and sometimes even more, but it can affect training.</p>
<p>In case of GPU usage with large batch size you can run into VRAM limitations.</p>
<p>This is because for each training sample we have to store some tensors, which will be used during backpropogation, and hence amount of memory often depends linearly on the batch size (if you are working on Linux, for example Ubuntu, you can test VRAM consumption during training by calling <code>watch nvidia-smi</code> in terminal).</p>
<p>To the question of time : you can test both perfomances using simple <code>%%timeit</code> function, in case you are working in Jupyter Notebook (as I do), otherwise, just use <code>time()</code> function and log the results.</p>
<p>Now you can try to research a little, and please, report back when you’re done <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>My predictions is that case with <code>batch size = 9</code> will be slightly faster than the other one.</p>

mohit117: <p>Thankyou indeed batch=9 will be memory heavy. And in question i just meant the inference statistics and training can be ignored. I have edited the question.</p>
Compute latent representation of a sentence with a transformer

Jimmy2027: <p>Hi, I am trying to build a variational autoencoder with pytorch and would like to use a transformer for the encoder and the decoder. But I’m not sure how to do that.<br>
I have implemented the tokenization of my dataset, meaning that each word of a sentence is translated into a number.<br>
The input of my encoder would be a tensor of shape (batch_size, sent_length) containing integers between 0 and the number of different words I have in my dataset.</p>
<p>How can I build a transformer that creates from this input a latent representation of shape (batch_size, latent_dim)? Is there any tutorial for this use-case?</p>

Abhilash_Srivastava: <p>I am wondering if you use a <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html#transformerencoder" rel="noopener nofollow ugc">TransformerEncoder</a> followed by a <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html#transformerdecoder" rel="noopener nofollow ugc">TransformerDecoder</a>, will it help your case?<br>
The output of the TransformerEncoder could be the latent representation in this case. This <a href="https://ieeexplore.ieee.org/document/8852155" rel="noopener nofollow ugc">paper</a> maybe useful.</p>
How to deal with dataset with different size image

Yu_Cao: <p>As the question,I’m building a CNN, I got a dataset with different size images, for example size=198 * 256, size = 210 * 220, etc. I want use <code>tt.RandomCrop</code> to improve my model, but I’m confused what size I should take in <code>tt.RandomCrop</code>, should I zoom those picture to a fixed size or do something else. I’m just starting learning deep learning, any kind of hint is welcome.</p>

dazzle-me: <p>Well, it is almost always the case when you want to resize your images to a concrete shape,<br>
for example <code>224x224</code> (worth mentioning, that it is highly depends on which size your test images have).</p>
<p>I’ve used resizing too, when I encountered datasets with different image shapes.</p>
<p>By doing so, now you can clearly understand, which shape an input has and build your network.</p>
<p>For resizing you can you <code>image = cv2.resize(image, new_size)</code>, see also<br>
<a href="https://docs.opencv.org/master/da/d54/group__imgproc__transform.html#ga47a974309e9102f5f08231edc7e7529d" rel="noopener nofollow ugc">OpenCV resize function documentation</a></p>
Constraints to kernels during training

Sherzod_Bek: <h2>Hello,<br>
is it possible to put constraints to layer’s kernel…<br>
For example: in last layer I used 5x5 kernel, but I want to make it symmetric.</h2>
<h2>like this:<br>
1 2 3 2 1<br>
2 4 5 4 2<br>
3 5 6 5 3<br>
2 4 5 4 2<br>
1 2 3 2 1</h2>
<p>if I try to change weight manually, I got error about “something related to gradient”…<br>
is there any way to make kernel shape symmetric?</p>
Estimate running stats from a different input for batch norm

deJQK: <p>I encounter the case that the input to some batch norm layer are the same across a batch, and this makes the running stats too small and training not stable. I wonder if it is possible to implement with functions in pytorch using a different input to estimate the running stats. The function of batch norm seems to be implemented in C (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L2033" class="inline-onebox" rel="noopener nofollow ugc">pytorch/functional.py at master · pytorch/pytorch · GitHub</a>).</p>

deJQK: <p>Following this discussion (<a href="https://discuss.pytorch.org/t/implementing-batchnorm-in-pytorch-problem-with-updating-self-running-mean-and-self-running-var/49314/2" class="inline-onebox">Implementing Batchnorm in Pytorch. Problem with updating self.running_mean and self.running_var - #2 by SeoHyeong</a>), it is implemented as the following:</p>
<pre><code class="lang-auto">class SepBN2d(nn.BatchNorm2d):
    def __init__(self, num_features, eps=1e-5, momentum=0.1,
                 affine=True, track_running_stats=True):
        super(SepBN2d, self).__init__(
            num_features, eps, momentum, affine, track_running_stats)
        self.multiple_input = True

    def forward(self, input):
        input, input_for_stats = input
        self._check_input_dim(input)
        if input_for_stats is None:
            input_for_stats = input
        self._check_input_dim(input_for_stats)

        exponential_average_factor = 0.0

        if self.training and self.track_running_stats:
            if self.num_batches_tracked is not None:
                self.num_batches_tracked += 1
                if self.momentum is None:
                    exponential_average_factor = 1.0 / float(self.num_batches_tracked)
                else:
                    exponential_average_factor = self.momentum

        if self.training:
                        mean = input_for_stats.mean([0, 2, 3])
            var = input_for_stats.var([0, 2, 3], unbiased=False)
            n = input.numel() / input.size(1)
            with torch.no_grad():
                self.running_mean = exponential_average_factor * mean + (1 - exponential_average_factor) * self.running_mean
                self.running_var = exponential_average_factor * var * n / (n - 1) + (1 - exponential_average_factor) * self.running_var
        else:
            mean = self.running_mean
            var = self.running_var

        input = (input - mean[None, :, None, None]) / (torch.sqrt(var[None, :, None, None] + self.eps))
        if self.affine:
            input = input * self.weight[None, :, None, None] + self.bias[None, :, None, None]

        return input
</code></pre>
<p>Thanks.</p>
Running multiple versions of DistributedDataParallel in bash script

btil: <p>I am hyperparameter searching and want to utilize all GPU’s with DistributedDataParallel. Let’s say all my distributed training code is in main.py. I want to test different hyperparmeters in a bash script like this so that I can train both models at once:</p>
<pre><code class="lang-auto">python main.py --param1 45 --param2 20 &amp;
python main.py --param1 33 --param2 41
</code></pre>
<p>Should I expect any funny business from DistributedDataParallel training like this?</p>

mrshenli: <p>Assuming you <code>main.py</code> script assigns different <code>MASTER_PORT</code> for different experiments (so that processes in the sample experiment can successfully rendezvous).</p>
<p>It should work with GLOO backend. But if you are using NCCL backend, it could hang, because NCCL requires using one communicator per device at a time. If there are multiple processes operating on the same CUDA device, DDP instances in different processes might launch <code>AllReduce</code> concurrently. If you are lucky and the experiments do not hang, the result should still be correct I think.</p>
How to compute matrix multiplication of multidimensional matrices efficiently

Fly2Fly: <p><strong>Regular matrix multiplication:</strong></p>
<ul>
<li>
<p>If I have N1 samples and N2 samples, their dimensions are both D.<br>
X1 = [N1,D], X2 = [N2,D]<br>
Calculate the similarity matrix between samples, I can use <code>S = X2.mm(X1.T)</code>, where S = [N2,N1]</p>
</li>
<li>
<p>But if X1 = [B,N1,D], X2 = [B,N2,D], and the B notes bathsize,<br>
If I want batch-wise calculation of matrix multiplication,<br>
the below is a for-loop version,<br>
how to do it efficiently?</p>
</li>
</ul>
<pre><code class="lang-python">def MatrixMulti(A,B):
    """
    The first dimension is independent 
    and computes matrix multiplication 
    in the remaining dimensions
    inputs:
        A : [b,N1,D]
        B : [b,N2,D]
    output:
        S : [b,N2,N1]
    """

    b,N1,D = A.size()
    b,N2,D = B.size()
    S = torch.zeros((b,N2,N1))
    for i in range(b):
        # [Rb,d] * [d,Ra]
        S[i,:,:] = B[i,:,:].mm(A[i,:,:].transpose(0,1))
    return S
</code></pre>

KFrank: <p>Hi Fly!</p>
<aside class="quote no-group" data-username="Fly2Fly" data-post="1" data-topic="107209" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fly2fly/48/23386_2.png" class="avatar"> Fly2Fly:</div>
<blockquote>
<ul>
<li>But if X1 = [B,N1,D], X2 = [B,N2,D], and the B notes bathsize,<br>
If I want batch-wise calculation of matrix multiplication,</li>
</ul>
</blockquote>
</aside>
<p>Use <a href="https://pytorch.org/docs/1.7.0/generated/torch.bmm.html#torch.bmm" rel="noopener nofollow ugc">torch.bmm()</a> (“batch matrix-matrix”), together with <a href="https://pytorch.org/docs/1.7.0/generated/torch.transpose.html#torch.transpose" rel="noopener nofollow ugc">torch.transpose()</a><br>
to line up the <code>D</code> dimensions properly:</p>
<pre data-code-wrap="python"><code class="lang-python">S = torch.bmm (X1, torch.transpose (X2, 1, 2))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Fly2Fly: <p>That is brilliant thank you!</p>
How can I replicate this tensor.unfold operation with F.unfold?

ProGamerGov: <p>I need the dilation parameter, and tensor.unfold lacks it.</p>
<pre><code class="lang-auto">import torch


x = torch.randn(1,32,16,16)

kernel_size=(3,3)
dilation = 1
stride=(1,1)

x_out = x.unfold(2, kernel_size[0], stride[0]).unfold(3, kernel_size[1], stride[1])
print(x_out.shape)


x_out = torch.nn.functional.unfold(x, kernel_size=kernel_size, dilation=dilation, padding=0, stride=stride)
print(x_out.shape)
</code></pre>
<p>nn.MaxPool2d has a dilation value of 1, and I’m using the above to make AvgPool2d match MaxPool2d’s parameters.</p>
<p>Edit: Looks like I don’t need it after all.</p>
Detecting the number of objects in an image

FaultyBagnose: <p>I’m training a mushroom image classifier, and need to identify images with more than one mushroom. Would this require full object detection or can it be done in a simpler way?</p>
<p>Btw, I’m looking into the VGG Image Annotation Tool for bounding box annotation. Is this a good choice or is there something simpler out there for this specific task?</p>
How to split a tensor based on a list, then get the average along dim=0?

Harris_Yin: <p>Hi,</p>
<p>Suppose there is a tensor with shape of [M,N,F], and a list of segement size (the sum is M, the length is K). After splitting the tensor into a K-length tuple T, is there a way to get the average of each tensor in the tuple T along dim=0 without using a loop? Since the size of dim=0 for each tensor [K_i, N, F] is different, I could not stack it or apply torch.mean() directly.</p>
<p>Thanks!</p>

Emre_Yalcinoglu: <p>You can get the tuple by using <a href="https://pytorch.org/docs/stable/generated/torch.split.html" rel="noopener nofollow ugc"><code>torch.split</code></a>, and then define a function:<br>
<code>f = lambda x : torch.mean(x, dim=0)</code>, which you can use on your tuple <code>T</code> as<br>
<code>list(map(f,T))</code>, to get the output that you want.</p>

Harris_Yin: <p>Thanks, one more question regarding the lamba and map function. Will it slow down the tensor operation, compared to the case that each chunk has the same size of dim=0 so the tensor can be reshaped and get the mean?</p>

Emre_Yalcinoglu: <p>I am afraid, I can’t answer that, besides not understanding your question (what’s being reshaped to what?), I also don’t know much about bench-marking.</p>
<p>However, I did a quick test (please see below) after reading <a href="https://stackoverflow.com/questions/1247486/list-comprehension-vs-map" rel="noopener nofollow ugc">List comprehension vs map</a> and indeed, for this case too, having a loop and using list comprehension is faster than using lambda expression and map.</p>
<pre><code class="lang-auto">import torch
import timeit
import numpy as np

torch.manual_seed(0)
np.random.seed(0)

M, N, F, K = 50, 30, 30, 10
N = 100

t0 = torch.randint(0,N,(M,N,F), dtype=torch.float)
split = list(np.random.multinomial(M, np.ones(K)/K, size=1)[0])
split_stab = list((np.ones(K)*(M/K)).astype(int))

def list_comp():
    return [torch.mean(t, dim=0) for t in torch.split(t0,split)]

def map_lambda():
    return list(map(lambda x : torch.mean(x, dim=0), torch.split(t0,split)))

def list_comp_stab():
    return [torch.mean(t, dim=0) for t in torch.split(t0,split_stab)]

def map_lambda_stab():
    return list(map(lambda x : torch.mean(x, dim=0), torch.split(t0,split_stab)))

number = 10**4

print(timeit.timeit("list_comp()", globals=locals(), number=number))
print(timeit.timeit("map_lambda()", globals=locals(), number=number))
print(timeit.timeit("list_comp_stab()", globals=locals(), number=number))
print(timeit.timeit("map_lambda_stab()", globals=locals(), number=number))
</code></pre>
Adaptive Average Pooling - Implementation

Susmit_Agrawal: <p>I was a bit confused about how Adaptive Average Pooling worked. Based on the explainations provided <a href="https://discuss.pytorch.org/t/what-is-adaptiveavgpool2d/26897">here</a>, I tried to implement my own version:</p>
<pre><code class="lang-auto">def torch_pool(inputs, target_size):
    kernel_size = (inputs.shape[-1] + target_size - 1) // target_size
    points_float = torch.linspace(0, inputs.shape[-1] - kernel_size, target_size)
    points = torch.cat([torch.squeeze(torch.round(points_float)).int(), torch.tensor([inputs.shape[-1]], dtype=torch.int32)], 0)
    
    # Points is [0 2 3 5 7] for the call given below the function

    pooled = []
    for idx in range(points.shape[0] - 1):
        pooled.append(torch.mean(inputs[:, :, points[idx]:points[idx + 1]], dim=-1, keepdim=False))
    pooled = torch.cat(pooled, -1)
    return pooled

inps = np.array([0, 1, 2, 3, 4, 5, 6], dtype=np.float32)[None, :, None]
inps_torch = np.transpose(inps, (0, 2, 1))

x = torch_pool(torch.tensor(inps_torch), 4)
print(x)

x = nn.AdaptiveAvgPool1d(4)(torch.tensor(inps_torch))
print(x)
</code></pre>
<p>The first print (my code) gives the output:</p>
<blockquote>
<p>tensor([[0.5000, 2.0000, 3.5000, 5.5000]])</p>
</blockquote>
<p>But the built-in function (second print) gives:</p>
<blockquote>
<p>tensor([[[0.5000, 2.0000, 4.0000, 5.5000]]])</p>
</blockquote>
<p>Can someone help me out? Where did I go wrong?</p>
Getting original module from RecursiveScriptModule

AfonsoSalgadoSousa: <p>Hello. This is a follow-up question from <a href="https://discuss.pytorch.org/t/torch-jit-modules-without-parameters/105939" class="inline-onebox">Torch Jit Modules without parameters</a>. How can I get the original modules from a RecursiveScriptModule object? I want to get the quantized weights of the convolutional layers to compare their distribution with their full precision counterparts, but even after studying the documentation of TorchScript, I could not get around the RecursiveScriptModule objects.<br>
I really appreciate any help you can provide.</p>

Vasiliy_Kuznetsov: <p>hi <a class="mention" href="/u/afonsosalgadosousa">@AfonsoSalgadoSousa</a>, if you have a scripted convolution layer <code>qconv</code>, you can inspect the weight and bias by doing <code>qconv._packed_params.__getstate__()</code>.</p>

AfonsoSalgadoSousa: <p>Thank you very much again. You seem to be always the one with the answer I need <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
Proper way to use an embedding layer as a linear layer?

seankala: <p>Hi. I’m currently working on a personal reimplementation of the <a href="https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="noopener nofollow ugc">Transformer paper</a> and had a question.</p>
<p>On page 5 in section “3.4 Embeddings and Softmax,” it states:</p>
<blockquote>
<p>In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation.</p>
</blockquote>
<p>I’ve currently implemented my model to use just one embedding layer for both source and target tensors, but I’m wondering if there would be a way that I could use the weights of the embedding layer as a linear layer. What I’ve currently done is something like:</p>
<pre><code class="lang-auto">output = previous_layer(previous_input)
final_output = torch.matmul(output, embedding_layer.embedding.weight.transpose(1, 0))
</code></pre>
<p>I’ve transposed the weight matrix before matrix multiplication because it’s of shape <code>(vocab_size, embedding_dim)</code> and the shape of <code>output</code> is <code>(batch_size, seq_len, embedding_dim)</code>. Is this the proper way to use an embedding layer as a linear layer? If not, I’d like some tips on what I should be doing.</p>
<p>Thanks.</p>
Cosine_similarity on two huge vector (RAM error)

Jay_Yun: <p>Hi, I’m training model on semantic_search about news title and text -&gt; is_related(bool)</p>
<p>I have changed title and text into vector using transformer and want to calculate similarity.</p>
<p>Since torch vector(each are tensors) was so huge (142565x1024), the code below gives me an error saying “allocated more memory than is available”.</p>
<p>util.pytorch_cos_sim(embeddings1[i], embeddings2[i])</p>
<p>The code above calculate mat mul of two vectors(A,B^T) but I only need diagonal result of cos_sim, since I only need to get a similarity only between matching title and context (cos_sim[i][i]).</p>
<p>Is there any way to calculate without using lots of rams?</p>

Abhilash_Srivastava: <p>In 142565x1024, I can understand 1024 is the embedding dim. What is 142565?</p>
<p>If it doesn’t fit in the memory, try using lazy loading (say load just one pair at a time and compute similarity score). This will slow down the computation process (unless you parallelize it), but save RAM.</p>

Jay_Yun: <p>thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> the sparse vector dimension is (142565 x 1024) which there’s 142565 sentence with 1024 embedding_dim</p>

Abhilash_Srivastava: <p>Makes sense.<br>
Don’t do mat multiplication, instead use lazy loading with a for loop (or yield). This removes vectorization (hence memory consumption) but would increase the computation time.</p>
Permute on tensor whose ndim > 2

Hannibal046: <p>Hello,</p>
<p>I am always confused about the <code>permute</code> operation on tensors whose dim are greater than 2.<br>
When in 2D dimension, the permute operation is easy to understand, it is just a transpose of a matrix.<br>
But when it comes to higher dimension, I find it really hard to think.<br>
Personally, I will consider 2D tensor as matrix, 3D tensor as a list of matrix, 4D tensor as a list of cubic.</p>
<p>In practice, I know that permute operation is used to change the dimension. For example in NLP, we can use <code>example_tensor.permute(1,0,2)</code> to change from <code>(batch_size,seq_len,hid_dim)</code> to <code>(seq_len,batch_size,hid_dim)</code>. But how does this happen? Why the float number in a tensor just miraculously become the shape we want it to be?</p>
<p>Is there any mental model to figure this out? Thanks very much !</p>
<p>Actually this confusion comes from this code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
x = torch.arange(3*4*5).reshape(3,4,5)
y = x.reshape(4,3,5)
z = x.permute(1,0,2)
</code></pre>
<p>And it turns out that y is not equal to z.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bc6ebb3a45b7fabe79597fd34d842a0f90d74c0.png" data-download-href="https://discuss.pytorch.org/uploads/default/2bc6ebb3a45b7fabe79597fd34d842a0f90d74c0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bc6ebb3a45b7fabe79597fd34d842a0f90d74c0.png" alt="image" data-base62-sha1="6fgIrHGzVpUrnXhoGw9UanKvkAM" width="690" height="400" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">948×550 13.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I know maybe I can take 3D as a list of matrix to figure out why y is not equal to z. But I want to know if there is more general way to this kind of problem. Or how do you guys think about high dimensional tensor?</p>

googlebot: <p>In your example, memory is shared by three views, and 0…60 sequence is never reordered in memory, you’re just assigning different indexes to same memory cells. Tensor.stride() may may helpful to understand permuted tensors:</p>
<p>x.stide() -&gt; 20,5,1<br>
y.stride() -&gt; 15,5,1<br>
z.stride() -&gt; 5, 20,1<br>
multiplying these values by indexes gives you a memory offset used to access data<br>
so y[1,0,0] = 15 (value 15 at offset 15), but z[1,0,0] = 5</p>

Hannibal046: <p>But How can we calculate z.stride() manually? thanks!</p>

googlebot: <p>It is permuted x.stride(), i.e. you just select dimensions in a different order.<br>
Initial (contiguous tensor) values for A,B,C,D sized tensor are <code>B*C*D</code>,C*D,D,1</p>
Autoencoder reconstruction metric

Rojin: <p>I am building a Variational Autoencoder, and I am looking for a metric to compare the input with the reconstruction.</p>
<p>My input is a sparse matrix (not an image).</p>
<p>my model looks like:</p>
<pre><code class="lang-auto">class VAE(nn.Module):
    def __init__(self, inputDim, hiddenDim, zdim):
        super(VAE, self).__init__()

        self.dropout = nn.Dropout(0.4)

        self.inputDim = inputDim
        self.hiddenDim = hiddenDim
        self.zdim = zdim

        self.hiddenEnc1   = nn.Linear(self.inputDim, self.hiddenDim) #1
        self.hiddenEncBN1 = nn.BatchNorm1d(num_features=self.hiddenDim)
        
        self.mu2  = nn.Linear(self.hiddenDim, self.zdim)
        self.var2 = nn.Linear(self.hiddenDim, self.zdim)
        
        self.hiddenDec1   = nn.Linear(self.zdim, self.hiddenDim) #1
        self.hiddenDecBN1 = nn.BatchNorm1d(num_features=self.hiddenDim)
        
        self.hiddenDecL3 = nn.Linear(self.hiddenDim, self.inputDim) #3

    def encode(self, x):
        h1 = F.relu(self.hiddenEncBN1(self.dropout(self.hiddenEnc1(x))))
        return self.mu2(h1), self.var2(h1)

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        h3 = F.relu(self.hiddenDecBN1(self.hiddenDec1(z)))
        return torch.sigmoid(self.hiddenDecL3(h3))

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar, z
</code></pre>
<p>In order to compare the input with the reconstructed, I basically measured the Pearson correlation for each data and plotted a Pearson distribution. The result is not good. My main concern is that is this even a good metric? given that I start with a sparse matrix, and end up with a sigmoid reconstructed matrix? What would be the best way of comparing the two matrices?</p>
<p>The result is better when I compare the reconstruction to the sigmoid of the input. But I wanted to see if I should change the model, activation, etc.?</p>
Torch.tensor returns error while creating a copy

_singh: <p>Using a model serialized as torchscript.<br>
While inference, getting the following error at model’s forward method.</p>
<p>Doing type conversion here. Can’t use tensor.type(torch.float16). Leads to a different error. Hence creating a new tensor. But can’t infer using this approach either.</p>
<pre><code class="lang-auto">out = self.decoder(x).clone().detach()
cast = torch.tensor(out , dtype=torch.float16)
               ~~~~~~~~~~~~ &lt;--- HERE
return cast
RuntimeError: Cannot input a tensor of dimension other than 0 as a scalar argument
</code></pre>

googlebot: <p>out.to(dtype=torch.float16) should work</p>

Jide_Olabisi: <p>This doesn’t work for me, I keep getting the frustrating error.</p>
How to sync/combine processed data from distributed dataset?

Johnson_Mark: <p>I have a data loader that defined as <a href="https://github.com/cavalleria/cavaface.pytorch/blob/db6e2d46ee5a274381dda3499f8198448a80dd9f/train.py#L106" rel="noopener nofollow ugc">link</a></p>
<pre><code class="lang-auto">    train_sampler = torch.utils.data.distributed.DistributedSampler(dataset_train)
    train_loader = torch.utils.data.DataLoader(dataset_train, batch_size=per_batch_size,
                                                shuffle = (train_sampler is None), num_workers=workers,
                                                pin_memory=True, sampler=train_sampler, drop_last=DROP_LAST)
  
</code></pre>
<p>During training, I used 4 GPUs and the code likes</p>
<pre><code class="lang-auto">        train_sampler.set_epoch(epoch)
        DISP_FREQ = 100  # 100 batch
        batch = 0  # batch index
        inputs_from_all_gpus
        for inputs, labels in tqdm(iter(train_loader)):
              #process inputs and labels
              
       #-------------combine inputs from each GPUs to inputs_from_all_gpus --------------------
</code></pre>
<p>My question is how can we combine all <code>inputs</code> from each gpus to the <code>inputs_from_all_gpus</code> when all process on distributed is done?</p>
Input of video with Conv3D to resnet3D

Hassan_Imani: <p>I have a question abou using video with Conv3D.<br>
My input is like [batch_size=128, channels=3, depth=32, height=64, width=64]<br>
Like 32 RGB videos with size 64 * 64</p>
<p>I want to feed this input to resnet3D (3D version of resnet model) and fine tune it.<br>
I am not sure how to feed 32 videos to resnet.<br>
Does it matter to feed the input like:<br>
first case: [batch_size=128, channels=3, depth=32, height=64, width=64]<br>
or<br>
second case: [batch_size=128, channels=3, height=64, width=64, depth=32]</p>
<p>It accepts both of the cases, but I dont know which one is the right way.</p>
<p>If I feed like second case, what will happen? The network will not learn or it will not learn efficiently?</p>
<p>Thank you</p>

InnovArul: <p>As per the documentation for <code>Conv3d</code>, the input shall be passed as <code>[batch_size=128, channels=3, depth=32, height=64, width=64]</code>.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html#conv3d" class="onebox" target="_blank" rel="noopener">https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html#conv3d</a></p>
<aside class="quote no-group" data-username="Hassan_Imani" data-post="1" data-topic="107085">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hassan_imani/48/32805_2.png" class="avatar"> Hassan_Imani:</div>
<blockquote>
<p>If I feed like second case, what will happen? The network will not learn or it will not learn efficiently?</p>
</blockquote>
</aside>
<p>It might learn something and the effect of this on performance is unknown.<br>
The common understanding might be lost between the design of the network and how the input is organized. It all depends on how you formulate the kernel size I guess. (whether it is designed keeping <code>[batch_size, channel, depth, height, width]</code> or <code>[batch_size, channel, height, width, depth]</code> in mind).</p>
<p>Given that you are using pretrained resnet3D model, it would have been trained with <code>[batch_size, channel, depth, height, width]</code>.</p>

Hassan_Imani: <p>Thank you for the reply.<br>
I have another isue related to this issue.</p>
<p>I have a code in keras. I have a problem understanding the “same” padding.<br>
My input shape to this layer is:  [128, 2048, 4, 2, 2]<br>
This is my keras code:<br>
combined = Conv3D(128, (3, 3, 3),strides=1,  padding=‘same’)(combined)</p>
<p>And I want to do it in pytorch. What I did:<br>
self.conv1 = nn.Conv3d(2048, 128, kernel_size=(3,3,3), stride=1,  padding=( 1,1,1))</p>
<p>Are these perform the same?</p>

InnovArul: <aside class="quote no-group" data-username="Hassan_Imani" data-post="3" data-topic="107085">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hassan_imani/48/32805_2.png" class="avatar"> Hassan_Imani:</div>
<blockquote>
<p>Are these perform the same?</p>
</blockquote>
</aside>
<p>Yes, in my understanding, they do the same.</p>
ValueError and ImportError occurred when import torch

11115: <p>The following error occurred when I try to run a python script called <code>math.py</code> in the <code>Desktop/math</code> folder, which is a mathematical symbol classifier:</p>
<pre><code class="lang-auto">stupid-idiot:math likunshu$ python math.py
ImportError: numpy.core.multiarray failed to import
Traceback (most recent call last):
  File "math.py", line 3, in &lt;module&gt;
    import torch
  File "/Users/likunshu/anaconda/lib/python3.6/site-packages/torch/__init__.py", line 53, in &lt;module&gt;
    from torch._C import *
ValueError: module functions cannot set METH_CLASS or METH_STATIC
</code></pre>
<p>Then in the same shell, I typed <code>ipython</code>, and successfully import torch like this:</p>
<pre><code class="lang-auto">In [1]: import torch

In [2]: torch.__path__
Out[2]: ['/Users/likunshu/anaconda/lib/python3.6/site-packages/torch']
</code></pre>
<p>I’ve googled it on the Internet and tried many solutions but nothing worked. Does anyone have some ideas about how to solve this problem?<br>
My Python version is 3.6.1 and Anaconda version is 4.4.0</p>

Jing: <p><strong>Remove</strong> the old numpy  and run the following code.<br>
<code>pip install -U numpy</code></p>

NTUshark: <p>You’re exactly right! Thanks for your answer~, help me a lot!</p>

yezhengli-Mr9: <p>Yes this helps! Thanks a lot~</p>
Model slows down significantly after first iteration?

tueboesen: <p>I’m testing out a pretrained network, but for some reason my code reports that it slows down significantly after the first iteration, and I don’t know why.</p>
<p>I’m running the following:</p>
<pre><code class="lang-auto">    for i in range(maxiter):
        with torch.no_grad():
            tt2 = time.time()
            results = model(seq_ref_batch)
            tt3 = time.time()
        print("{:}/{:} times {:2.2f}s".format(i,maxiter,tt3-tt2))
</code></pre>
<p>Which produces the following:</p>
<pre><code class="lang-auto">0/603 times 0.16s
1/603 times 3.42s
2/603 times 3.49s
3/603 times 3.49s
4/603 times 3.50s
5/603 times 3.48s
6/603 times 3.48s
7/603 times 3.48s
8/603 times 3.48s
9/603 times 3.49s
10/603 times 3.50s
11/603 times 3.50s
12/603 times 3.54s
13/603 times 3.51s
14/603 times 3.50s
15/603 times 3.50s
16/603 times 3.50s
17/603 times 3.49s
18/603 times 3.50s
19/603 times 3.50s
20/603 times 3.50s
21/603 times 3.51s
22/603 times 3.49s
23/603 times 3.50s
24/603 times 3.50s
25/603 times 3.51s
</code></pre>
<p>Does anyone know what is going on here?</p>

googlebot: <p>you should add torch.cuda.synchronize() line before tt3</p>
Multivariate LSTM classification

tyterry: <p>Hi all,</p>
<p>I am trying out multivariate LSTM for classification problem, starting with a simple custom dataset as follows:</p>
<pre><code class="lang-auto">for i in range(2000):
    seq = random.sample(range(0,100), 30)
    seq = np.array(seq).reshape(1,-1)
    if i == 0:
        data = pd.DataFrame(seq)
    else:
        data = pd.concat((data, pd.DataFrame(seq)), axis = 0)
</code></pre>
<p>which is essentially a dataset with 2000 samples and 30 “features” and the custom target of this dataset is as follow:</p>
<pre><code class="lang-auto">sum_ = data.sum(axis=1)
reduced = sum_ - sum_.shift(periods = 5)
</code></pre>
<p>The target will be 1 if the variable <em>reduced</em> is positive and 0 if negative.</p>
<p>Trial was made with a LSTM with 2 layers with size of 256. 5 epoches into training the BCE loss do not seem to drop but instead fluctuate a lot as follow:<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecc4b6f5f7137792f368ec9a4943f384c2ef72f1.png" alt="losses_4" data-base62-sha1="xMy8P1IBrQZrhn5JoQbYpZ3VL0J" width="640" height="480"></p>
<p>The loss was initially around 0.6 but then fluctuating among 2.5 to 1e-3.</p>
<p>The adopted optimizer is Adam with one cycle lr (max lr is 1e-3).</p>
<p>Is there any reason that may case such result? or the custom dataset is still too complicated?</p>
<p>Thanks in advance for any advice!</p>
<p>Merry Christmas to you all~</p>
Taking lot of GPU how to optimize

Jaideep_Valani: <pre><code class="lang-auto">def run(x,y, z, d_k, mask, dropout, zero_pad=False, gamma=None):
    
    scores = torch.matmul(x,y.transpose(-2, -1)) / \
        math.sqrt(d_k)  # BS, 8, seqlen, seqlen
    
    bs, head, seqlen = scores.size(0), scores.size(1), scores.size(2)

    x1 = torch.arange(seqlen).expand(seqlen, -1).to(device)
    x2 = x1.transpose(0, 1).contiguous()

    with torch.no_grad():
        scores_ = scores.masked_fill(mask == 0, -1e32)
        scores_ = F.softmax(scores_, dim=-1)  # BS,8,seqlen,seqlen
        scores_ = scores_ * mask.float().to(device)
        distcum_scores = torch.cumsum(scores_, dim=-1)  # bs, 8, sl, sl
        disttotal_scores = torch.sum(
            scores_, dim=-1, keepdim=True)  # bs, 8, sl, 1
        position_effect = torch.abs(
            x1-x2)[None, None, :, :].type(torch.FloatTensor).to(device)  # 1, 1, seqlen, seqlen
        # bs, 8, sl, sl positive distance
        dist_scores = torch.clamp(
            (disttotal_scores-distcum_scores)*position_effect, min=0.)
        dist_scores = dist_scores.sqrt().detach()
    m = nn.Softplus()
    gamma = -1. * m(gamma).unsqueeze(0)  # 1,8,1,1
    # Now after do exp(gamma*distance) and then clamp to 1e-5 to 1e5
    total_effect = torch.clamp(torch.clamp(
        (dist_scores*gamma).exp(), min=1e-5), max=1e5)
    scores = scores * total_effect

    scores.masked_fill_(mask == 0, -1e32)
    scores = F.softmax(scores, dim=-1)  # BS,8,seqlen,seqlen
    if zero_pad:
        pad_zero = torch.zeros(bs, head, 1, seqlen).to(device)
        scores = torch.cat([pad_zero, scores[:, :, 1:, :]], dim=2)
    scores = dropout(scores)
    output = torch.matmul(scores,z)
    return output
</code></pre>
Freeze single weight in CNN

Leaves: <p>Hi, I found that require_grad = False is feasible only for a single layer other than a single weight. Is there an approach to freeze single/individual weight in the network? BTW, I have noticed a method of setting weight.grad() to zero through a mask (<a href="https://discuss.pytorch.org/t/how-to-freeze-a-single-weight-of-a-conv2d-layer/57396/2" class="inline-onebox">How to freeze a single weight of a conv2d layer - #2 by ptrblck</a>), but it there another direct approach?</p>
Torch.distributions.kl.kl_divergence outputs negative values

RogerAylagas: <p>Dear all,</p>
<p>I am trying to compute the KL Divergence between 2 Multivariate Gaussians.</p>
<p>My Code is as follows:</p>
<pre><code class="lang-auto">from torch.distributions.multivariate_normal import MultivariateNormal
p = MultivariateNormal(mu1,  torch.diag_embed(std1))
q = MultivariateNormal(mu2, torch.diag_embed(std2))              
kld = ( torch.distributions.kl.kl_divergence(p, q) +
 torch.distributions.kl.kl_divergence(q, p) )
</code></pre>
<p>where mu1, mu2, std1, std2 are positive vectors of the same length.</p>
<p>A priori all seems to be well programmed but the output of the KLD is sometimes negative.</p>
<p>Can you please help me with this issue?</p>
<p>Environment details:<br>
python: 3.8<br>
pytorch: 1.6.0</p>

googlebot: <p>The KL function for full-rank MVN is rather involved, so it is perhaps a numerical issue. Try  using distributions.Independent to wrap univariate gaussians. Maybe avoiding tiny std values will also work.</p>
Torchvision fcn_resnet101 Pretrained Backbone

danielz: <p>I’m looking through the <a href="https://github.com/pytorch/vision/blob/7b9d30eb7c4d92490d9ac038a140398e0a690db6/torchvision/models/segmentation/segmentation.py#L20" rel="noopener nofollow ugc">source code</a> and documentation of <code>fcn_resnet101</code> and felt a little bit of confused about the pretraining. Is the resnet backbone pretrained on ImageNet while the classifier is pretrained on the subset of Coco?</p>