akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is a minimum code that would cause RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)` error?

Mona_Jalal: <p>I am looking for a minimal PyTorch code that would cause the following error.</p>
<blockquote>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling <code>cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)</code></p>
</blockquote>
<p>The reason for this request is that I get this error in two out of my three machines and I am not sure what’s the best way to go about debugging it in a large code-base. Here’s the related post <a href="https://discuss.pytorch.org/t/vertices-torch-matmul-vertices-unsqueeze-0-rotations-init-runtimeerror-cuda-error-cublas-status-execution-failed-when-calling-cublassgemmstridedbatched-in-centos/116123" class="inline-onebox">Vertices=torch.matmul(vertices.unsqueeze(0), rotations_init), RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched in CentOS</a></p>

ptrblck: <p>You could enable <a href="https://docs.nvidia.com/cuda/cublas/index.html#cublasLoggerConfigure">cublas logging</a> and check for errors of post the logs here.</p>
Debugging slow GPU operations

ljj7975: <p>So I have a network implementation and two machines of same hardware.</p>
<p>precisely</p>
<pre><code class="lang-auto">----------------------  --------------------------------------------------------------------------------
sys.platform            linux
Python                  3.6.9 (default, Oct  8 2020, 12:12:24) [GCC 8.4.0]
numpy                   1.19.2
detectron2              0.3 @/home/roboeye-2/repos/roboeye/nova/src/detectron2
Compiler                GCC 7.5
CUDA compiler           CUDA 11.1
detectron2 arch flags   7.5
DETECTRON2_ENV_MODULE   &lt;not set&gt;
PyTorch                 1.7.1 @/home/roboeye-2/.virtualenvs/nova/lib/python3.6/site-packages/torch
PyTorch debug build     False
GPU available           True
GPU 0                   GeForce RTX 2080 Ti (arch=7.5)
CUDA_HOME               /usr/local/cuda
Pillow                  7.2.0
torchvision             0.8.2 @xxxx/lib/python3.6/site-packages/torchvision
torchvision arch flags  3.5, 5.0, 6.0, 7.0, 7.5
fvcore                  0.1.3.post20210306
cv2                     4.4.0
----------------------  --------------------------------------------------------------------------------
</code></pre>
<p>on one machine forward operation takes only 0.5 seconds, 40% of GPU utils<br>
however, on the other machine, the same operation takes 22 seconds, 100% of GPU utils</p>
<p>Since all the hardware setting is same, code is identical, I am quite lost on what is causing the issue.</p>
<p>Can anyone help me with what I can try?</p>

ptrblck: <p>Are you using the same PyTorch installation (conda binaries, pip wheels, or a source build) on both machines?<br>
If so, are you also using the same CUDA and cudnn version? In case you are using cudnn (e.g. in conv layers), are you using <code>torch.backends.cudnn.benchmark = True</code>?</p>
PyTorch vs TensorFlow Input Shapes

RyanElliott: <p>I’ve been messing around with a Transformer using Time2Vec embeddings and have gone down a rabbit hole concerning input tensor shapes. It appears that PyTorch’s input shapes are uniform throughout the API, expecting <code>(seq_len, batch_size, features)</code> for timestep models like <code>nn.Transformer</code>, <code>nn.LSTM</code>, <code>nn.GRU</code>. TensorFlow’s API inverts the first two dimensions, expecting <code>(batch_size, seq_len, features)</code>.</p>
<p>It seems like small discrepancies between the APIs like this would cause critical and difficult-to-debug bugs for developers looking to make the switch from one library to the other. This makes me wonder what the rationale is, on either side, for preferring one input shape over the other.</p>

ptrblck: <p>PyTorch uses the mentioned shape for performance reasons. If you prefer to have the batch dimension in <code>dim0</code>, you could set <code>batch_first=True</code> while creating the module, which could then be easier to port code.</p>
Why does nn.Embedding expects inputs to be on the same device as the weights?

dangne: <p>I am implementing an algorithm that has to send many small queries to the nn.Embedding while training. Since my embedding is on CUDA, I have to move my indices to CUDA every time I want to do so, which is extremely expensive. So I have the following questions:</p>
<ol>
<li>Why the inputs have to be on the same device as the weight when nn.Embedding is basically just a lookup table?</li>
<li>I am thinking of a workaround and that is to reimplement the embedding layer with raw tensors and perform manual indexing, for example:</li>
</ol>
<pre><code class="lang-auto">embs = torch.rand((5, 5), device="cuda", requires_grad=True)
optimizer = torch.optim.Adam([embs], 0.005)
loss = ((embs[[0, 2, 4]] - torch.ones_like(embs[[0, 2, 4]]))**2).sum()
loss.backward()
optimizer.step()
</code></pre>
<p>is this way safe and optimized? Does manual indexing is just the same as moving indices to CUDA, then retrieve the values?<br>
3. Any other suggestions to solve this problem?</p>
<p>Thanks a lot.</p>

ptrblck: <ol>
<li>
<p>Since the operation is executed on the GPU, the inputs as well as other tensors must be loaded onto the device before the kernel can be launched.</p>
</li>
<li>
<p>The indices will also be moved to the device, so there shouldn’t be a difference.</p>
</li>
</ol>
How to load a parrallel model with one gpu?

Igo312: <p>I have trained one model in two gpus and saved it.</p>
<p>And now I want to reload it just in one gpu,the error cause which says</p>
<blockquote>
<p>Missing key(s) in state_dict: “base_network.conv1.weight”,<br>
Unexpected key(s) in state_dict: “module.base_network.conv1.weight”</p>
</blockquote>
<p>It seems save a multi-gpu model will get a prefix name <code>module</code>. Is there a more elegant way rather than using <code>model_state_dict = state_dict[7:]</code>?</p>

ptrblck: <p>You could save the <code>model.module.state_dict()</code> as described in <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models">this tutorial</a> instead of <code>model.state_dict()</code>. If you cannot change it now (as the <code>state_dict</code> was already saved), your approach seems to be the correct workaround.</p>
Inceptionv3 failing to work with different Loss functions

Taha_Iftikhar: <p>Hello,<br>
I am trying to retrain the Inception v3 on some sign language alphabets, but am running into an issue when I train it:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/600ca2d1004ae3e94d97722b2606f40659e34001.png" data-download-href="https://discuss.pytorch.org/uploads/default/600ca2d1004ae3e94d97722b2606f40659e34001" title="incept issue"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/600ca2d1004ae3e94d97722b2606f40659e34001.png" alt="incept issue" data-base62-sha1="dHGUoMewVQi8iUaDVntPkfXFE5z" width="689" height="420" data-dominant-color="F8DAD9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">incept issue</span><span class="informations">1292×787 57.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The issue appears to be with inception v3 not reconizing log_softmax.<br>
I tried using other loss functions such as NLLLoss, but that gives a different error .<br>
I tried BCEWithLogitsLoss and MSELoss and got this issue:<br>
Attribute Error: InceptionOutpurs object has no arrtibute size</p>
<p>What loss function should I use that will help me avoid these issues? Or is there a problem with inception i should focus on resolving?</p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/error-in-training-inception-v3/23933/6">this post</a>, which explains how to use the <code>InceptionOutputs</code>.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Increasing the contrast of images in pytorch transform

nafise: <p>I have a cnn network and I want to increase the image contrast, I use the code below:<br>
trfm_train = transforms.Compose([transforms.Resize((sz1,sz2)),</p>
<pre><code>                      transforms.ToTensor(),

                      transforms.ColorJitter(brightness=(0.5,1.5), contrast=(1), saturation=(0.5,1.5), hue=(-0.1,0.1)),

                      transforms.RandomHorizontalFlip(p=0.5),

                      transforms.RandomRotation(45, resample=False, expand=False, center=None),

                      transforms.Normalize([0.485,0.456,0.406], [0.229, 0.224,0.225])])
</code></pre>
<p>but I want to increase the contrast and resolution of my images. how can I do it?</p>

ptrblck: <p>Similar question answered <a href="https://discuss.pytorch.org/t/max-value-for-contrast-in-pythorch-transform/116489/2">here</a>. The docs give you the information:</p>
<blockquote>
<ul>
<li><strong>contrast</strong> (<a href="https://docs.python.org/3/library/functions.html#float"><em>float</em></a> <em>or</em> <em>tuple of python:float</em> <em>(</em> <em>min</em> <em>,</em> <em>max</em> <em>)</em>) – How much to jitter contrast. contrast_factor is chosen uniformly from [max(0, 1 - contrast), 1 + contrast] or the given [min, max]. Should be non negative numbers.</li>
</ul>
</blockquote>
<p>If you want to change the resolution on the image, change <code>sz1</code> and <code>sz2</code> to the desired values.</p>
Is it possible to load weights from a pre-trained model layer by layer?

Siladittya_Manna: <p>I trained a ResNet-50 + MLP (2 nn.Linear) model. Next, I want to use the same model, except that I now have a Dropout layer in between the 2 nn.Linear layers in the MLP. But I also want to use the weights from the previous model.</p>
<p>I am receiving an error because the model changes and ‘load_state_dict’ cannot load the weights.</p>
<p>Is it possible to load the weights in this manner?</p>
<p>I attempted to do like this:</p>
<pre><code>for n,p in model.named_parameters():
    try:
        p = model_state_dict[n]
    except:
        p = model_state_dict['projector.2.weight']
</code></pre>
<p>The name of the key in the saved model dict is ‘projector.2.weight’ while it becomes ‘projector.3.weight’ in the new model with dropout.</p>
<p>Also, weights of other layers not correctly loaded using this piece of code. Any suggestions??</p>

ptrblck: <p>You could either load the <code>state_dict</code> into the model before applying any manipulations, change the <code>state_dict</code> keys to map your new modules names, or load the parameters layer-wise, which would most likely also need a mapping between the currently modified model and the pretrained <code>state_dict</code>.</p>
<p>I don’t know how you are manipulating the model, but I think the first approach might be the easiest one.<br>
E.g. you could write a custom <code>nn.Module</code>, use the pretrained model inside it, add the new layer(s), and change the <code>forward</code> method as needed.</p>
computer screen crashes

zhouxin666251: <p>Hello, when I am training neural network, there will always be computer screen crashes? Under normal circumstances, what is the reason?<br>
ubuntu18.04, GTX1080ti, nvidian driver 460.32.03</p>

ptrblck: <p>This issue sounds like a Xorg/Windows and potentially NVIDIA driver issue and doesn’t seem to be Pytorch-specific, so you might get a better and faster answer in an NVIDIA support board or e.g. StackOverflow.</p>
Implementation of Inception v_3

MaximeEm: <p>Hello everyone,<br>
I struggle to understand the details of the Inception_v3 implementation in PyTorch. It seems to me that there are some differences between the documentation and the actual implementation. Could someone confirm that there are differences ? If so, is there somewhere a schema to help understand what happens under the hood ?</p>
<p>I attached the table you can find on <a href="https://pytorch.org/hub/pytorch_vision_inception_v3/" class="inline-onebox" rel="noopener nofollow ugc">Inception_v3 | PyTorch</a> as well as in the original paper. The implementation I consider is here: <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py" class="inline-onebox" rel="noopener nofollow ugc">vision/torchvision/models/inception.py at main · pytorch/vision · GitHub</a></p>
<p>For example I don’t see any equivalence of the layer ‘self.Conv2d_3b_1x1’ in the table.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/828a0e7b0b9bc7a540e3ed3853a9ebf581d33632.png" alt="inception_v3" data-base62-sha1="iCNR4p8V7QqrLGkSKwSqNA7Ow0i" width="544" height="474"></p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/pretrained-inception-v3-not-matched-with-research-paper/113345/2">this post</a>, which answers a similar question.</p>
Please help me，Error in jetson nano

AoLie: <p>When connecting the camera through USB and calling the camera recognition, an error occurred<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29c08f74872e820ae4379e1e0012998f5727d9ec.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/29c08f74872e820ae4379e1e0012998f5727d9ec" title="IMG_20210330_185057"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29c08f74872e820ae4379e1e0012998f5727d9ec_2_665x499.jpeg" alt="IMG_20210330_185057" data-base62-sha1="5Xm8mS4yLzu6RomWtZMAdImOdK4" width="665" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29c08f74872e820ae4379e1e0012998f5727d9ec_2_665x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29c08f74872e820ae4379e1e0012998f5727d9ec_2_997x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29c08f74872e820ae4379e1e0012998f5727d9ec.jpeg 2x" data-dominant-color="72506C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20210330_185057</span><span class="informations">1280×961 211 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>This error doesn’t seem to be PyTorch-specific as it points to OpenCV, so you might want to ask in a Jetson/OpenCV developer board or e.g. StackOverflow to get faster support. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>That being said, based on the error message I assume OpenCV cannot find or access the camera, so you might want to check, if the camera is successfully installed and detected as well as the permissions to access it.</p>
SIGBUS error when doing inference with a torchscript model with CPU

Prashant: <p>Hi I have a speech recognition model torchscript and a C++ engine that I’ve written to perform inference.</p>
<p>I’m noticing that some of the time when I run my model, I get SIGBUS exceptions with stack traces that don’t have much information.</p>
<p>Here is one of those stack traces that I got from boost stacktrace</p>
<pre><code class="lang-auto">FAILED WITH SIGNAL: 7
 0# handler(int) at /home/psridhar/speech-engine/bin/eval_session.cpp:79
 1# 0x00007FB303B2DFD0 in /lib/x86_64-linux-gnu/libc.so.6
 2# mkl_blas_avx512_sgemm_kernel_0 in /home/psridhar/speech-engine/third_party/libtorch/lib/libtorch_cpu.so
</code></pre>
<p>I got my libtorch from <a href="https://download.pytorch.org/libtorch/nightly/cpu/libtorch-cxx11-abi-shared-with-deps-latest.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/nightly/cpu/libtorch-cxx11-abi-shared-with-deps-latest.zip</a> on March 10th.</p>
<p>Anyone have any idea why this is happening / how I can debug further?</p>

ptrblck: <p>Based on the stack trace it seems that MKL is raising the SIGBUS error in an AVX512 instruction.<br>
If I’m not mistaken, this error could be raised, if the expected memory alignment is not met, so it could point towards an internal MKL error.<br>
Would you mind posting this issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> with (if possible) an executable code snippet?</p>
Simple producer-consumer problem causing 100% CPU usage

sunder: <p>The below code is causing 100% CPU usage.<br>
There’s no GPU.</p>
<pre><code class="lang-auto">import random
from queue import Queue
from threading import Thread

import torch
from torchvision.models import resnet18
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

model = resnet18(pretrained=True)
model.fc = torch.nn.Identity()

#list of 50 image tensors of varying sizes
ims = [torch.randn(1,3,random.randint(224,512),random.randint(224,512)) for i in range(50)]

def producer(out_q):

    for im_tensor in ims:
        with torch.no_grad():
            feat_vector = model(im_tensor).detach().numpy()
        out_q.put(feat_vector)

def consumer(in_q):
    feat_vector1 = in_q.get()
    feat_vector2 = np.random.randn(1,512)
    sim_score = cosine_similarity(feat_vector1,feat_vector2)[0][0]
    in_q.task_done()
    return sim_score

for i in range(50):
    q = Queue()
    t1 = Thread(target = consumer, args =(q, ))
    t2 = Thread(target = producer, args =(q, ))
    t1.start()
    t2.start()
    q.join()
</code></pre>
<p>How to avoid the CPU cranking up?</p>

ptrblck: <p>You are not using the GPU, so a high CPU workload is expected.<br>
<a href="https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html#get-device-for-training">This tutorial</a> explains how to use the GPU by pushing the model as well as the tensors to the device.</p>
Slow For Loop On Small Inputs?

ZeroMax: <p>Hello,</p>
<p>I’m trying to find a way to prevent a painfully slow for loop in Pytorch. Basically, I have a tensor, and I want to split it up into pieces and feed those pieces into my model, similar in spirit to a grouped convolution of sorts.</p>
<pre><code class="lang-auto">
        self.C = C
        self.block = Block(C, 3, 64)
    def forward(self, x):
        x_shape = x.shape
        x = torch.flatten(x, start_dim=1, end_dim=-1).unsqueeze(0)
        x = torch.split(x, self.C, -1)
        attention = []
        for i in x:
            attended = self.block(i)
            attention.append(attended)
        attention = torch.stack(attention, 1)
</code></pre>
<p>Small values of C, alongside a large tensor, makes this operation surprisingly much slower, due to the Python for-loop the above code runs through. Is there any way to fix this issue other than the slightly-hacky way of exchanging the batch dimension for the num_block dimension and then running it through the model?</p>
<p>Thanks.</p>
TorchScript for the backward (autograd) graph

ljubisa: <p>Hi,</p>
<p>I am trying to get a sense for the level of support in TorchScript for back propagation graphs produced by using autograd. If anyone can provide a quick summary, and/or pointers to how one can experiment with this - it would be much appreciated.</p>
<p>Ljubisa</p>

Michael_Suo: <p>TorchScript has full support for PyTorch’s tape-based autograd. You can call <code>backward()</code> on your tensors if you are recording gradients and it should work.</p>

ljubisa: <p>Hi Michael,</p>
<p>Thanks for the prompt response. I am interested in tracing through the backward graph using TorchScript and dumping the IR for the autodiff-ed backdrop graph, for full graph optimization in a separate framework. To be precise - on the example of a backward op for a matmul, I’d expect to get the appropriately transposed matmul relevant to the backward pass in the dumped IR graph. Would you expect this to be possible?</p>
<p>Ljubisa</p>

Michael_Suo: <p>Ah, we do not have a public API for exposing a static backward graph, as PyTorch relies on dynamic autograd for doing automatic differentiation. We do have an internal API for symbolic differentiation (see <code>torch/csrc/jit/runtime/autodiff.cpp</code> which you can play with, but it is not complete and we don’t have any stability guarantees about it <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>

valuenumbering: <p>Hi Michael,<br>
My understanding is that torchscript is converted to SSA IR and then executed. How does the autograd work on the SSA IR?<br>
DO you mean autodiff.cpp isn;t enabled for torchscript now?</p>

Linux-cpp-lisp: <p>Hi <a class="mention" href="/u/michael_suo">@Michael_Suo</a>,</p>
<p>Curious if you could elaborate on how “dynamic” plays into this — if a compiled TorchScript model has been through profile-guided optimization and had all of the control flow stripped out, the actual autograd graph structure should be the same at each inference pass, yes?</p>
<p>When I run autograd with <code>create_graph = True</code>, is a graph being created that PyTorch knows how to execute, or only how to differentiate further?</p>
<p>The motivation for these questions is models whose inference pass involves a gradient of their initial prediction, and the hope that it might be possible to save/cache the compute graph representing their gradient as a model in its own right.</p>
<p>Thanks for your help!</p>
Peephole LSTM cell implementation

Monne: <p>Hi,<br>
just want to confirm, is this implementation of peephole LSTM cell are correct?</p>
<pre><code class="lang-auto">class JitLSTMCell(jit.ScriptModule):
    
    def __init__(self, input_size, hidden_size):
        super(JitLSTMCell, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size

        self.weight_ih = nn.Parameter(torch.Tensor(4 * hidden_size, input_size))
        self.weight_hh = nn.Parameter(torch.Tensor(4 * hidden_size, hidden_size))
        
        self.bias_ih = nn.Parameter(torch.Tensor(4 * hidden_size))
        self.bias_hh = nn.Parameter(torch.Tensor(4 * hidden_size))

        self.weight_ch_i = nn.Parameter(torch.Tensor(hidden_size))
        self.weight_ch_f = nn.Parameter(torch.Tensor(hidden_size))        
        self.weight_ch_o = nn.Parameter(torch.Tensor(hidden_size))

        self.reset_parameter()        

    @jit.ignore(drop=True)
    def reset_parameter(self):
        stdv = 1.0 / math.sqrt(self.hidden_size)
        for weight in self.parameters():
            nn.init.uniform_(weight, -stdv, stdv)

    @jit.script_method
    def forward(self, input, state):
        # type: (Tensor, Tuple[Tensor, Tensor]) -&gt; Tuple[Tensor, Tuple[Tensor, Tensor]]
        hx, cx = state
        xh = (torch.mm(input, self.weight_ih.t()) + self.bias_ih + torch.mm(hx, self.weight_hh.t()) + self.bias_hh)

        i, f, _c, o = xh.chunk(4, 1)

        i = torch.sigmoid(i + (self.weight_ch_i * cx))
        f = torch.sigmoid(f + (self.weight_ch_f * cx))
        _c = torch.tanh(_c)

        cy = (f * cx) + (i * _c)

        o = torch.sigmoid(o + (self.weight_ch_o * cy))
        hy = o * torch.tanh(cy)

        return hy, (hy, cy)

</code></pre>
<p>Thanks</p>
Sudden peaks in mean loss during training

rgonzalez60: <p>Hello,<br>
I have trained a model base on SegNet architecture, and when I do it from scratch presents some odds behaviours:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/647cd3b3f64e23b0583c4beb1d4272604d91dfd3.png" data-download-href="https://discuss.pytorch.org/uploads/default/647cd3b3f64e23b0583c4beb1d4272604d91dfd3" title="newplot"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/647cd3b3f64e23b0583c4beb1d4272604d91dfd3_2_517x279.png" alt="newplot" data-base62-sha1="ekXbD4omjjEaoqH62n319qdkW55" width="517" height="279" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/647cd3b3f64e23b0583c4beb1d4272604d91dfd3_2_517x279.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/647cd3b3f64e23b0583c4beb1d4272604d91dfd3_2_775x418.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/4/647cd3b3f64e23b0583c4beb1d4272604d91dfd3.png 2x" data-dominant-color="F0F0F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot</span><span class="informations">972×525 29.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My dataset size is small (165 samples) and I used the following input parameters:</p>
<ul>
<li>Batch size = 1</li>
<li>Learning rate = 0.1</li>
<li>Momentum = 0.9</li>
<li>Loss function = Cross-entropy loss</li>
</ul>
<p>Why does it behave like this? I have been reading about this topic, but I can not be able to find an explanation of this sudden peaks. However, when I apply transfer learning (from VGG16), the odd behaviour does not happen:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/2324181fd8ab6ec2f6672c008ecff845e8fe1c34.png" data-download-href="https://discuss.pytorch.org/uploads/default/2324181fd8ab6ec2f6672c008ecff845e8fe1c34" title="newplot(1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/2324181fd8ab6ec2f6672c008ecff845e8fe1c34_2_517x279.png" alt="newplot(1)" data-base62-sha1="50S2FknKmCbss9rfPg9vAKXQB7u" width="517" height="279" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/2324181fd8ab6ec2f6672c008ecff845e8fe1c34_2_517x279.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/2324181fd8ab6ec2f6672c008ecff845e8fe1c34_2_775x418.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/2324181fd8ab6ec2f6672c008ecff845e8fe1c34.png 2x" data-dominant-color="F0F2F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot(1)</span><span class="informations">972×525 27.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could it be related to the small training set I used?</p>
The initialization of the LSTM model is not understood

Sjk_Jae: <pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.embed = nn.Embedding(input_size, hidden_size)
        self.lstm = nn.LSTM(hidden_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x, hidden, cell):
        out = self.embed(x)
        out, (hidden, cell) = self.lstm(out.unsqueeze(1), (hidden, cell))
        out = self.fc(out.reshape(out.shape[0], -1))
        return out, (hidden, cell)

    def init_hidden(self, batch_size):
        hidden = torch.zeros(self.num_layers, batch_size, self.hidden_size).to(device)
        cell = torch.zeros(self.num_layers, batch_size, self.hidden_size).to(device)
        return hidden, cell
</code></pre>
<blockquote>
<p>self.hidden_size = hidden_size<br>
self.num_layers = num_layers<br>
Both hidden_size and num_layers are initialized, why is input_size not initialized?</p>
</blockquote>

omarfoq: <p>Hi,</p>
<p>You don’t need input size elsewhere in your class, while you need <code>num_layers</code> and <code>hidden_size</code> in <code>init_hidden</code>.</p>

Sjk_Jae: <p>Got it, thank you <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Build pytorch 1.8 failure on Windows with cuda 11.2.2

xq114: <p>Error message:</p>
<pre><code class="lang-auto">C:/Users/xq114/AppData/Local/.xmake/cache/packages/2103/l/libtorch/v1.8.0/source/libtorch/aten/src/ATen/native/cuda/AdaptiveAveragePooling.cu(69): error : calling a __host__ function("__floorf") from a __global__ function("at::native::_NV_ANON_NAMESPACE::adaptive_average_pool&lt;double&gt; ") is not allowed [C:\Users\xq114\AppData\Local\.xmake\cache\packages\2103\l\libtorch\v1.8.0\source\libtorch\build_D45C6846\caffe2\torch_cuda.vcxproj]
</code></pre>
<p>It repeated for many times. The corresponding line reads</p>
<pre><code class="lang-auto">#define START_IND(a,b,c) (int)std::floor((float)(a * c) / b)
</code></pre>
<p>It seems that calling std::floor from a cuda kernel function is forbidden. Is there anyone who can help me with this? Thanks!</p>

ptrblck: <p>This might be related to <a href="https://github.com/pytorch/pytorch/issues/54382">this issue</a>, which is caused by a VS regression. You could use the mentioned workarounds from the issue.</p>
Collective communication across nodes

lkp411: <p>Hi there,</p>
<p>I was wondering if there was a way to do an all_reduce between two GPUs on different nodes. For example if I have a tensor on GPU0 of machine 0 and another tensor on GPU0 of machine 1, is it possible to issue a dist.all_reduce call across the nodes using the NCCL backend.</p>
<p>The following code hangs:</p>
<pre><code class="lang-auto">import argparse
import torch
import torch.distributed as dist


def ProcessArgs():
    parser = argparse.ArgumentParser()
    parser.add_argument("--local_rank", type=int)
    return parser.parse_args()


def MultiNodeTest(global_rank, local_rank):
    torch.cuda.set_device(local_rank)
    t = torch.ones(2, 2).to(local_rank)
    dist.all_reduce(t)


if __name__ == '__main__':
    args = ProcessArgs()
    dist.init_process_group(backend='nccl', init_method='env://')
</code></pre>
<p>I am launching this simple script using the following:</p>
<p>On Node 0:</p>
<p>python3 -m torch.distributed.launch --nproc_per_node=1 --nnodes=2 --node_rank=0 --master_addr=192.168.11.2 --master_port=12347 multinode.py</p>
<p>On Node 1:</p>
<p>python3 -m torch.distributed.launch --nproc_per_node=1 --nnodes=2 --node_rank=1 --master_addr=192.168.11.2 --master_port=12347 multinode.py</p>

lkp411: <p>This code runs just fine when I use gloo as my backend. It only deadlocks when I use NCCL. Any ideas what could be happening?</p>

wayi: <blockquote>
<p>is it possible to issue a dist.all_reduce call across the nodes using the NCCL backend.</p>
</blockquote>
<p>This is definitely supported.</p>
<p>Your code seems correct to me, according to the <a href="https://pytorch.org/docs/stable/distributed.html#launch-utility" rel="noopener nofollow ugc">launch utility tutorial</a>.<br>
Just my guess, have you tried to specify <code>world_size</code> arg in <code>init_process_group</code>?</p>
<p>Or have you tried to use TCP as the <code>init_method</code>? Like <code>init_method="tcp://{}:{}".format(args.master_addr, args.master_port)</code></p>
<p><a class="mention" href="/u/mrshenli">@mrshenli</a> Any idea?</p>

lkp411: <p>I found a workaround by recreating a new group object with all the ranks and then passing that group to all comm calls. If I do this it doesn’t hang anymore. Not sure why just using group.WORLD as the group in the comm calls hangs. Is this a known issue / am I missing something?</p>

wayi: <p>If you still need more investigation, please set the NCCL environment variable <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-socket-ifname" rel="noopener nofollow ugc">NCCL_SOCKET_IFNAME</a> (probably as “eth1”) for debugging.</p>
Custom weight init: do I need detach( )?

PhysicsIsFun: <p>Greetings!</p>
<p>Suppose I want to rescale the initialized weight matrix in a certain way like:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, R):
        super(Net, self).__init__()
        self.lin1 = nn.Linear(2, 100)
        self.lin2 = nn.Linear(100, 3)
        self.lin2.weight.mul_( (R / torch.sqrt(torch.sum(self.lin2.weight.detach()**2 ,axis=1)))[:, None] )
</code></pre>
<p>So I want to change the weight of lin2 based on the current elements of the weights. Do I need the detach ()?<br>
Also, code gives the error “<strong>a leaf Variable that requires grad is being used in an in-place operation</strong>”, so the mul_ does not seem to work. Can I just reassign the right-hand expression to my weight matrix, i.e. <code>self.lin2.weight = ...</code>?</p>

crcrpar: <p>Hi,</p>
<p>I’d write it in a (maybe verbose) way like</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self, R):
        super().__init__()
        self.lin1 = nn.Linear(2, 100)
        self.lin2 = nn.Linear(100, 3)

        with torch.no_grad():
            weight = self.lin2.weight.data.clone().detach()
            weight.mul_( (R / torch.sqrt(torch.sum(weight**2 ,axis=1)))[:, None] )
            self.lin2.weight.data.copy_(weight)
            del weight

    def forward(self, x):
        return self.lin2(F.relu(self.lin1(x)))
</code></pre>
<p>A toy colab is here: <a href="https://colab.research.google.com/drive/12yqnYoRF1C_of6s8fokTEqK2h9wl-wvj?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/12yqnYoRF1C_of6s8fokTEqK2h9wl-wvj?usp=sharing</a></p>

PhysicsIsFun: <p>Thank you, that would do!</p>
<p>Do I even need .data and detach()? Usually, .detach() is enough, is it now?</p>

crcrpar: <p>I’m not sure but I did so because I felt I just wanted to use <code>torch.Tensor</code> and avoid sharing the storage.</p>

ptrblck: <p>Yes, don’t use the deprecated <code>.data</code> attribute as it’s dangerous and can yield unwanted side effects! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

PhysicsIsFun: <p>Thanks ptrblck.</p>
<p>How would you rewrite crcrpar’s solution, or do you have a better suggestion?</p>

ptrblck: <p>You should be able to remove the <code>.data</code> usage from your code without any additional changes.</p>
Mysterious CUDNN kernel

11245: <p>Hello!</p>
<p>Doing performance profiling wit nvprof for the PyTorch model with one convolution layer.</p>
<p>Setup:<br>
For simplicity, the model consists of one Conv1d + Relu layer.<br>
Input size: [32, 64, 1664] kernel_size: 11<br>
Enabled cudnn.benchmark = True<br>
Pre-warmed up.</p>
<p>In the profiler, I see that the volta_fp16_s884cudnn_fp16_256x128_ldg8_relu_f2f_exp_small_nhwc2nchw_tn_v1 function takes most of the time.</p>
<p>Could you tell me, please?<br>
What is the purpose of this function?<br>
Why are there no references to convolutions in the name of the function?</p>
<p>Link to json file with nvprof tracing info <a href="https://gist.github.com/II245/270e541547d12d18fbc3b171d422c651" class="inline-onebox" rel="noopener nofollow ugc">profile_one_layer_pytorch.sqlite.json · GitHub</a><br>
File can be opened in chrome://tracing</p>

vadimkantorov: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> would you have an advice? what does <code>exp</code> and <code>tn</code> mean? is it some conv implementation?</p>

ptrblck: <p>The <code>tn</code> could declare the transposes as described <a href="https://discuss.pytorch.org/t/cuda-kernel-volta-sgemm-128x32-nn-means-what/112686/2">here</a>.</p>
<p><a class="mention" href="/u/11245">@11245</a> is there any particular issue with this kernel, i.e. are you seeing a slowdown or invalid results?</p>

11245: <p>The results are correct. I was just having trouble figuring out if this kernel is a convolution or not since it doesn’t have gemm/fft/winograd/inplace in the function name (other kernels would contain h884gemm, then it’s clear). Is it a conv implementation? What do exp/relu mean? In the trace threshold (relu) op is a separate one.</p>

vadimkantorov: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> It is indeed a gemm kernel, right?</p>
<p>It would be nice to have somewhere a summary table explaining the kernels and their naming suffixes scheme.</p>

ptrblck: <p>This might be a good feature/documentation request for the cudnn team, so feel free to create this request on the cudnn bug tracker.</p>
Calculating derivative without first calculating the Jacobian?

jwilson: <p>Hello all! I need to calculate the derivative of a function. Is there a more efficient way to do this than first calculating the Jacobian and then taking the diagonal? A lot of the computation time is wasted since I will throw most of the values away. Here is a minimal working example of what I would like to accomplish more quickly.</p>
<pre><code class="lang-auto">x = torch.linspace(0,10,11, requires_grad=True)

def f(x):
     return x**2

def gradf(x):
 return torch.diag(torch.autograd.functional.jacobian(f,x))

dfdx = gradf(x)
</code></pre>
Accessing Modules from Modulelist for static quantization

hirenpatel1207: <p>Hi,<br>
I am trying to apply static quantization to a model which has nn.ModuleList() object as one of its module.</p>
<p>To fuse the layers we need to pass the list of layers as string to  ´torch.quantization.fuse_modules´<br>
So I tried:</p>
<pre><code class="lang-auto">import torch.nn as nn

class MyModule(nn.Module):
   def __init__(self):
       super(MyModule, self).__init__()

       self.quant = torch.quantization.QuantStub()
       self.linears = nn.ModuleList([nn.Linear(10, 10), nn.ReLU(), nn.Linear(10, 10), nn.ReLU()])
       self.dequant = torch.quantization.DeQuantStub()

   def forward(self, x):
       # ModuleList can act as an iterable, or be indexed using ints
       x = quant(x)
       x = self.linears[0](x)
       x = self.linears[1](x)
       x = self.linears[2](x)
       x = self.linears[3](x)
       x = self.dequant(x)

       return x

model_fp32 = MyModule()

model_fp32.eval()

input_fp32 = torch.randn(4, 10,10)

out = model_fp32(input_fp32)

model_fp32.qconfig = torch.quantization.get_default_qconfig('fbgemm')

model_fp32_prepared = torch.quantization.fuse_modules(model_fp32, ["linears[0]","linears[1]" ])  
-&gt; throws error: AttributeError: 'MyModule' object has no attribute 'linears[0]'
</code></pre>

Vasiliy_Kuznetsov: <p>hi <a class="mention" href="/u/hirenpatel1207">@hirenpatel1207</a> ,</p>
<p>if you use <code>["linears.0","linears.1"]</code> instead of <code>["linears[0]","linears[1]" ]</code>, it should work.</p>

hirenpatel1207: <p>This works!!! Thanks…  <img src="https://discuss.pytorch.org/images/emoji/apple/raised_hands/3.png?v=9" title=":raised_hands:t3:" class="emoji" alt=":raised_hands:t3:"></p>
I want to know module type in c++ environment

111480: <p>In a c++ (libtorch) environment, I obtain submodule of Torchvison’s models using children() and print type() and then it is printed in binary form. I want to know the type like “nn.Conv2d” of submodule.<br>
What should I do?</p>

jbschlosser: <p>Not sure there’s any way to get just the name “Conv2d”, but the <code>pretty_print()</code> function outputs the name (alongside other module properties).</p>
<p>As a workaround, you could use that with, for example, a <code>std::stringstream</code> and do some post-processing on the string to pull out the part you want.</p>
<p>If that doesn’t work for you, I suggest opening a feature request issue on the <a href="https://github.com/pytorch/pytorch" rel="noopener nofollow ugc">PyTorch GitHub page</a>.</p>

thomasjo: <p>Use the <code>name()</code> method on <code>Module</code> perhaps? It relies on <a href="https://en.cppreference.com/w/cpp/language/typeid" rel="noopener nofollow ugc">RTTI</a> to resolve the type name. This is the same function that <code>pretty_print()</code> uses to get the module name.</p>
<p><a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_module.html#_CPPv4NK5torch2nn6Module4nameEv" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_module.html#_CPPv4NK5torch2nn6Module4nameEv</a></p>
Splitting unbalanced classes of images into training, validation and testing sets

Eric_Gill: <p>Hi there,</p>
<p>I have 3 classes of images, Covid, Normal and Viral Pneumonia(VP) in three different folders. There are roughly 3,600; 12,000; and 1,600 images in each folder respectively. As this set is imbalanced, I would like to split it into training, validation and testing subsets while maintaining the proportion of each class in each of the subsets.</p>
<p>I have been doing some research and tried <a href="https://discuss.pytorch.org/t/how-to-do-a-stratified-split/62290">this</a> answer. I loaded all the images using ImageFolder, got an array of the classes (represented as 0, 1 and 2) and retrieved the indexes in a balanced way:</p>
<pre><code class="lang-auto">img_dataset = datasets.ImageFolder(
                              root = first_model_im_path,
                              transform = image_transforms["train"]
                       )

train_idx, test_idx= train_test_split(
np.arange(len(class_array)),
test_size=0.2,
shuffle=True,
stratify=class_array)
</code></pre>
<p>So far so good, but I have hit the wall at this point. The next step would of course be taking that train_idx set of indices and splitting it further, but I also have to guarantee that it remains balanced. Does anyone have any idea how to proceed? Or indeed, is there a better way to achieve what I am attempting?</p>
<p>I appreciate the help, thanks so much in advance,<br>
Eric</p>

ptrblck: <p>You could call <code>train_test_split</code> again on <code>train_idx</code> instead of the <code>np.arange</code> input with <code>stratify=class_array[train_idx]</code> to create the train and validation indices. Let me know, if this works.</p>

Eric_Gill: <p>Thanks so much for your help!</p>
Cuda available False but torch.version.cuda 10.1

Chenchao_Zhao: <p>I installed pytorch 1.3 from conda inside a ubuntu container</p>
<p>Then I run the docker with host machine cuda=10.1 driver=418</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.version.cuda
'10.1.243'
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.is_available()
False
</code></pre>
<pre><code class="lang-auto">torch.backends.cudnn.enabled = True
</code></pre>

ptrblck: <p>Could you post the log of the installation, please?</p>

Chenchao_Zhao: <p>I installed it using official conda install</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
</code></pre>
<p>the install was pretty normal (I will try to get the installation log but maybe it’s not very informative).<br>
I also tried official pytorch/pytorch:latest docker image but still torch could find cuda version but cuda available was false.</p>
<p>The host machine I’m using doesn’t have nvidia-docker or support nvidia runtime. The gpu devices and libcuda* were manually mounted (if i ran nvidia-smi the docker was able to find gpus, driver version, and cuda version).</p>
<p>Are there ENV variables that pytorch search for in order to bind to cuda devices?</p>

ptrblck: <aside class="quote no-group" data-username="Chenchao_Zhao" data-post="3" data-topic="65945">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenchao_zhao/48/36481_2.png" class="avatar"> Chenchao_Zhao:</div>
<blockquote>
<p>The host machine I’m using doesn’t have nvidia-docker or support nvidia runtime.</p>
</blockquote>
</aside>
<p>Are you using plain <code>docker</code> without the <code>nvidia-runtime</code> then?<br>
Would it be possible to install <code>nvidia-docker</code> on this machine?<br>
I’m no expert in using docker, but I wouldn’t recommend trying to mount the GPUs somehow manually.</p>

Chenchao_Zhao: <p>Hi ptrblck,</p>
<p>Thanks!<br>
I don’t have the permission to customize the host machine unfortunately.<br>
I also ran some other check</p>
<pre><code class="lang-auto">torch.cuda.is_driver_compatible() = True
</code></pre>
<p>I can see the nvidia0, nvidia1 … in /dev directory. somehow pytorch cannot find them</p>

TLi347: <p>I encounter the same problem. Have you solve this yet? thanks.</p>

Chenchao_Zhao: <p>The problem was solved by upgrading the infrastructure</p>
Translate weight initialization from tensorflow to pytroch

Mikel_B: <p>How would you go about initializing this TF fully connected layer</p>
<blockquote>
<p>Dense(2, activation=‘linear’, kernel_initializer=RandomNormal(mean=0.0, stddev=0.2, seed=seed),<br>
bias_initializer=Constant(value=[3.,-3.]), name=‘pi’)</p>
</blockquote>
<p>in PyTorch?</p>
<p>I tried this:</p>
<blockquote>
<p>self.pi = nn.Linear(input_dim, 2)<br>
nn.init.normal_(self.pi.weight, mean=0, std=0.2)<br>
nn.init.constant_(self.pi, 0)</p>
</blockquote>
<p>But I am not sure about the last line…<br>
Thank you for your time</p>

Nikronic: <p>Hi,</p>
<p>About the last line, you need to initialize it like the way you have initialized your weights. In other words, use <code>self.pi.bias</code> instead of <code>self.pi</code> which is the module.</p>
<p>Also, <code>nn.init.constant_(self.pi, 0)</code> initialized the entire <code>bias</code> tensor with zeroes while the TF counterpart has initialized it with <code>[3, -3]</code>. To do so, you can simply assign a tensor with these values to the <code>bias</code> attribute of <code>self.pi</code>:</p>
<pre><code class="lang-auto">with torch.no_grad():
    self.pi.bias = nn.Parameter(torch.tensor([3., -3.]))
</code></pre>
<aside class="quote" data-post="2" data-topic="45902">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-manually-set-the-weights-in-a-two-layer-linear-model/45902/2">How to manually set the weights in a two layer linear model?</a> 
  </div>
  <blockquote>
    You would just need to wrap it in a torch.no_grad() block and manipulate the parameters as you want: 
model = torch.nn.Sequential(nn.Linear(10, 1, bias=False))

with torch.no_grad():
    model[0].weight = nn.Parameter(torch.ones_like(model[0].weight))
    model[0].weight[0, 0] = 2.
    model[0].weight.fill_(3.)
    ...
  </blockquote>
</aside>
Strange behavior of logsumexp

YazidJanati: <p>Hi,</p>
<p>There seems to be a strange behavior regarding torch.logsumexp. I get different results depending on the dimension and value of the vector, is this normal?</p>
<pre><code class="lang-auto">a = torch.tensor(-130.)
b = torch.tensor(-130.).view(1,1)

torch.logsumexp(a,0) #returns -130
torch.logsumexp(b,0) #returns -130
torch.logsumexp(b,(0,1)) #returns -inf
</code></pre>
<p>but if i take a smaller number i get</p>
<pre><code class="lang-auto">a = torch.tensor(-80.)
b = torch.tensor(-80.).view(1,1)

torch.logsumexp(a,0) #returns -80
torch.logsumexp(b,0) #returns -80
torch.logsumexp(b,(0,1)) #returns -80
</code></pre>
<p>For an intermediate number, i don’t get -inf but a slightly different number</p>
<pre><code class="lang-auto">a = torch.tensor(-100.).view(1,1)

torch.logsumexp(a,(0,1)) #returns -99.981
</code></pre>
<p>Is this normal?</p>
<p>Thank you</p>

KFrank: <p>Hi Yazid!</p>
<aside class="quote no-group" data-username="YazidJanati" data-post="1" data-topic="116465" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yazidjanati/48/36474_2.png" class="avatar"> YazidJanati:</div>
<blockquote>
<pre><code class="lang-auto">a = torch.tensor(-100.).view(1,1)

torch.logsumexp(a,(0,1)) #returns -99.981
</code></pre>
</blockquote>
</aside>
<p>I can’t reproduce this on pytorch version 1.7.1.  (It does look like a<br>
minor bug.)  Could you tell us what version you are using?</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.logsumexp (torch.tensor (-130.).view (1, 1), (0, 1))
tensor(-130.)
&gt;&gt;&gt; torch.logsumexp (torch.tensor (-100.).view (1, 1), (0, 1))
tensor(-100.)
&gt;&gt;&gt; torch.logsumexp (torch.tensor (-80.).view (1, 1), (0, 1))
tensor(-80.)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

YazidJanati: <p>Thanks KFrank for your quick reply.</p>
<p>I’m on 1.6.0. Just updated to 1.7.0 and the bug disappeared.</p>