akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Where is the C/C++ code for torch.nn.functional.fold()?

zfzhang: <p>Hi,<br>
I’m new to Pytorch C/C++ extention. My question is naive: Where is the C/C++ code for torch.nn.functional.fold()?</p>
<p>Thank you!</p>

zfzhang: <p>Ok, I’ve found it.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/torch/csrc/api/include/torch/nn/functional/fold.h">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/torch/csrc/api/include/torch/nn/functional/fold.h" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/torch/csrc/api/include/torch/nn/functional/fold.h" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/torch/csrc/api/include/torch/nn/functional/fold.h</a></h4>


    <pre><code class="lang-h">#pragma once

#include &lt;torch/nn/options/fold.h&gt;

namespace torch {
namespace nn {
namespace functional {

#ifndef DOXYGEN_SHOULD_SKIP_THIS
namespace detail {
inline Tensor fold(const Tensor&amp; input,
                   ExpandingArray&lt;2&gt; output_size,
                   ExpandingArray&lt;2&gt; kernel_size,
                   ExpandingArray&lt;2&gt; dilation,
                   ExpandingArray&lt;2&gt; padding,
                   ExpandingArray&lt;2&gt; stride) {
  if (input.dim() == 3) {
    return torch::col2im(
        input,
        output_size,
</code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/torch/csrc/api/include/torch/nn/functional/fold.h" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


InnovArul: <p>Just incase if you want to dig deeper on <code>col2im</code>:</p>
<p>Native C++: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Col2Im.cpp" class="inline-onebox">pytorch/Col2Im.cpp at master · pytorch/pytorch · GitHub</a></p>
<p>Cuda code:<br>
<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Col2Im.cu" class="inline-onebox">pytorch/Col2Im.cu at master · pytorch/pytorch · GitHub</a>,<br>
<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/im2col.cuh" class="inline-onebox">pytorch/im2col.cuh at master · pytorch/pytorch · GitHub</a></p>

zfzhang: <p>Thanks for the useful information!<br>
A follow-up question is: If I want to write a modified version of col2im() with a different name, can I write it in C++ extensions? Or must I write it inside the ATen library and recompile?</p>

InnovArul: <p>I haven’t tried it. In my understanding, it should be possible to write a modified version of col2im in a standalone way than inside ATen.</p>

zfzhang: <p>Good, thanks for the info!</p>
How to skip some batch when using distributed model for training?

Ferd: <p>I use torch.multiprocessing to launch distributed training, but some batches may raise cuda_out_of_memory exception. I just wanna skip these batches. I can successfully skip them when using only one GPU for traning by using try and except.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1a05c4e411d448f4486722b763eb75c55adee5b.png" alt="image" data-base62-sha1="rCTIOb7pXDP3nSo4rSoNm8s5D0D" width="629" height="177"></p>
<p>But it dosen’t work for distributed training case, the training process will just stuck. I guess it may caused by the communication between different threads.</p>
<p>I’d be appreciated if someone could help me.</p>

rvarm1: <p>Yes, doing something similar will indeed result in stuckness issues because some ranks will likely have kicked off communication, and other ranks could be skipping the batch so there will be an inconsistency in terms of no. of collective calls launched.</p>
<p>The tricky thing here is that you need to know a-priori whether a batch will be skipped or not, so that it can be communicated consistently across all distributed processes.</p>
<p>Do you know why some batches result in OOM? Are the sizes somehow different across batches, and can you use the size to estimate whether you’ll need to skip it?</p>

Ferd: <p>thanks! it is a object detection task, the memory will fluctuate because the targets number varies. and i set a maximum target number now, thus i can successfully train the model. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Ctrl+c to stop distributed

Ardeal: <p>Hi,</p>
<p>I tried 2 methods to start distributed:<br>
method 1:<br>
call torch.multiprocessing.spawn function to start n processes. on 1 computer with multi-GPUs</p>
<p>method 2:<br>
call torch.distributed.launch to start n processes on 1 computer with multi-GPUs</p>
<p>if I used method 1, and used ctrl + c to stop code, sub-processing will not stop.<br>
if I used mehod 2, and used ctrl + c to stop code, sub-processing will stop.</p>
<p>my questions are:</p>
<ol>
<li>for method 1, how to stop sub-processing in python code?</li>
<li>for method 1, could the start code be run in python code?</li>
</ol>
<pre><code class="lang-python">#!/bin/bash
NUM_PROC=$1
shift
python3 -m torch.distributed.launch --master_port=44145 --nproc_per_node=$NUM_PROC train.py "$@"
</code></pre>
Extracting the weights from a saved model's state_dict()?

avalon1511: <p>I have a saved <code>state_dict()</code> in  a <code>.pth</code> file and I’m trying to extract only the weights from this dictionary.</p>
<p>I tried loading the state_dict() using <code>load_state_dict()</code> and accessing the weights as follows</p>
<pre><code class="lang-auto">m = model.load_state_dict()
print(m['weight'])
</code></pre>
<p>However, this is giving me the following error: <code>TypeError: tuple indices must be integers or slices, not str </code></p>
<p>which tells me I can’t access the weights by just using the key, how can I obtain the weights in this case?</p>

eqy: <p>I’m not actually sure what the return value of <code>load_state_dict</code> is, is it the state dict itself?<br>
Does</p>
<pre><code class="lang-auto">m = model.load_state_dict()
d = model.state_dict()
print(d.keys())
</code></pre>
<p>work for you?</p>

avalon1511: <p>So I looked it up and <code>load_state_dict()</code> is just initializing m to whatever my previously saved model is. Also I tried what you suggested with a slight modification:</p>
<pre><code class="lang-auto">m = model.load_state_dict()
d = m.state_dict()
print(d.keys())
</code></pre>
<p>and it printed the list of keys in the state_dict!</p>
<p>So I see that the keys are labelled as <code>main.0.weight, main.1.weight, main.1.bias</code> and so on - I want the weights and biases separately, so do I just filter by <code>type</code> ? Is there an easier way to do this?</p>
What is the use of conv2 layer in torchvision's googlenet implementation?

jsg: <p>In torchvision’s googlenet implementation, where is 1x1 convolution layer in the stem region.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/vision/blob/master/torchvision/models/googlenet.py#L92">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/googlenet.py#L92" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/torchvision/models/googlenet.py#L92" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/master/torchvision/models/googlenet.py#L92</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="82" style="counter-reset: li-counter 81 ;">
          <li>assert len(blocks) == 3</li>
          <li>conv_block = blocks[0]</li>
          <li>inception_block = blocks[1]</li>
          <li>inception_aux_block = blocks[2]</li>
          <li>
          </li>
<li>self.aux_logits = aux_logits</li>
          <li>self.transform_input = transform_input</li>
          <li>
          </li>
<li>self.conv1 = conv_block(3, 64, kernel_size=7, stride=2, padding=3)</li>
          <li>self.maxpool1 = nn.MaxPool2d(3, stride=2, ceil_mode=True)</li>
          <li class="selected">self.conv2 = conv_block(64, 64, kernel_size=1)</li>
          <li>self.conv3 = conv_block(64, 192, kernel_size=3, padding=1)</li>
          <li>self.maxpool2 = nn.MaxPool2d(3, stride=2, ceil_mode=True)</li>
          <li>
          </li>
<li>self.inception3a = inception_block(192, 64, 96, 128, 16, 32, 32)</li>
          <li>self.inception3b = inception_block(256, 128, 128, 192, 32, 96, 64)</li>
          <li>self.maxpool3 = nn.MaxPool2d(3, stride=2, ceil_mode=True)</li>
          <li>
          </li>
<li>self.inception4a = inception_block(480, 192, 96, 208, 16, 48, 64)</li>
          <li>self.inception4b = inception_block(512, 160, 112, 224, 24, 64, 64)</li>
          <li>self.inception4c = inception_block(512, 128, 128, 256, 24, 64, 64)</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>However, from this paper(<a href="https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43022.pdf" rel="noopener nofollow ugc">https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43022.pdf</a>),</p>
<p>I cannot find any clue where this layer came from.</p>
<p>Is this an error?</p>
Multiple loss backward

Yingtao_Huo: <pre><code class="lang-auto">a = cross_entropy(output, label)
b = nllloss(output, label)
a.backward(retain_graph=True)
b.backward()
</code></pre>
<p>Memory used by CUDA will increase until memory out.</p>
<p>However</p>
<pre><code class="lang-auto">a = cross_entropy(output, label)
a += nllloss(output, label)
a.backward()
</code></pre>
<p>The problem mentioned before will not appear.</p>

ptrblck: <p>The peak memory usage would be higher in the first example, since the gradients for each parameters will be calculated a second time and accumulated to the already calculated <code>.grad</code> attribute.<br>
Besides that the allocated memory should be equal as seen here:</p>
<pre><code class="lang-python">device = 'cuda'
model = models.resnet50().to(device)
x = torch.randn(1, 3, 224, 224, device=device)
label = torch.randint(0, 1000, (1,), device=device)
output = model(x)

if False:
    a = F.cross_entropy(output, label)
    b = F.nll_loss(output, label)
    a.backward(retain_graph=True)
    b.backward()
else:
    a = F.cross_entropy(output, label)
    a += F.nll_loss(output, label)
    a.backward()

print(torch.cuda.memory_summary())
</code></pre>
Using Custom ResNet32 Model Trained by Fastai on Pytorch

yildirimege: <p>Hey, I am currently trying to use a model I trained with FastAI ( Colab: <a href="https://colab.research.google.com/drive/1sBr-wdxenj5Koe8WRvMfgPGN5JQX2BuO#scrollTo=92xBBHHnpVXQ" class="inline-onebox" rel="noopener nofollow ugc">Google Colaboratory</a>) on Pytorch. Here are the steps I use on pytorch side:</p>
<p>//Create the instance of ResNET34 model, if I don’t use the num_classes = 2 overload, I end up with incompatible size error on output layer of the neural network (By default ResNET model has 1000 on Fc layer even though my trained model is trained on fc layer with size 2?).</p>
<p>model = torchvision.models.resnet34(pretrained=False, num_classes = 2)<br>
model_dict = model.state_dict()<br>
loc =  torch.load(“models/stage-2.pth”, map_location=“cpu”)[‘model’]</p>
<pre><code>model.load_state_dict(
    loc, strict=False
)
</code></pre>
<p>transform = torchvision.transforms.ToTensor()<br>
image = transform(image).unsqueeze(0)<br>
result = model(image)</p>
<p>However, my results make no sense. Sometimes it shows an output like:</p>
<p>tensor([[ 3.6697, -6.6085]], grad_fn=)</p>
<p>And when I rerun I end up with something like;</p>
<p>dim=tensor([[ 2.4605, -5.6492]], grad_fn=)</p>
<p>Can anyone lead me on which step I’m missing? Thanks!</p>

yildirimege: <p>I trained testing the model with the images I had on training folder. It still gives this input:</p>
<p>Fake Result: tensor([[ 4.7905, 13.1582]], grad_fn=)<br>
Real Result: tensor([[ 3.6796, 10.8140]], grad_fn=)</p>
<p>Something is definetly off here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I’m unsure what the concern is and why the outputs wouldn’t make sense, so please add some more details in case I misunderstand it.<br>
For multi-class classification/segmentation use cases usually <code>nn.CrossEntropyLoss</code> is used as the criterion, which expects raw logits as the model output. These logits are not bound to a specific range and can take arbitrary values. If you want to see the probabilities, you could apply a <code>softmax</code> on the output (but don’t pass the softmax output to <code>nn.CrossEntropyLoss</code>).</p>
Logdet of a big matrix: Autograd is too slow

Amine08: <p>Hi,</p>
<p>in this case autograd is too slow. This takes using my GPU Tesla V100-SXM2 about 0.5s. Any suggestions to accelerate it?</p>
<pre><code class="lang-auto">import torch
import time
device = "cuda" if torch.cuda.is_available() else "cpu"
w=torch.ones(8000,8000,requires_grad=True,device=device)
loss=torch.logdet(w)
time_start=time.time()
loss.backward()
print(time.time()-time_start)
</code></pre>
<p>Thanks!</p>

tom: <p>If you have algebraic structure such as s.p.d. matrices, it might beworth doing the cholesky and take (twice) the logsum of the diagonal. This can be more efficient than the sowhat elaborate logdet backward.</p>
<p>Best regards</p>
<p>Thomas</p>

Amine08: <p>Right, that works also when the matrix is triangular. Unfortunately this is not my case <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=9" title=":expressionless:" class="emoji" alt=":expressionless:"></p>

Amine08: <p>How works backward exactly? is logdet executed several times?</p>

tom: <p>Looking at the source,</p>
<ul>
<li>the <a href="https://github.com/pytorch/pytorch/blob/666dff381d887eccfe6d1b0ee22cff65e79230d2/aten/src/ATen/native/LinearAlgebra.cpp#L109-L130" rel="noopener nofollow ugc">forward uses LU decomposition</a>,</li>
<li>the <a href="https://github.com/pytorch/pytorch/blob/666dff381d887eccfe6d1b0ee22cff65e79230d2/torch/csrc/autograd/FunctionsManual.cpp#L2650-L2662" rel="noopener nofollow ugc">backward does a SVD and then calls SVD’s backward</a>.</li>
</ul>
<p>The original <a href="https://github.com/pytorch/pytorch/pull/5393" rel="noopener nofollow ugc">PR introducing logdet</a> has some discussion around the reasoning, but I must admit I don’t fully follow it (and maybe things have changed since).</p>
<p>Note that your benchmarking is wrong for CUDA: You need to torch.cuda.synchronitze() before taking times (both start and end).</p>

Amine08: <p>I see! Thank you Thomas! Can i for example accelerate torch.cholesky or torch.logdet by using many GPUs/CPUs simultaneously?</p>
TFX equivalent in pytorch

Isaac_Kargar: <p>Hi,</p>
<p>I’m trying o learn how to use PyTorch in production. There are a lot of tools out there and I’m confused a little bit. I see most of the books and tutorials on Tensorflow. It has TFX for example. Is torchscript the equivalent to TFX? What pipeline and tools do you suggest to use for PyTorch in production? What is the drawback of using sth like flask+docker+kubernetes to put the trained model in production?</p>
<p>Any link or reference with a complete comparison would be appreciated.</p>
<p>Thank you</p>

Isaac_Kargar: <p>I found <a href="https://github.com/pytorch/serve/blob/14f3883382503bc040d767c40294a8b3c2b07024/docs/README.md" rel="noopener nofollow ugc">this</a> and it seems it is similar to TFX. One more question? Is it again (using torchserve) slower than using a saved model using torchscript in C++ or JS?</p>

tom: <p>There also is <a href="https://pytorch.org/torchx/latest/" rel="noopener nofollow ugc">TorchX</a>…</p>
How does graph network transform MNIST data into probability vector? Pytorch-Geometric

szymonindy: <p>I am analyzing example from pytorch-geometric library and I have a question.<br>
How does graph network transform MNIST data into probability vector in pytorch geometric?<br>
Here is the GCN:</p>
<pre><code class="lang-auto">def normalized_cut_2d(edge_index, pos):

    row, col = edge_index

    edge_attr = torch.norm(pos[row] - pos[col], p=2, dim=1)

    return normalized_cut(edge_index, edge_attr, num_nodes=pos.size(0))

class GCN1(torch.nn.Module):

    def __init__(self, d):

        print(d.num_features)

        super(GCN1, self).__init__()

        self.conv1 = SplineConv(d.num_features, 32, dim=2, kernel_size=5)

        self.conv2 = SplineConv(32, 64, dim=2, kernel_size=5)

        self.fc1 = torch.nn.Linear(64, 128)

        self.fc2 = torch.nn.Linear(128, d.num_classes)

    def forward(self, data):

        data.x = F.elu(self.conv1(data.x, data.edge_index, data.edge_attr))

        weight = normalized_cut_2d(data.edge_index, data.pos)

        cluster = graclus(data.edge_index, weight, data.x.size(0))

        data.edge_attr = None

        data = max_pool(cluster, data, transform=transform)

        data.x = F.elu(self.conv2(data.x, data.edge_index, data.edge_attr))

        weight = normalized_cut_2d(data.edge_index, data.pos)

        cluster = graclus(data.edge_index, weight, data.x.size(0))

        x, batch = max_pool_x(cluster, data.x, data.batch)

        x = global_mean_pool(x, batch)

        x = F.elu(self.fc1(x))

        x = F.dropout(x, training=self.training)

        return F.log_softmax(self.fc2(x), dim=1)
</code></pre>
Values of 17 from policy network although it should be between -1-1

TheUnnamed22: <p>Hello,</p>
<p>my policy network give values between 0-17 although it should be between -1 - 1.<br>
The network consists of 4 linear layers and relu activations. I am taking the raw data of the network. How can i let the network predict values between -1  and 1 and not between -inf to +inf ?</p>
<p>Do i have to add layernom between the layers?</p>
torch.nn.CrossEntropyLoss() torch.Size([32, 96, 128]) 3 dimension

PatricYan: <p>loss = torch.nn.CrossEntropyLoss()</p>
<p>loss_values = loss(train_output, train_label)</p>
<p>***** train_label shape: torch.Size([32, 96, 128])<br>
***** train_output shape: torch.Size([32, 5, 96, 128])<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f688b9ae4c4cb4ed8ab6385d4a66322ad9c7bbc.png" data-download-href="https://discuss.pytorch.org/uploads/default/4f688b9ae4c4cb4ed8ab6385d4a66322ad9c7bbc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f688b9ae4c4cb4ed8ab6385d4a66322ad9c7bbc_2_690x144.png" alt="image" data-base62-sha1="bktHREFTdaE1CQz8lOh5GmWTCNu" width="690" height="144" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f688b9ae4c4cb4ed8ab6385d4a66322ad9c7bbc_2_690x144.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f688b9ae4c4cb4ed8ab6385d4a66322ad9c7bbc_2_1035x216.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f688b9ae4c4cb4ed8ab6385d4a66322ad9c7bbc_2_1380x288.png 2x" data-dominant-color="35383C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1616×338 54.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
when chang it to Long type, or float type , it will get new error</p>

safakkbilici: <p>I assume that 32 is your batch size. <code>torch.nn.CrossEntropyLoss()</code> is expecting you to pass output with dimensions <code>[batch_size, n_classes, seq_len]</code> and training label with dimensions <code>[batch_size, seq_len]</code> which cointains training class labels.</p>
<p>Also you can pass input with dimensions <code>(N,C)</code> and target with dimensions <code>(N)</code>, from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">documentation</a>.</p>
<p>I do not know what your task is but I hope this will help you.</p>

PatricYan: <p>Hi, thanks. I want to use it for FCN pixels, now I don’t know how to check it. For FCN input and output have the same torch size. Do you have any idea?</p>

safakkbilici: <aside class="quote no-group" data-username="PatricYan" data-post="1" data-topic="126433">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/patricyan/48/40086_2.png" class="avatar"> PatricYan:</div>
<blockquote>
<p>torch.Size([32, 96, 128])</p>
</blockquote>
</aside>
<p>What do 96 and 128 represent for size <code>[32, 96, 128]</code>?. Can you give more information about your task?</p>
Confused behavior of autograd

yuhaoooo: <p>I try to implement a higher order gradient to optimizing parameters of networks, i.e.,</p>
<pre><code class="lang-auto">var_input = autograd.Variable(input_data, requires_grad=True)
loss = model(var_input)
gradient = autograd.grad(loss, input=var_input, create_graph=True, retain_graph=True, only_inputs=True)[0]
train_loss = loss_fn(gradient)
gradients = autograd.grad(train_loss, input=model.parameters(), create_graph=False, retrain_graph=False, only_inputs=True)
</code></pre>
<p>However, I got an error <code>One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</code></p>
<p>Could you tell me how to checkout which one tensor is not in the graph?</p>

tom: <p>This will happen if the loss is linear in a parameter and thus the first gradient does not depend on it.<br>
The quickest way to find which one is to use <code>allow_unused=True</code> in the grad call and then check the gradients it returns.</p>
<p>Best regards</p>
<p>Thomas</p>

yuhaoooo: <p>Thanks for your timely response.<br>
What is the meaning of " the loss is linear in a parameter and thus the first gradient does not depend on it"?<br>
Do you mean that, assuming y = 2x, d(dy/dx)/dx = 0? But why does PyTorch just set the gradient to 0?</p>

tom: <p>If you have this for the first derivative, it likely <em>is</em> a bug, only with higher order derivatives this is benign.</p>
Inserting part of tensor into another

will-leeson: <p>I have two tensors, A and B. A is of dimension [n+1, m]. B is of dimension [n,k] and all value are from -1 to m-1. What I need to to is check if the value in if any value in B is equal to -1. If it is, I will set it to value A[n+1]. If it is not, I will set the value to A[i]. For example, let A = [x_0, x_1, x_2, x_3,x_4] and B = [[0,-1,-1],[1,2,-1],[2,0,1]]. Then, I want to end up with C = [[x_0,x_4,x_4],[x_1,x_1,x_4],[x_3,x_3,x_3]]. I want this to be as efficient as possible, so I’m trying to avoid loops and stick to torch functions.</p>
<p>Any suggestion?</p>

ptrblck: <p>I don’t quite understand the desired output, as it seems you are not setting the values in <code>C</code> to <code>A[i]</code> if <code>B[i]!=-1</code>.<br>
E.g. the second row is given as <code>[x_1, x_1, x_4]</code> and the last one as <code>[x_3, x_3, x_3]</code>. Could you explain a bit more how these results are created?</p>

will-leeson: <p>I think this summarizes it well. If B[i][j]!=-1, then C[i][j]=A[i]. else C[i][j]=A[-1]. I tried using where, but An issue I’m running into is A[i][j].size() != B[i][j].size(). I should specify, x_i is a tensor of length &gt; 1</p>

will-leeson: <p>If it helps, this loop does what I want, but I feel like there is a pytorch way to do what I want without a loop:<br>
C = []<br>
for x in range(len(B)):<br>
CPrime= []<br>
for y in range(len(B[x])):<br>
if B[x][y]!=-1:<br>
CPrime.append(A[x])<br>
else:<br>
CPrime.append(A[-1])<br>
C.append(torch.stack(CPrime))<br>
C = torch.stack(C)<br>
(Sorry for formatting, I’m new to the forums)</p>
<p>where(B!=1, A, A[-1]) would be what I want, but this doesnt work because A is 3 dimensional and B is two dimensional</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">A = torch.arange(5)
B = torch.tensor([[0,-1,-1],[1,2,-1],[2,0,1]])
C = []
for x in range(len(B)):
    CPrime= []
    for y in range(len(B[x])):
        if B[x][y]!=-1:
            CPrime.append(A[x])
        else:
            CPrime.append(A[-1])
    C.append(torch.stack(CPrime))
C = torch.stack(C)
print(C)


D = A[torch.arange(B.size(0))].unsqueeze(1).expand_as(B).clone()
D[B==-1] = A[-1]
print((C==D).all())
&gt; tensor(True)
</code></pre>
<p>You can post code snippets by wrapping them into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

will-leeson: <p>Thanks so much. I had to make a slight change since A is multidimensional. Here is what I ended up with:</p>
<pre><code class="lang-auto">D = A[torch.arange(B.size(0))].unsqueeze(1).expand(-1, B.size(1),-1).clone()
D[B==-1]=torch.full_like(A[0], -float("inf"))
</code></pre>
Batchnorm/Layernorm

TheUnnamed22: <p>Instead of normalizing the data yourself is it possible to just put a layernorm or batchnorm layer in front of your nn and it normalizes the data over the last 100 data points? And lets say i have as input 6 features and i want to normalize over last 100 inputs how to set up the layernorm/batchnorm? How would it look in code?</p>
Jit compile custom C++ function with OpenMP threads

Donnie_Blanco: <p>Hi,</p>
<p>I am writing my custom layers in C++. Now I have created the Linear layer and I want to parallelize the GEMM function (I use a custom naive gemm). Can I jit compile with openmp threads? Because when I try to use openmp pragma for a parallel for, it seems pytorch give me weird errors that it does not know some of my Aten functions. i.e. (‘Tensor’ does not name a type; did you mean ‘THTensor’)</p>
<p>Thanks in advance</p>
MPDQN - parameter values going to the moon

TheUnnamed22: <p>Hello,</p>
<p>i am implementing an actor-critic - MP-DQN. Now in the toy environment it works great and the policy network produces values between -1 and 1. If i use the same network with same hyper parameters in my own environment the values of the policy network  going up and up or down and down. After tanh squashing the parameters are stuck at 1 or -1 and don’t really produce parameters based on the actual state. So that the agent only has for example the option to turn full left or only to turn full right. What might be the problem? I am new to this field and idk what to look for. What do i have to check/look for? What do i have to adjust? Would a hyperparameter obtimisation work or hast it to do with the input data? The input data is normalized between -1 and 1 and some between 0  and 1.</p>
<p>Thanks for your help!</p>
The fastest optimizer, loss for CNN image classification problem

andreiliphd: <p>Hello!<br>
I need to fast converge a neural network in a limited number of epochs. Image classification: CNN<br>
What I discovered:</p>
<ol>
<li>The network should be shallow.</li>
<li>Learning rate balance is needed to converge fast.</li>
</ol>
<p>Could you give me the fastest converging optimizer and recommendation for learning rate and activation function in the last layer to finish what I need?</p>
<p>Great thank you!</p>

ptrblck: <p>Fast.ai performed some interesting experiments with super-convergence and released a blog post about it <a href="https://www.fast.ai/2018/07/02/adam-weight-decay/" rel="nofollow noopener">here</a>. Maybe it could also be applicable to your use case.</p>

andreiliphd: <p>I found great article <a href="https://medium.com/octavian-ai/which-optimizer-and-learning-rate-should-i-use-for-deep-learning-5acb418f9b2" rel="nofollow noopener">https://medium.com/octavian-ai/which-optimizer-and-learning-rate-should-i-use-for-deep-learning-5acb418f9b2</a></p>
<p>It’s a great research and I solved the problem.<br>
Thank you very much for help!</p>

J_Johnson: <p>If accuracy is important to you, SGD is certainly a safe bet. However, it is also incredibly slow.</p>
<p>But nothing in “the book” says you can’t start with a faster optim, such as Adagrad or Adadelta, and then switch to SGD for the final few epochs, just to squeeze that last bit of accuracy out of a model. I find this to be the best approach.</p>
<p>Happy inferencing,</p>
Training Loss fluctuates, decrease then increase and vice-versa

Rajat_Sharma1: <p>I am trying to optimize a model using ADAM optimizer. The initial learning rate is set to 0.01 with weight decay of 0.00001. I am reducing the learning rate after every 10th epoch by a factor of 0.1. The model optimizes for few epoch and then suddenly jumps. This process repeats for few times. What could be the reason for this?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c7633531095414ff21f260f6d7e2e435a5b4869.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c7633531095414ff21f260f6d7e2e435a5b4869" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c7633531095414ff21f260f6d7e2e435a5b4869_2_690x245.png" alt="image" data-base62-sha1="mk7JbmPbOvvHZxveIKchn9iynAB" width="690" height="245" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c7633531095414ff21f260f6d7e2e435a5b4869_2_690x245.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c7633531095414ff21f260f6d7e2e435a5b4869_2_1035x367.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c7633531095414ff21f260f6d7e2e435a5b4869_2_1380x490.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2404×854 182 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Define dataset class for Fully Convolutional Networks

harshatejas: <p>Hi,</p>
<p>I have a custom segmentation dataset. How do i define the dataset class for <code>torchvision.models.segmentation.fcn_resnet50</code></p>
<p>I learned how to define the dataset class for Faster-RCNN from <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>
<p>It would be helpful if someone could help me get started with defining dataset class and loading the data for <code>torchvision.models.segmentation.fcn_resnet50</code></p>
<p>Thanks You.</p>
Why the output of children part of a network has low resolution?

students: <p>Hello. I train one network and save it, then load this network and I partition network to three module and give one images to input of first module and take output from end module but the output of this three module has low resolution… I dont understand why this happened?!!</p>
<pre><code class="lang-auto">model =  ConvAutoencoder()
max_epochs =10
outputs = train(model, num_epochs=max_epochs)

torch.save(model, 'model1.pth')
model1 = torch.load('model1.pth')

imgs3, labels = next(iter(test_loader))

f= model1(imgs3)

plt.imshow(f[13][0].detach().numpy(), cmap="gray") 

l1 = nn.Sequential(*list(model1.children())[:3])
print('l1=',l1)

l2 = nn.Sequential(*list(model1.children())[3:4])
print('l2=',l2)

l3 = nn.Sequential(*list(model1.children())[4:])
print('l3=',l3)

imgs2, labels = next(iter(test_loader))

y1= l1(imgs2)
#y1.shape

y2=l2(y1)

y3=l3(y2)


plt.subplot(1, 2, 1)
plt.imshow(imgs2[13][0].detach().numpy(), cmap="gray") 
plt.subplot(1, 2, 2)
plt.imshow(y3[13][0].detach().numpy(), cmap="gray") 
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a17a59c49666dfc4b8864b4ab9636ae9b9b6b75.png" alt="without shift2" data-base62-sha1="azrZKZPmycdeTOgHMX8oZrqgJ37" width="368" height="184"></p>

students: <p>pls reply this question</p>

students: <p>please reply to this question.</p>

ptrblck: <p>Based on the posted code I assume the left image represents the input while the right one the model output?<br>
If that’s the case, I guess your model isn’t able to create sharp images and you could check the literature for new architectures, which could avoid the blurry output.</p>
Getting the autograd counter of a tensor in PyTorch

learner47: <p>I am using PyTorch for training a network. I was going through the autograd documentation and <a href="https://pytorch.org/docs/stable/notes/autograd.html#in-place-correctness-checks" rel="noopener nofollow ugc">here</a> it is mentioned that for each tensor there is a counter that the autograd implements to track the “version” of any tensor. How can I get this counter for any tensor in the graph?</p>
<p><strong>Reason why I need it.</strong></p>
<p>I have encountered the autograd error</p>
<blockquote>
<p>[torch.cuda.FloatTensor [x, y, z]], which is output 0 of torch::autograd::CopySlices, is at version 7; expected version 6 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>This is not new to me and I have been successful in handling it before. This time around I am not able to see why the tensor would be at version 7 instead of being at 6. To answer this, I would want to know the version at any given point in the run.</p>
<p>Thanks.</p>

ptrblck: <p>You could access it via <code>tensor._version</code>, but note that it’s an internal argument, so I would use it only for debugging and wouldn’t depend on it for other use cases.</p>
Convert torch tensor to boolean according to max values

valgi0: <p>Hi there,</p>
<p>I have a very newby question …</p>
<p>I have a torch tensor of float values as:</p>
<pre><code class="lang-auto">torch.tensor([[1., 2., 3.], [1., 3., 2.]])
</code></pre>
<p>From it, I want to create a mask vector of 0 and 1 where 1 is the max value of the row:</p>
<pre><code class="lang-auto">torch.tensor([[0,0,1],[0,1,0]])
</code></pre>
<p>I can use the indices returned by torch.max(dim=-1,…) iterate over them and write 1 but I want to know if exists a more fast way to create such mask vectors.</p>
<p>Thank you!</p>

ptrblck: <p>Using the indices of <code>torch.max</code> sounds like a good idea, but you wouldn’t have to iterate them and could use them directly:</p>
<pre><code class="lang-python">x = torch.tensor([[1., 2., 3.], [1., 3., 2.]])
y = torch.zeros_like(x)
y[torch.arange(y.size(0)), x.argmax(dim=1)] = 1.
print(y)
&gt; tensor([[0., 0., 1.],
          [0., 1., 0.]])
</code></pre>
Plotting hidden unit activations

jordanjsg: <p>Hi,</p>
<p>Im working on a network for a project that distinguishes two intertwined rectuangular spirals with which the user can select between a 1 hidden layer or 2 hidden layer model. I’ve got both models working, but am having some trouble creating a plot of the hidden unit activations at each layer which is contained in the graph_hidden() function. This is my code so far :</p>
<pre><code class="lang-auto"># rect.py

import torch
import torch.nn as nn
import matplotlib.pyplot as plt

class Network(torch.nn.Module):
    def __init__(self, layer, hid):
        super(Network, self).__init__()
        self.layer = layer
        
        # Hidden Layer 1
        self.hid1 = nn.Sequential(
            nn.Linear(2, hid),
            nn.Tanh()
        )
        # Hidden Layer 2
        self.hid2 = nn.Sequential(
            nn.Linear(hid, hid),
            nn.Tanh()
        )
        # Output Layer
        self.out = nn.Sequential(
            nn.Linear(hid, 1),
            nn.Sigmoid()
        )

    def forward(self, input):
        if (self.layer == 1):
            output = self.hid1(input)
            output = self.out(output)
            return output
        elif (self.layer == 2):
            output = self.hid1(input)
            output = self.hid2(output)
            output = self.out(output)
            return output

def graph_hidden(net, layer, node): 
    xrange = torch.arange(start=-8,end=8.1,step=0.01,dtype=torch.float32)
    yrange = torch.arange(start=-8,end=8.1,step=0.01,dtype=torch.float32)
    xcoord = xrange.repeat(yrange.size()[0])
    ycoord = torch.repeat_interleave(yrange, xrange.size()[0], dim=0)
    grid = torch.cat((xcoord.unsqueeze(1),ycoord.unsqueeze(1)),1)

    with torch.no_grad():   # suppress updating of gradients
        net.eval()          # toggle batch norm, dropout
        if (layer == 1):
            output = net.hid1(grid)
        elif (layer == 2):
            output = net.hid2(grid)
        net.train()         # toggle batch norm, dropout back again

        pred = (output &gt;= 0.5).float()

        # plot function computed by model
        plt.clf()
        plt.pcolormesh(xrange,yrange,pred.cpu().view(yrange.size()[0],xrange.size()[0]),
                       cmap='Wistia', shading='auto')
        
</code></pre>
<p>Really not sure what im doing here so any pointers are greatly appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Spectrogram into Transfer Learning

Wamiq-Sohail: <p>New to Pytorch!<br>
I am currently working on a project where i am trying to classify ECG Signals. I am using the 2017 Physionet dataset. My approach is converting the 1D signals into 2D spectrograms and using Transfer learning, but i am not getting accuracy better than 72 percent? Any help on how to increase the accuracy on this?</p>
Loss with two different ranges but the same accuracy

mortezamg63: <p>Hi,<br>
I have a question.  I am trying to figure out the answer. I really appreciate you can help me.<br>
I have a recommendation system dataset (including numerical and categorical columns) and I train a  neural network on it. The loss is close to 0.57. The loss function is binary cross entropy.</p>
<p>When I add gaussian noise to numerical columns, the loss shows a value less than zero like 0.57 or a big value like 25.097, but in both of them the accuracy is 74.030.<br>
I am wondering why I have loss in two different ranges with the same the accuracy. I don’t know why it is happening. The random noise can cause change in result, but I don’t get the point of changing the the range of loss.</p>
CNN-RNN-CTCloss for OCR generates Nan in training

mohamad_qodosi: <p>Hi.<br>
I am using a CNN-RNN model for OCR task and then I’m using a CTC loss to train the model. the training is done in GPU and unfortunately I have zero in my target sizes (I can’t remove them).<br>
I have read some other posts about this problem and I know there is a problem in GPU version of CTC loss with zero target size but I can’t train my model on CPU (I have about 10M data samples).<br>
Is there anything I can do to solve this problem?</p>
<p>and by the way, I have traced model outputs and grads. I have seen that in the training procedure, gradients become Nan but the loss is not Nan at same step. and after updating, weights become Nan and the output and then loss becomes Nan.<br>
and also the first time, the gradients of weights and biases of batch normalization layer became Nan. and also the whole grad tensor was Nan.</p>
Libtorch with pure cuda codes cannot compiled

WeNeedMore: <p>OS : win10 64bits<br>
libtorch : 1.9.0+cu102<br>
compiler : vc2019<br>
cuda: 10.2</p>
<p>There are 4 files, CMakeLists.txt, main.cpp, gpu.hpp, gpu.cpp. It is fine it I include torch/torch.h in main.cpp, but the compiler always tell me  “Cannot open include file: ‘torch/torch.h’” if I declare torch.h in gpu.hpp.</p>
<p>CMakeLists.txt</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.5.1)

message(STATUS "CMake version: ${CMAKE_VERSION}")

project(diana)

option(USE_CUDA "Use CUDA" ON)

file(GLOB cpu_source_files "${CMAKE_SOURCE_DIR}/src/*.cc")

file(GLOB gpu_source_files "${CMAKE_SOURCE_DIR}/src/*.cu")

set(3RDLIBS_PATH ${CMAKE_SOURCE_DIR}/../../3rdLibs)

set(LIBTORCH_PATH ${3RDLIBS_PATH}/libtorch/libtorch-win-shared-with-deps-1.9.0+cu102/libtorch)

set(CMAKE_PREFIX_PATH "${LIBTORCH_PATH};${CMAKE_PREFIX_PATH};")

add_executable(diana ${cpu_source_files})

find_package(CUDA 10.2)

find_package(Torch REQUIRED)

CUDA_ADD_LIBRARY(diana_gpu ${gpu_source_files})

target_link_libraries(diana diana_gpu ${TORCH_LIBRARIES})
</code></pre>
<p>main.cc</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt; //works fine if I declare the header in main.cpp

#include &lt;iostream&gt;

#ifdef USE_CUDA
#include "gpu.hpp"
#endif

int main()
{
    std::cout &lt;&lt; "Hello, world!" &lt;&lt; std::endl;

    printCudaVersion();

    return 0;
}
</code></pre>
<p>gpu.hpp</p>
<pre><code class="lang-auto">//#include &lt;torch/torch.h&gt; //this will cause error message "Cannot open include file: 'torch/torch.h'"

void printCudaVersion();
</code></pre>
<p>gpu.cu</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include "gpu.hpp"

void printCudaVersion()
{
    std::cout &lt;&lt; "CUDA Compiled version: " &lt;&lt; __CUDACC_VER__ &lt;&lt; std::endl;

    int runtime_ver;
    cudaRuntimeGetVersion(&amp;runtime_ver);
    std::cout &lt;&lt; "CUDA Runtime version: " &lt;&lt; runtime_ver &lt;&lt; std::endl;

    int driver_ver;
    cudaDriverGetVersion(&amp;driver_ver);
    std::cout &lt;&lt; "CUDA Driver version: " &lt;&lt; driver_ver &lt;&lt; std::endl;
}
</code></pre>
Crypto PRice Prediction

Muhammad_Ali: <p>Hi ,hope all of you are fine.<br>
I am trying out to come up with a viable crypto-price-prediction model, but as it is quite unstable, I am figuring out that maybe the technical indicators like RSI,4 EMA, hash ribbons, etc, which professional traders use will do the trick.<br>
Right now, most of the tutorials and blogs about using LSTMs, GRUs, etc, are just fooling and I couldn’t find any workable solution.<br>
So, if anyone can guide me, will be very grateful.</p>
<p>Thanks.</p>

ako: <p>can you elaborate a bit? ‘coming up with a model’ can mean a million different things.</p>

Muhammad_Ali: <p>Thanks <a class="mention" href="/u/ako">@ako</a>, thanks for your time.<br>
Sorry for my poor words.I mean ML prediction model, so that it can forecast the prices for about 7-20 days in future.</p>
<p>Thanks</p>

ako: <p>I see. All those technical indicators you mentioned are derived from previous prices - so I’d simply give previous price history to sci-kit <a href="https://scikit-learn.org/stable/supervised_learning.html#supervised-learning" rel="noopener nofollow ugc">https://scikit-learn.org/stable/supervised_learning.html#supervised-learning</a> - which is “ML” .</p>
<p>However - before you hinge a lot of hope on it - think whether your project is possible. Are future prices simple a function of past prices? Are the patterns random or repeatable?<br>
Pytorch and to a lesser extent ML (sci-kit) are powerful tools for finding patterns - question is your particular problem has any patters or is it like a roulette wheel - a random number generator.</p>

Muhammad_Ali: <p>Yes definetly, crypto market is totally based on trust, and definetly the effects are seen in market in some time.<br>
So, I think if we figure out the right way of correct inputs to the model, I hope that model can make sense out of it.</p>
Regarding parameter scaling of a model and gradient on the updated parameters

kiranpurohit: <p>I have made a one layer network. I have model’s linear.weight and linear.bias in a dictionary, and I have passed the dictionary to the new nn.module class I have made. I have to find the gradient of loss wrt theta which I have multiplied with the model’s parameter. But the error is coming like “One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior”. Please help me with this so that the computation graph is maintained.</p>
<p>Thanks in advance</p>

kiranpurohit: <p>class Net1(nn.Module):</p>
<pre><code>def __init__(self):
    super(Net1, self).__init__()
    self.linear = nn.Linear(3072, 10)

def forward(self, x):
    out = x.view(x.size(0), -1)
    out = self.linear(out)
    return out
</code></pre>
<p>class lin(nn.Module):</p>
<pre><code>def __init__(self, wt, bs, inp, output):
    super(lin, self).__init__()
    self.linear  = nn.Linear(inp, output)
    self.linear.weight = nn.parameter.Parameter(wt)
    self.linear.bias = nn.parameter.Parameter(bs)

def forward(self, input):
    out = input.view(input.size(0), -1)
    out = self.linear(out)
    return out
</code></pre>
<p>class Net_test(nn.Module):</p>
<pre><code>def __init__(self,paramsel):
    super(Net_test, self).__init__()
    self.lnr = lin(paramsel['linear.weight'], paramsel['linear.bias'], 3072, 10)

def forward(self, x):
    print("Inside net test")
    out = x.view(x.size(0), -1)
    out = self.lnr(out) 
    print(out.requires_grad)
    return out
</code></pre>
<p>classf = Net1()<br>
classf = classf.cuda()<br>
paramNettest = {}<br>
theta = torch.randn(1, requires_grad=True).cuda()</p>
<p>for i, data in enumerate(train_loader, 0):</p>
<pre><code>inputs, labels = data
inputs, labels = inputs.cuda(), Variable(labels.cuda())

for name, param in classf.named_parameters(): 
    paramNettest[name] = param * theta

classft = Net_test(paramNettest)    
classft = classft.cuda()
op2 = classft(inputs)
loss =F.cross_entropy(op2,labels,reduction='mean')
</code></pre>
<p>print(grad(loss,theta))</p>

kiranpurohit: <p><a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> any suggestions?</p>
Detected that PyTorch and torch_scatter were compiled with

FengZhiheng-coder: <p>When I use torch_scatter, I get error as followed here.</p>
<p><code>RuntimeError: Detected that PyTorch and torch_scatter were compiled with different CUDA versions. PyTocatter has CUDA version 10.1. Please reinstall the torch_scatter that matches your PyTorch install.</code></p>
<p>My torch vision is torch==1.8.0+cu111 and torch-scatter’s is torch-scatter==2.0.6 , how can I get right torch-scatter</p>

ptrblck: <p>You would have to specify the same CUDA version (in your case CUDA11.1) as described in the <a href="https://github.com/rusty1s/pytorch_scatter#pytorch-190">README of <code>pytorch_scatter</code></a>.</p>

FengZhiheng-coder: <p>Thanks a lot, sir<br>
I have tried</p>
<pre><code class="lang-auto">pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cu111.html
</code></pre>
<p>Now torch-scatter==2.0.7, but there is still the same error.</p>

ptrblck: <p>Since it’s the same error I assume <code>pytorch-geometric</code> still claims to be using CUDA10.1?<br>
In that case, your install command wasn’t successful and you would have to uninstall the previous package and reinstall the new one.</p>

FengZhiheng-coder: <p>It worked! <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=9" title=":laughing:" class="emoji" alt=":laughing:"><br>
Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Applying k-means clustering to 4D tensor [1,2048,25,19]

i_m_a_q: <p>I have image features’ tensor with 4 dimensions. I want to apply <strong>k-means clustering</strong> to <strong>3rd and 4th dimension</strong> only leaving the first 2 dimensions as is. In short, I want to reduce the the size of 3rd and 4th dimension to <strong>36</strong>. At the end I need tensor of size <strong>[2048,36]</strong>. If someone could guide me how do I do that ?</p>