akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How do I find the right dimension for sensor data for neural network inputs?

wroko: <p>Hey, I’m trying to solve a classification problem with a neural network based on time series sensor data. Basically I have 2 types of sensor data: <em>a</em> and <em>b</em>. Each sample consists of 8 time points, basically:<br>
SAMPLE:<br>
------Sensor 1, Sensor 2<br>
time1: 0.10 , 20<br>
time1: 0.20 , 20<br>
time1: 0.30 , 20<br>
time1: 0.40 , 20<br>
time1: 0.50 , 20<br>
time1: 0.60 , 20<br>
time1: 0.70 , 20<br>
time1: 0.80 , 20</p>
<p>To what shape do I have to form the data for a linear layer (8x2,16x1)? It’s basically a 2d Input, but I’m unsure because its a time series</p>
<p>I tried shaping it to [[0.10 , 20],[0.20 , 20],[0.30 , 20],[0.40 , 20]…] but my loss calculation wants a 2D label for that. I also tried [ 0.1000, 0.2000, 0.3000, 0.4000, 0.5000, 0.6000, 0.7000, 0.8000, 20.0000, 20.0000, 20.0000, 20.0000, 20.0000, 20.0000, 20.0000, 20.0000], but the results are bad.</p>
Building torch from source for old GPU

KBOPYM: <p>I need a 1.13 version of torch with compute capability 3.5 and cuda 10.2, since there is no prebuilt one, I’m doing it myself.<br>
Have been using this guide (changing versions here and there) : <a href="https://datagraphi.com/blog/post/2021/9/13/building-pytorch-from-source-on-windows-to-work-with-an-old-gpu" class="inline-onebox" rel="noopener nofollow ugc">Building PyTorch from source on Windows to work with an old GPU</a><br>
I’m at last steps<br>
performing command</p>
<pre><code class="lang-auto">python setup.py install --cmake
</code></pre>
<p>gives me</p>
<pre><code class="lang-auto">Could not find any of CMakeLists.txt, Makefile, setup.py, LICENSE, LICENSE.md, LICENSE.txt in D:\torch\pytorch\pytorch\third_party/ios-cmake                      
Did you run 'git submodule update --init --recursive --jobs 0'?  
</code></pre>
<p>I did the submodule update already, it does nothing.</p>

marksaroufim: <p>Those instructions seem out of date, specifically I’m not sure what the <code>ccmake</code> flag is supposed to do this in this case</p>
<p>Would suggest reading the official instructions <a href="https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md" class="inline-onebox">pytorch/CONTRIBUTING.md at main · pytorch/pytorch · GitHub</a></p>

KBOPYM: <p>Had to start all over, but I’m getting new errors now.<br>
Thank you.</p>
<p>Current problem - having VS2022, when requirements are from 2015 to 2019.</p>
The model output result is incorrect after exporting the code that called the torch_geometric package as onnx model

boy_little: <pre><code class="lang-auto">import torch
from torch import Tensor
import torch.nn as nn
import torch.nn.functional as F
from torch_sparse import SparseTensor, matmul
from torch_geometric.nn.conv import MessagePassing

class GINConv(MessagePassing):

    def __init__(self, nn:Callable, eps: float=0., train_eps: bool=False,
                 activation="softplus", **kwargs):
        super(GINConv, self).__init__(aggr='add', **kwargs)
        self.nn = nn    # MLP
        self.initial_eps = eps
       ......

    def forward(self, x: Union[Tensor, OptPairTensor], edge_index: Adj,
                edge_attr: OptTensor=None, size: Size=None) -&gt; Tensor:
       
        out = self.propagate(edge_index, x=x, edge_attr=edge_attr, size=size)

       ......  
    
    def message(self, x_j: Tensor, edge_attr: Tensor) -&gt; Tensor:

       .......
    
    def __repr__(self):
        ......

</code></pre>
<p>As shown in the above code, after exporting the module as onnx, the inference result of the model is inconsistent with the original model. I don’t know if it’s because some operators in the ‘pyg’ package don’t support exporting as onnx. The specific code line that caused the problem is: out=self. propagate (edgeIndex, x=x, edgeAttr=edgeAttr, size=size)</p>
<p>According to my investigation, the code “out=self. propagate (edgeindex, x=x, edgeattr=edgeattr, size=size)” calls a function called scatter-add_. Does this function not support exporting to onnx?</p>
<p>The following is the onnx diagram of this code：<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/5659750ff3fb8c7e9283bddaf9198cce22be6212.png" data-download-href="https://discuss.pytorch.org/uploads/default/5659750ff3fb8c7e9283bddaf9198cce22be6212" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/5659750ff3fb8c7e9283bddaf9198cce22be6212_2_635x500.png" alt="image" data-base62-sha1="cjSISeokSVaS4ZohLQm0uUJIcVA" width="635" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/5659750ff3fb8c7e9283bddaf9198cce22be6212_2_635x500.png, https://discuss.pytorch.org/uploads/default/original/3X/5/6/5659750ff3fb8c7e9283bddaf9198cce22be6212.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/6/5659750ff3fb8c7e9283bddaf9198cce22be6212.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">834×656 29.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Global_mean_pool gives different results every time with same input

Jing_Bi: <p>I use global_mean_pool in torch geometric “x = global_mean_pool(x, batch)” to average node features into graph level features, however, I found that this operator gives me slightly different results every time.<br>
I print the input x and calculated the mean feature in excel, compare to torch output and then I also checked with "x=global_mean_pool(x, None).<br>
Excel calculation:<br>
|0.2111766|0.0019979|0.2978659|0.1123972|0.1230657|<br>
x=global_mean_pool(x,data.batch)<br>
|0.2111767|0.0019979|0.2978655|0.1123973|0.1230659|<br>
x=global_mean_pool(x,None)<br>
|0.2111766|0.0019979|0.2978659|0.1123972|0.1230657|</p>
<p>So most of the output from x=global_mean_pool(x,data.batch) are wrong/off, and everytime I run the same it gives a different solution. Other options are consistent. Why is this operation random? BTW I have fixed all random seed, numpy random seed, torch and torch cuda random seeds.</p>

ptrblck: <p>The outputs might be expected if non-deterministic algorithms are used. If so, none of the outputs are wrong as they should show the same error to a wider dtype.</p>

Jing_Bi: <p>Thanks a lot for your reply! I can see a few operators listed on this page that are non-deterministic: <a href="https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms" class="inline-onebox" rel="noopener nofollow ugc">torch.use_deterministic_algorithms — PyTorch 2.2 documentation</a></p>
<p>However, I can’t find any documentation that says global_mean_pool in torch geometric is deterministic or not, or does it have a deterministic implementation. Could you please help point me to a documentation page?</p>
<p>Why defining None instead of batch gives a consistent result?</p>
<p>I also hit an error if the optional “batch” parameter is not defined. The documentation says definition of ”batch” is optional:</p>
<p><a href="https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.global_mean_pool.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.pool.global_mean_pool.html</a></p>

ptrblck: <p>I’m not familiar enough with PyG so you might need to ask the authors how the algorithms are implemented and if a deterministic version exists.</p>
Unsupervised clustering of univariate time series

JD59: <p>I’m new to pytorch.</p>
<p>I have a 23-year time series of remotely sensed vegetation index data (as a data file, not images).  It is a univariate dataset - 1 variable, 23 time steps - in n observations (rows) and 23 columns.  My hope was to use an RNN as an autoencoder and use the bottleneck as input into a clustering routine.  As I understand it though, my data do not fit the expected format for an RNN since I only have one variable.</p>
<p>I looked at TorchCoder (<a href="https://github.com/hellojinwoo/TorchCoder" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hellojinwoo/TorchCoder: PyTorch based autoencoder for sequential data</a>) and Sequitur (<a href="https://github.com/shobrook/sequitur" class="inline-onebox" rel="noopener nofollow ugc">GitHub - shobrook/sequitur: Library of autoencoders for sequential data</a>). Does pytorch itself have a way to format my data so I can treat time as 23 variables and use GRU/LSTM in an autocoder format?</p>
<p>Thanks!<br>
JD59</p>

JuanFMontesinos: <p>Hi,<br>
you can use pytorch standard RNN module to acomplish that.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.RNN.html</a><br>
You have an easy example at the very end.</p>
Most effiecient way to move padding tokens to the right side of a tensor?

ShawnGuo: <p>Hi, I was wondering the most efficient way to move the padding tokens in a given tensor to the rightmost? A more concrete example is illustrated below.</p>
<p>Suppose the padded tensor is the following one.</p>
<pre data-code-wrap="python"><code class="lang-python">x = [
    [PAD, X, X, X, PAD],
    [PAD, PAD, X, X, X],
    [X, X, X, X, X]
]
</code></pre>
<p>I’d like the output to be:</p>
<pre data-code-wrap="python"><code class="lang-python">x = [
    [X, X, X, PAD, PAD],
    [X, X, X, PAD, PAD],
    [X, X, X, X, X]
]
</code></pre>
<p>Thanks in advance for your help!</p>
How to share code without making visible

robinho: <p>How to share code for others to use without making it visible to them (encapsulate?)? That is, those who use the code can’t see how the code is written at all.</p>

ptrblck: <p>Sharing code without seeing code sounds contradictory. You could build an application or web service if you want users to use your model without understanding what’s under the hood.</p>

robinho: <p>Anyone knows how to encapsulate into a .pyd, which seems to do the job?</p>
Torch/lib/libtorch_global_deps.so: cannot open shared object file: No such file or directory

Max_Unhold: <p>Hey, I was trying to install pytorch from source using the comands</p>
<pre><code class="lang-auto">export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py develop
</code></pre>
<p>after getting the source code by</p>
<pre data-code-wrap="python"><code class="lang-python">git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
</code></pre>
<p>When I run a <code>test.py</code> skript in this github folder with the <code>torch</code> folder, I get the<br>
error message</p>
<pre data-code-wrap="python"><code class="lang-python">Traceback (most recent call last):
  File ".../pytorch/test.py", line 1, in &lt;module&gt;
    import torch
  File ".../pytorch/torch/__init__.py", line 236, in &lt;module&gt;
    _load_global_deps()
  File ".../pytorch/torch/__init__.py", line 195, in _load_global_deps
    raise err
  File ".../pytorch/torch/__init__.py", line 176, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File ".../anaconda3/envs/testenv/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: .../pytorch/torch/lib/libtorch_global_deps.so: cannot open shared object file: No such file or directory
</code></pre>
<p>Should torch not be callable directly from this folder?</p>

ptrblck: <p>Change the directory to e.g. the test folder and check if this would work.</p>
Mps does not work jupyter notebook

betul: <p>hello,i am using macbook pro m1 and i want to use pytorch mps on jupiter notebook.although i have written jupiter notebook and pytorch properly,it works on cpu,it doesn’t switch to gpu.i have installed all necessary versions.mps is installed as shown in screen shots but it still works on cpu.why can this happen?<br>
Also I use Yolov8</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a26ce88cb2aafe77310b965debbce1b72f28163.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8a26ce88cb2aafe77310b965debbce1b72f28163" title="Ekran Resmi 2024-03-01 16.18.49"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a26ce88cb2aafe77310b965debbce1b72f28163_2_478x500.jpeg" alt="Ekran Resmi 2024-03-01 16.18.49" data-base62-sha1="jI926TUvhlfoWdzgAFluO7YehBV" width="478" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a26ce88cb2aafe77310b965debbce1b72f28163_2_478x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a26ce88cb2aafe77310b965debbce1b72f28163_2_717x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a26ce88cb2aafe77310b965debbce1b72f28163_2_956x1000.jpeg 2x" data-dominant-color="F3F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Ekran Resmi 2024-03-01 16.18.49</span><span class="informations">1550×1620 124 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see the gpu says 0 and it doesn’t work.<br>
I also tried to change the kernel but it still doesn’t work.please help.<br>
ın jupyter notebook python version is= Python 3.10.13</p>
Fully connected network with intermediate inputs

muttistefano: <p>Hi all,<br>
i want to create a fully connected network that also takes inputs from intermediate layers; these inputs are optional, and where they are lacking, I fill them with zeros.</p>
<p>My class looks like this now:</p>
<pre><code>class FCC(nn.Module):

  def __init__(self, hidden_size):
      super(FCC, self).__init__()
      self.drp  = nn.Dropout(0.1)
      self.fc1  = nn.Linear(in_features=hidden_size,out_features=60)
      self.fc2  = nn.Linear(in_features=60,out_features=50)
      self.fc3  = nn.Linear(in_features=52,out_features=30)
      self.fc4  = nn.Linear(in_features=34,out_features=10)
      self.fc5  = nn.Linear(in_features=10,out_features=2)
      self.fc1.weight.data.normal_(mean=0.0, std=0.1)
      self.fc2.weight.data.normal_(mean=0.0, std=0.1)
      self.fc3.weight.data.normal_(mean=0.0, std=0.1)
      self.fc4.weight.data.normal_(mean=0.0, std=0.1)
      self.fc5.weight.data.normal_(mean=0.0, std=0.1)

  def forward(self,x,cl_input,vis_input):
      # print(x.shape)
      out    = self.fc1(x)
      out    = F.relu(out,inplace=True)
      out    = self.fc2(out)
      out    = F.relu(out,inplace=True)
      merge0  = torch.cat((out,vis_input),1)
      out    = self.fc3(merge0)
      out    = F.relu(out,inplace=True)
      merge  = torch.cat((out,cl_input),1)
      out   = self.fc4(merge)
      out    = F.relu(out,inplace=True)
      out    = self.fc5(out)
      return out
</code></pre>
<p>I noticed that compared with the model without mid-inputs, my training have a mush worse performance, and the losses way more discontinous.<br>
Am I doing something wrong with the gradients?<br>
Thanks</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/muttistefano">@muttistefano</a>,</p>
<aside class="quote no-group" data-username="muttistefano" data-post="1" data-topic="198004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muttistefano/48/50462_2.png" class="avatar"> muttistefano:</div>
<blockquote>
<pre><code class="lang-auto">      self.fc1.weight.data.normal_(mean=0.0, std=0.1)
</code></pre>
</blockquote>
</aside>
<p>If you want to change the initial weights, don’t use the <code>.data</code> attribute as that won’t change the metadata of the tensor. You should do something like this instead,</p>
<pre><code class="lang-auto">self.fc1=nn.Linear(6,4)
with torch.no_grad():
  self.fc1.weight.copy_(torch.randn_like(self.fc1.weight))
</code></pre>
<p>Also, don’t use <code>inplace=True</code> updates as it will mess with gradient calculations.</p>
<aside class="quote no-group" data-username="muttistefano" data-post="1" data-topic="198004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muttistefano/48/50462_2.png" class="avatar"> muttistefano:</div>
<blockquote>
<pre><code class="lang-auto">      out    = F.relu(out,inplace=True)
</code></pre>
</blockquote>
</aside>
How to implement 3-order hyper edge conv?

Shihab_Shahriar: <p>I need to do something like:</p>
<pre><code class="lang-auto">def message(self, x_i, x_j, x_k):
    input = torch.cat([x_i-x_j, x_i-x_k])
</code></pre>
<p>HypergraphConv could have been an option, but I can’t figure out a simple way to adapt that to this use case. Any help will be highly appreciated. Thanks.</p>
Segmentation fault: Call native cpu operator in custom cpu operator

daocaoren360: <p>Hi,<br>
I register a cpu custom operator sum_op_forward for aten::sum.dim_IntList. If input.dim() !=4, call back to the native cpu operator at::sum. If I call at::sum directly in sum_op_forward, it will actually call sum_op_forward back. Finally, the program exits abnormally due to infinite recursive calls. So how can I call the native cpu at::sum ?</p>
<p>Could anyone give me some help? thanks.</p>
<p>Pytorch version: 2.2</p>
<p>the code is as follows:</p>
<pre><code class="lang-auto">#include &lt;torch/all.h&gt;
#include &lt;ATen/ATen.h&gt;
#include &lt;ATen/NativeFunctions.h&gt;
#include &lt;torch/csrc/autograd/custom_function.h&gt;
#include "torch/library.h"

namespace torch_ex {
namespace cpu {

class ExSumOp : public torch::autograd::Function&lt;ExSumOp&gt; {
 public:
  static at::Tensor forward(
    torch::autograd::AutogradContext* ctx,
    const at::Tensor&amp; input,
    c10::OptionalIntArrayRef dim,
    bool keepdim,
    c10::optional&lt;c10::ScalarType&gt; dtype){

      // some custom forward implementations
      // ....
      return out;
    }

  static torch::autograd::variable_list backward(
      torch::autograd::AutogradContext* ctx,
      torch::autograd::variable_list grad_outputs){

      // some custom backward implementation 
      // ......
      return {grad_input, at::Tensor(), at::Tensor(), at::Tensor()};    
    }
};

at::Tensor sum_op(
    const at::Tensor&amp; input,
    c10::OptionalIntArrayRef dim,
    bool keepdim,
    c10::optional&lt;c10::ScalarType&gt; dtype){

    return ExSumOp::apply(input, dim, keepdim, dtype);
}

at::Tensor sum_op_forward(
    const at::Tensor&amp; input,
    c10::OptionalIntArrayRef dim,
    bool keepdim,
    c10::optional&lt;c10::ScalarType&gt; dtype){

  auto ndim = input.dim();
  if (ndim == 4){
    // call custom operator
    return sum_op(input, dim, keepdim, dtype);
  }else{

    //// actually, it will call custom sum_op_forward. Recursive call to itself causes program to exit 
    //// abnormally, so how to call native at::sum ????
    // return at::sum(input, dim, keepdim, dtype);
 
   }
}

TORCH_LIBRARY_FRAGMENT(aten, m) {
  m.impl(
      TORCH_SELECTIVE_NAME("aten::sum.dim_IntList"),
      c10::DispatchKey::CPU,
      torch_ex::cpu::sum_op_forward);

  m.impl(
      TORCH_SELECTIVE_NAME("aten::sum.dim_IntList"),
      c10::DispatchKey::AutogradCPU,
      torch_ex::cpu::sum_op_forward);
}

} // namespace cpu
} // namespace torch_ex
</code></pre>
How to use L-BFGS in the C++ LibTorch?

rolyluoli: <p>Hi!</p>
<p>How to use L-BFGS in the C++ LibTorch?<br>
Particularly, how to implement “closure function” in C++?</p>
<p>Would you please offer an example using L-BFGS?</p>
<p>Thank you!</p>

Fitanium: <pre data-code-wrap="CPP"><code class="lang-CPP">torch::optim::LBFGS optimizer = ... // init
auto closure = [&amp;](){
    optimizer.zero_grad();
    // must call model forward inside, and then calculate loss
    torch::Tensor out = model-&gt;forward(...);
    torch::Tensor loss = loss_func(out,...);
    loss.backward();
    return loss;
};
optimizer.step(closure);
</code></pre>
Nvlink error : Undefined reference to 'XXX' when build pytorch from source

shadowshadow: <p>I’ m trying to install pytorch from source folloing the official guidance，when I run <code>python setup.py develop</code>. It reports many nvlink errors like below:</p>
<pre><code class="lang-auto">nvlink error   : Undefined reference to '_Z28ncclAllReduceCollNet_min_f64P14CollectiveArgs' in 'dir_to_pytorch/build/nccl/obj/collectives/device/functions.o'
</code></pre>
<p>the error output:</p>
<pre><code class="lang-auto">FAILED: nccl_external-prefix/src/nccl_external-stamp/nccl_external-build nccl/lib/libnccl_static.a 
cd /home/wtx/workspace/cpp_project/pytorch/third_party/nccl/nccl &amp;&amp; env make CXX=/usr/bin/c++ CUDA_HOME=/home/wtx/.local/cuda-11.8 NVCC=/home/wtx/.local/cuda-11.8/bin/nvcc NVCC_GENCODE=-gencode=arch=compute_86,code=sm_86 BUILDDIR=/home/wtx/workspace/cpp_project/pytorch/build/nccl VERBOSE=0 -j &amp;&amp; /usr/bin/cmake -E touch /home/wtx/workspace/cpp_project/pytorch/build/nccl_external-prefix/src/nccl_external-stamp/nccl_external-build
make -C src build BUILDDIR=/home/wtx/workspace/cpp_project/pytorch/build/nccl
make[1]: Entering directory '/home/wtx/workspace/cpp_project/pytorch/third_party/nccl/nccl/src'
make[2]: Entering directory '/home/wtx/workspace/cpp_project/pytorch/third_party/nccl/nccl/src/collectives/device'

</code></pre>
<p>Pytorch version: 1.9.0<br>
Python version: 3.10<br>
Can somebody tell me how to solve the problem?</p>

ptrblck: <p>Which NCCL version are you using when building this older PyTorch version from source?</p>

shadowshadow: <p>nccl version is 2.7.8</p>

ptrblck: <p>I don’t know why the error is raised, but seems to be related to <a href="https://github.com/NVIDIA/nccl/issues/320#issuecomment-612321797"><code>CollNet</code></a>. You could try to set <code>NCCL_COLLNET_ENABLE=0</code> and check if it would work or alternatively you might need to install the network plugins.</p>
Active Learning vs Hard negative mining

vivekstorm: <p>Hi Guys,<br>
I would like to understand the difference between active learning and hard negative mining.<br>
Based on my understanding:</p>
<ol>
<li>Active learning is a way to label data which the model is uncertain about</li>
<li>Hard negative mining is a way to choose the data in which model performance is bad.<br>
In both technique we are choosing the data where the model finds it difficult right?</li>
</ol>
<p><strong>Is there any other difference between both technique? Also if my understanding is correct, Active learning requires a base model which is trained on a dataset?</strong></p>
<p>Thanks in advance.</p>
Error with custom batch norm

luckyseadog: <p>Hi everyone,</p>
<p>I am trying to implement a custom version of  BatchNorm with backward pass, but, when I try to use it in training, I get an error “Trying to backward through the graph a second time”</p>
<p>Could anyone help me with this?</p>
<p>My Function:</p>
<pre><code class="lang-auto">from torch.autograd import Function

class _batch_norm_function(Function):

    @staticmethod
    def forward(x, mean, variance):
        EPS = 1e-12
        return (x - mean) / torch.sqrt(variance + EPS)
      
    @staticmethod
    def setup_context(ctx, inputs, output):
        x, mean, variance = inputs
        ctx.save_for_backward(x, mean, variance)

    @staticmethod
    def backward(ctx, grad_output):
      EPS = 1e-12
      inp, mean, var = ctx.saved_tensors
      grad_input = None

      if ctx.needs_input_grad[0]:
        N, D = inp.shape
        
        std = torch.sqrt(var + EPS)
      
        xmu = inp - mean
        dxmu1 =  1/std * grad_output
        dxmu2 = 2 * xmu * 1/N * 1/(2 * torch.sqrt(var + EPS)) * torch.sum(-(inp - mean)/std**2 * grad_output, axis=0)
        dx1 = dxmu1 + dxmu2
        dmu = -1 * torch.sum(dxmu1 + dxmu2, axis=0)
        dx2 = 1 /N * dmu
        grad_input = dx1 + dx2

      return grad_input, None, None
  
def bn_function(x, mean, variance):
    return _batch_norm_function.apply(x, mean, variance)
</code></pre>
<p>Module:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.distributed as dist

class MySyncBatchNorm1D(nn.Module):
  def __init__(self, alpha=0.0):
    super().__init__()
    self.training = True
    self.alpha = alpha
    self.moving_mean = None
    self.moving_variance = None
  
  def sync(self):
    if self.training and torch.distributed.is_available() and torch.distributed.is_initialized():
      size = float(dist.get_world_size())
      dist.all_reduce(self.moving_mean, op=dist.ReduceOp.SUM)
      dist.all_reduce(self.moving_variance, op=dist.ReduceOp.SUM)
      self.moving_variance /= size
      self.moving_mean /= size
    else:
      raise RuntimeError("Sync is only supported during training with dist training turned on")
  
  def forward(self, x):
    n_samples, dim = x.shape
        
    mean = torch.mean(x, axis=0)
    varience = torch.var(x, axis=0)


    if self.moving_mean is None or self.moving_variance is None:
      self.moving_mean = mean
      self.moving_variance = varience

    elif self.training == True:
      self.moving_mean = self.alpha * self.moving_mean + (1 - self.alpha) * mean
      self.moving_variance = self.alpha * self.moving_variance + (1 - self.alpha) * varience

    
    return bn_function(x, self.moving_mean, self.moving_variance)

  def train(self, mode=True):
    self.training = mode
    return self

  def eval(self):
      return self.train(False)
</code></pre>
What port/s does DDP use?

TheVictor_777: <p>Hello, I have repeatedly attempted to train a model across two nodes using DDP and torchrun. For some unknown reason, my client node times out after 15 minutes of no connection. Thus far, I have been unable to resolve the issue. The only problem that I can think of is that perhaps DDP requires ports behind the scenes (besides 29500) that I have not opened. Is this true?<br>
<code>Also note that while I should and do specify the port with torchrun, the client node apparently attempts to use </code><strong>29500</strong> <code>regardless.</code></p>
Torch_cuda is never loaded without explicitly doing so

kwijibohan: <p><code>CMakeLists.txt</code>:</p>
<pre data-code-wrap="cmake"><code class="lang-plaintext">cmake_minimum_required(VERSION 3.18 FATAL_ERROR)
project(torchtest)
enable_language(CUDA)

get_filename_component(CMAKE_PREFIX_PATH ../libtorch ABSOLUTE)
get_filename_component(CUDNN_ROOT ../cudnn ABSOLUTE)
find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_executable(test1 test.cpp)
target_link_libraries(test1 PRIVATE "${TORCH_LIBRARIES}")

if (MSVC)
  file(GLOB TORCH_DLLS "${TORCH_INSTALL_PREFIX}/lib/*.dll")
  add_custom_command(TARGET test1
                     POST_BUILD
                     COMMAND ${CMAKE_COMMAND} -E copy_if_different
                     ${TORCH_DLLS}
                     $&lt;TARGET_FILE_DIR:test1&gt;)
endif (MSVC)
</code></pre>
<p><code>test1.cpp</code>:</p>
<pre><code class="lang-cpp">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;

int main() {
    std::cout &lt;&lt; torch::cuda::is_available() &lt;&lt; std::endl;
    std::cout &lt;&lt; torch::cuda::cudnn_is_available() &lt;&lt; std::endl;
    return 0;
}
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">0
0
</code></pre>
<p>If I edit <code>test1.cpp</code> to</p>
<pre><code class="lang-cpp">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;Windows.h&gt;

int main() {
    LoadLibraryA("torch_cuda.dll");
    std::cout &lt;&lt; torch::cuda::is_available() &lt;&lt; std::endl;
    std::cout &lt;&lt; torch::cuda::cudnn_is_available() &lt;&lt; std::endl;
    return 0;
}
</code></pre>
<p>It outputs</p>
<pre><code class="lang-auto">1
1
</code></pre>
<p>as intented. CUDA: 11.8, Libtorch: cu118 latest, OS: Windows 10, Generator: <code>Ninja Multi-Config</code>, Compiler: <code>icx</code><br>
How should I fix this?</p>

ptrblck: <p>I cannot reproduce the issue in Linux using a nightly release and get:</p>
<pre><code class="lang-auto">1
1
</code></pre>
<p>for he first code snippet already, so I guess the issue might be Windows-specific.</p>

kwijibohan: <p>Will this get fixed in the future? Or should continue to I use LoadLibraryA?</p>

ptrblck: <p>I don’t know if this issue is still there as I wasn’t able to reproduce it on Linux so we might need to wait for someone else using Windows or you could use the latest nightly release or source build to check if you can still see the issue. If so, you could create an issue on GitHub so that code owners could track and fix it.</p>

Gerald_Lodron: <p>i can say that this is still the case. I build libtorch on Windows with VS2022x64 and cuda 12.1 with cudnn 8.9.0</p>
<p>also, the pure cmake build workflow crashed (dlls were build without error). Only the ninja workflow worked</p>
<p>i disabled BUILD_PYTHON</p>
An efficient way to slice tensors with array indices?

JTiC: <p>How can I get an unaligned tensor from a large one efficiently?<br>
The pseudo-code is shown as follow (it can not work in pytorch):</p>
<pre><code class="lang-auto">x = torch.rand((3,1,6,6), requires_grad=True) # [batch_size, channel, w, h]
left_index = torch.randint(0,4,(3,))
right_index = left_index + 2
bottom_index = torch.randint(0,4,(3,))
top_index = bottom_index + 2
new_x = x[:,:,left_index:right_index, bottom_index:top_index]
</code></pre>
<p>that means, for the i-th image, we want to get a small tensor using slice<br>
<code>(i, :, left_index[i]: right_index[i], bottom_index[i]: top_index[i])</code><br>
Of course, if I write a for-loop code enumerating through all the images and stacking the results, I can obtain the wanted small tensor.<br>
However, I think “for-loop+stacking” algorithm takes time, and I want a more efficient method. Please give me some advice. Thank you!</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-sum-up-all-values-according-to-rectangular-rois-of-a-feature-map-quickly/75432/2">This post</a> from <a class="mention" href="/u/kfrank">@KFrank</a> might be useful. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

JTiC: <p>Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a>. I read this post. In my view, the solution (inclusion-exclusion version) provided in this post works because the sum of the mask_region is wanted. However, in my question, I want to obtain the raw values in the mask region. It seems that the inclusion-exclusion version is not very suitable in this case. I am not very sure… Can you give me some tips in using inclusion-exclusion method?</p>

ptrblck: <p>Ah OK, sorry for the misunderstanding.<br>
I’m not sure, if it would be possible to get the desired regions without a reduction operation, and I think you would have to use a loop.<br>
Since the slicing might return different output shapes, you wouldn’t be able to create one output tensor (nested tensors would be needed, which is still a work in progress).</p>

JTiC: <p>Thank you for your help~</p>

AKJ: <p>Hi is there any update to this?</p>
Is it efficient to pass model into a custom dataset to run model inference during training for sampling strategy

jupyter: <p>I’m trying to design a training flow for sampling samples during training.</p>
<p>My data look like this:</p>
<pre><code>defaultdict(list,
        {'C1629836-28004480': [0, 5, 6, 12, 17, 19, 28],
         'C0021846-28004480': [1, 7, 15],
         'C0162832-28004480': [2, 9],
         'C0025929-28004480': [3, 10, 30],
         'C1515655-28004480': [4],
         ...
        }
</code></pre>
<p>where key is label and value is list of data index</p>
<p>I custom dataset class in which my <code>__getitem__(self, idx)</code> function need to calculate distance between an anchor (which is chosen randomly) and other data points. It looks like this:</p>
<pre><code class="lang-auto">def __getitem__(self, idx):
    item_label = self.labels[idx] # C1629836-28004480
    item_data = self.data[item_label] # [0, 5, 6, 12, 17, 19, 28]

    anchor_index = random.sample(item_data,1)
    mentions_indices = [idx for idx in item_data if idx != anchor_index]
    
    with torch.no_grad():
        self.model.eval()
        anchor_input = ...
        anchor_embedding = self.model.mention_encoder(anchor_input)

        for idx in mention_indices: 
        ...
</code></pre>
<p>Another way to prevent from passing the model into custom dataset is to  run inference inside the <code>training_step</code> function during training.</p>
<p>But I read somewhere that, using dataset and dataloader to prepare data to feed into model might save the training time, as they have parallel mechanism or something like that.</p>
<p>But in fact, I need to compute these kind of distance base on the latest state of weight of my model during training, is this parallel mechanism ensure that? Though in python variable is reference variable instead of value variable.</p>
<p>So which way is more professional and correct?</p>
Assistance Required with ABSA-PyTorch Repository Execution

lukmanaj: <p>I hope this message finds you all  well. I am writing to request your assistance with running the ABSA-PyTorch <a href="https://github.com/songyouwei/ABSA-PyTorch" rel="noopener nofollow ugc">repository</a>.</p>
<p>While attempting to execute the repository, I encountered an issue that necessitated a modification in the lcf_bert.py script. Specifically, I adjusted an import statement, which subsequently led to an error indicating that a string was being passed to nn.Dropout instead of the expected tensor. This issue may be stemming from the fact that I was unable to install the exact versions of the required packages as specified in the repository.</p>
<p>I would greatly appreciate your guidance on how to resolve this issue, particularly whether it is related to the specific package versions or if there might be another underlying problem. Your expertise and advice would be invaluable in helping me navigate this challenge. <a href="https://colab.research.google.com/drive/1PH7mJzuWwWB30N6VndR_5kjyUKAcAFAM?usp=sharing" rel="noopener nofollow ugc">Here</a> is my attempt. I tried it on google colab as I do not have a GPU on my laptop.</p>
<p>Thank you very much for your time and assistance.</p>

ptrblck: <p>The issue isn’t caused by a mismatch between libraries, but by the actual usage in the dropout layer.<br>
I’m not familiar with the repository, but would assume tensors are expected as inputs while you might be passing strings?</p>

lukmanaj: <p>Thanks so much for your helpful reply. Really appreciate.</p>
An error occurred when I try to initialize a member of the subclass of nn.Module

BaeHann: <p>Hi ,everyone. After I  initialized the following class, no attribute can be found in the initialized member, let alone the subsequent training process. However, if I erase the nn.Module, not setting the InceptionTime as a subclass of nn.Module, the initialized member contain all the attributes as designed. I am curious about the phenomenon. Excuse me for my somewhat naive question.</p>
<p>class InceptionTime(nn.Module):</p>
<pre><code>def __init__(self, input_channel, num_class,
             filters,
             depth,
             models):



    # Build and save the models.
    self.the_models = [
        InceptionModel(
            input_size= input_channel,
            num_classes= num_class,
            filters=filters,
            depth=depth,
        ) for _ in range(models)
    ]
    self.length_before_classification = 4 * filters
</code></pre>

KFrank: <p>Hi Hann!</p>
<aside class="quote no-group" data-username="BaeHann" data-post="1" data-topic="197860" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/439d5e/48.png" class="avatar"> BaeHann:</div>
<blockquote>
<p>After I  initialized the following class, no attribute can be found in the initialized member</p>
</blockquote>
</aside>
<p>You need to explicitly initialize the superclass, <code>Module</code>, as well as the class<br>
itself, <code>InceptionTime</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>class InceptionTime(nn.Module):</p>
<pre><code>def __init__(self, input_channel, num_class,
             filters,
             depth,
             models):

    # Build and save the models.
</code></pre>
<p>…</p>
</blockquote>
</aside>
<p>Add <code>super().__init__()</code> to <code>InceptionTime</code>’s <code>__init__()</code> method.  Doing<br>
so will call the superclass’s (<code>Module</code>’s) <code>__init__()</code> method (with no explicit<br>
arguments).</p>
<p>(Unlike some other languages, python does not automatically initialize a<br>
class’s superclass, even with the superclass’s “default constructor.”)</p>
<p>Best.</p>
<p>K. Frank</p>

BaeHann: <p>Thank you very much, I thought it was a compiler, python version, pytorch version, or deployment issue, but now I feel so stupid that I am embarrassed to read this question again. <img src="https://discuss.pytorch.org/images/emoji/apple/face_exhaling.png?v=12" title=":face_exhaling:" class="emoji" alt=":face_exhaling:" loading="lazy" width="20" height="20"></p>
Tuple to Tensor conversion doesn't work

jwilk: <p>Hello, lately I tried to train a neural network using Sequential and DataLoader. Data loader returns a tuple with length=4 in which every element is an image as torch.Tensor. Because I can’t pass tuple but Tensor as an input to the neural network I tried converting it by firstly making it a numpy array and then converting it to Tensor. However, when trying to convert to numpy array I get a following error: <code>ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (4,) + inhomogeneous part.</code> Each tensor has the same shape as shown in the image below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9bdce6c39a97c5b16d81f2aa6a3c5dbc7354833e.png" data-download-href="https://discuss.pytorch.org/uploads/default/9bdce6c39a97c5b16d81f2aa6a3c5dbc7354833e" title="Screenshot from 2024-02-28 19-04-58"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bdce6c39a97c5b16d81f2aa6a3c5dbc7354833e_2_690x223.png" alt="Screenshot from 2024-02-28 19-04-58" data-base62-sha1="mePhNmd3IUg3Xdh1n8DED0h5zRQ" width="690" height="223" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bdce6c39a97c5b16d81f2aa6a3c5dbc7354833e_2_690x223.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bdce6c39a97c5b16d81f2aa6a3c5dbc7354833e_2_1035x334.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bdce6c39a97c5b16d81f2aa6a3c5dbc7354833e_2_1380x446.png 2x" data-dominant-color="212122"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-02-28 19-04-58</span><span class="informations">2314×750 91.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Is this expected behaviour with tensors as tuple elements? What is interesting is that the same code works just fine in google colab. A workaround I found is to firstly convert every element of tuple to numpy array and then convert whole thing to tensor.</p>

ptrblck: <p>You could use <code>torch.stack</code> to transform the <code>tuple</code> into a single batch:</p>
<pre data-code-wrap="python"><code class="lang-python">batch = tuple([torch.randn(3, 480, 640) for _ in range(4)])

for b in batch:
    print(b.shape)
# torch.Size([3, 480, 640])
# torch.Size([3, 480, 640])
# torch.Size([3, 480, 640])
# torch.Size([3, 480, 640])

batch = torch.stack(batch)
print(batch.shape)
# torch.Size([4, 3, 480, 640])
</code></pre>
When training with DataParallel in parallel, I encountered a data distribution issue

Chihiro1099: <p>I am trying to train my CNN network on multiple GPUs using <code>nn.DataParallel</code> . However, I’ve encountered the following issue: When distributing the input data <code>[a1, a2, a3, a4]</code> to two GPUs (gpu:0 and gpu:1), I expected each GPU to receive data <code>[a1, a2]</code> and <code>[a3, a4]</code> , respectively. However, in reality, I’m getting <code>[a1, a2]</code> on gpu:0  and <code>[0s or data from an unknown source]</code> on  gpu:1.<br>
original input:<br>
tensor([[1., 0., 0., 1.],<br>
[0., 0., 1., 1.],<br>
[0., 1., 0., 1.],<br>
[0., 1., 1., 0.]], device=‘cuda:0’, dtype=torch.float64)<br>
I use hook to obtain and print the data that flow in parallel gpus:<br>
tensor([[1., 0., 0., 1.],<br>
[0., 0., 1., 1.],<br>
[0., 0., 0., 0.],<br>
[0., 0., 0., 0.]], device=‘cuda:0’, dtype=torch.float64)<br>
Can someone tell me why there is such a problem？<br>
Post script：The training data for the network is generated based on the latest model, so there isn’t an existing dataset。</p>
<p>another questions：How to obtain output of intermediate layer When the model is trained on multiple GPUs</p>

ptrblck: <p>Based on the issue your system might have trouble with the p2p connectivity between your GPUs.<br>
You could check <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs">this NCCL FAQ</a> and check if IOMMU might need to be disabled.</p>

Chihiro1099: <p>Thank you for your help, the GPU I use is NVIDIA 4090. I just learned that 4090 does not support nvlink. The above error is most likely caused by that.</p>
Unable to extract features using huggingface swin transformer model

I_am_not_perfect: <p>I need to use swin transfomer as a backbone for my SVM model. I need to extract the features first, therefore I am trying to remove the last layer and train the model. This is a sample executable code.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
import numpy as np

HUB_URL = "SharanSMenon/swin-transformer-hub:main"
MODEL_NAME = "swin_tiny_patch4_window7_224"
model = torch.hub.load(HUB_URL, MODEL_NAME, pretrained=True)
model = nn.Sequential(*list(model.children())[:-1])          #--&gt; If you comment this line, the code is executing
for param in model.parameters():
    param.requires_grad = False


dummy_tensor = torch.randn(32, 3, 224, width)

# Print the shape of the dummy tensor
model(dummy_tensor)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42837d101f09deb3137be09faac9923f2dbc915e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/42837d101f09deb3137be09faac9923f2dbc915e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42837d101f09deb3137be09faac9923f2dbc915e_2_526x500.jpeg" alt="image" data-base62-sha1="9upe0r3TNWvflkKJj3WZkxMseHY" width="526" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42837d101f09deb3137be09faac9923f2dbc915e_2_526x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/4/2/42837d101f09deb3137be09faac9923f2dbc915e.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/2/42837d101f09deb3137be09faac9923f2dbc915e.jpeg 2x" data-dominant-color="F6D8D7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">655×622 122 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What am I missing, I tried the same process with resnet50 and it worked fine. Why am I not able to remove the last layer and get only the output features? what should I do? any suggestions?</p>

ptrblck: <p>I cannot download the model as I’m running into <code>HTTPError: rate limit exceeded</code>.<br>
However, you could check which modules <code>model.children()</code> returns and check if one of them isn’t implementing the <code>forward</code> pass.</p>
How to implement multiple discriminator updates per gen update in PytorchLightning

Aaron_Thomas: <p>Hi there,<br>
I’m currently trying to train a GAN using PyTorch lightning and I want to implement multiple discriminator updates per generator update but i am unsure of the best practice.</p>
<p>Here is my training step:</p>
<pre><code class="lang-auto"> def training_step(self, batch):

        optG, optD = self.optimizers()

        # data and real/fake labels
        real_data, coords = batch

        real_labels = torch.full((real_data.size(0),), 1.0, dtype=torch.float).type_as(
            real_data
        )
        fake_labels = torch.full((real_data.size(0),), 0.0, dtype=torch.float).type_as(
            real_data
        )

        # Generate fake-data using noise input
        input = torch.tensor([])
        fake_data = self.model.generator(input).type_as(real_data).reshape(-1, 1)

        # Configure Samples
        disc_value = self.model.discriminator(coords)

        # Training the generator
        self.toggle_optimizer(optG)
        optG.zero_grad()
        errG = self.adversarial_loss(disc_value, real_labels, fake_data)
        self.log("train_g_loss_step", errG, prog_bar=True)
        self.manual_backward(errG, retain_graph=True)
        optG.step()
        self.untoggle_optimizer(optG)

        # Training the discriminator
        self.toggle_optimizer(optD)
        optD.zero_grad()
        errD_real = self.adversarial_loss(
            disc_value, real_labels, real_data
        )  # Discriminator real loss
        errD_fake = self.adversarial_loss(
            disc_value, fake_labels, fake_data.detach()
        )  # Discriminator fake loss
        errD = (errD_real + errD_fake) / 2
        self.log("train_d_loss_step", errD, prog_bar=True)
        self.manual_backward(errD)
        optD.step()
        self.untoggle_optimizer(optD)
</code></pre>
<p>I have seen two schools of thought, either write a for loop over the discriminator training aspect or in the configure optimizers set the ‘frequency’ parameter, to something like the following:</p>
<pre><code class="lang-auto">return ( {'optimizer': optD, 'frequency': 5}, {'optimizer': optG, 'frequency': 1} ) 
</code></pre>
<p>Any advice on what implementation to use in this framework would be greatly appreciated!</p>

ptrblck: <p>You might want to cross-post the question into the lightning discussion board, as you’ll find the lightning experts there.</p>