akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torchscript symbol lookup / undefined symbol error

kekpirat: <p>Hello,</p>
<p>I was following the basic tutorial for loading an exported model in Torchscript (<a href="https://pytorch.org/tutorials/advanced/cpp_export.html" class="inline-onebox" rel="noopener nofollow ugc">Loading a TorchScript Model in C++ — PyTorch Tutorials 1.8.1+cu102 documentation</a>)<br>
but after I run either<br>
<code>cmake -DCMAKE_PREFIX_PATH="$(python -c 'import torch.utils; print(torch.utils.cmake_prefix_path)')" ..</code><br>
or</p>
<pre><code class="lang-auto">cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..
</code></pre>
<p>and <code>make</code> and try running the executable, I get the error</p>
<pre><code class="lang-auto">./aanet: symbol lookup error: ./aanet: undefined symbol: _ZN5torch3jit4loadERKSsN3c108optionalINS3_6DeviceEEERSt13unordered_mapISsSsSt4hashISsESt8equal_toISsESaISt4pairIS1_SsEEE
</code></pre>
<p>(I’m on Ubuntu 20.04, x86_64, and tried with both pytorch 1.7 and 1.8)<br>
I’ve tried some suggestions I found in previous threads like</p>
<ol>
<li>Creating a brand new conda environment and installing pytorch alone</li>
<li>Adding to CMake <code>set(CMAKE_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=0")</code>
</li>
<li>Using the pre-cxx11 ABI libtorch</li>
</ol>
<p>These are my files</p>
<details>
<summary> CMakeLists.txt </summary>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0)

project(aanet)

# add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0)

set(CMAKE_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=0")

find_package(Torch REQUIRED)

add_executable(${PROJECT_NAME} main.cpp)

target_link_libraries(${PROJECT_NAME} "${TORCH_LIBRARIES}")

set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 14)
</code></pre>
</details>
<details>
<summary> main.cpp </summary>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.

#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char *argv[])
{
    if (argc != 2)
    {
        std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
        return -1;
    }

    torch::jit::script::Module module;
    try
    {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load(argv[1]);
    }
    catch (const c10::Error &amp;e)
    {
        std::cerr &lt;&lt; "error loading the model\n";
        return -1;
    }

    std::cout &lt;&lt; "ok\n";
}
</code></pre>
</details>
<p>I’m completely lost on how to debug this problem (without resorting to a full reset) as I’ve no issues compiling and running this simple example on my other devices.</p>
<p>Any help would be greatly appreciated. Thanks!</p>

Rajesh_Kanna: <p>Hi Kekpirat. I tracked your username and found you here. Since I do not have enough rep in stackoverflow was not able to comment to your answer. I will thank you here. Thank you sensei! &lt;3</p>
How to build libtorch from source? where is the source? I want to build libtorch-win-shared-with-deps-1.12.0-cu116.zip

xoofee: <p>as in the title. I’m wondering how to build libtorch in windows, instead of downloading the prebuilt dll from <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
Calculating mean and stdev of entire dataset for transforms.Normalize() function

Prajakta_Rathod: <p>train_dataset = torchvision.datasets.ImageFolder(<br>
root=TRAIN_ROOT,<br>
transform=transforms.Compose([<br>
transforms.Resize((224,224)),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[3 mean values of RGB channels],<br>
std=[3 stdev values of RGB channels])<br>
])<br>
)<br>
My image size is 398, 398 (Height, Width). I am resizing it to 224, 224 (Height, Width). I need to pass mean and stddev values of entire image dataset to transforms.Normalize() function. I am using image width and height in number of pixels while calculating mean and stdev of entire dataset. shall I use original size i.e. 398, 398 pixels or 224, 224 pixels in mean and stdev calculation of entire dataset.</p>
How to clip gradient of part parameters?

Zhengyi_Li: <p>The optimized parameters use different optimizer and learning rate. They are quite different groups so that I want to clip them separately suing <code>clip_grad_norm_</code>. I made the parameter groups into lists and passed into the <code>clip_grad_norm_</code>, like setting different learning rate for groups. But this seems not work for the gradient clipping.</p>
<p>The document says the parameter needs to be <code>an iterable of Tensors or a single Tensor that will have gradients normalized</code>. What if I have a list to do it？</p>
Mixup: changes in labels when combining images in different classes

alexneakameni: <p>Hi all,<br>
I want to use mixup as the first step of my augmentation data, and I’m scared about the impact of this in my labels.<br>
Can I assume that : <code>t * img1 + (1-t) * img2</code> =&gt; <code>t * y1 + (1-t) * y2</code> regardless of the model I’m using ? In what context could this be true (without degrading the performance of my model)</p>
Training stops due to Caught RuntimeError in DataLoader worker process 0 with large dataset of files

Jere_Poveda_Martinez: <p>Hi everyone,</p>
<p>I’m training a model in a SLURM cluster. My dataset is a a folder tree with almost 7000 MRI files (it’s large). I’m getting the following errors (full error trace <a href="https://drive.google.com/file/d/1oBaLU8Rop49NujsoTdiXn3OnX866DC-g/view?usp=sharing" rel="noopener nofollow ugc">here</a> at the very end):</p>
<ul>
<li><code>RuntimeError: unable to open shared memory object &lt;/torch_3368351_1991753163&gt; in read-write mode</code></li>
<li><code>RuntimeError: Caught RuntimeError in DataLoader worker process 0.</code></li>
</ul>
<p>I’m using <code>torch.multiprocessing.set_sharing_strategy('file_system')</code> to avoid <a href="https://discuss.pytorch.org/t/runtimeerror-received-0-items-of-ancdata/4999">this</a>. I’ve seen many approaches for solving this error, like these <a href="https://discuss.pytorch.org/t/runtimeerror-unable-to-open-shared-memory-object-depending-on-the-model/116090">ones</a>. I wolud like to understant what’s really happening and what’s causing the problem in order to avoid it in future developments. After seeing <a href="https://github.com/pytorch/pytorch/issues/13246" rel="noopener nofollow ugc">this</a>, I remembered that I am cumputing my loss with the following function:</p>
<pre><code class="lang-auto">def weighted_average(
    inputs: Sequence[torch.Tensor], weights: torch.Tensor
) -&gt; torch.Tensor:
    """
    Computes the weighted average of the inputs.

    Args:
        inputs: a sequence of tensors with the same shape.
        weights: a sequence of weights for each tensor

    Returns:
        Weighted average of the inputs.
    """
    _msg_len_ = f"Expected lengths to be equal len(inputs) == len(weights) but got {len(inputs) = } and {len(weights) = }."
    assert len(inputs) == len(weights), _msg_len_
    weighted_sum = [torch.mul(w_i, in_i) for in_i, w_i in zip(inputs, weights)]
    return torch.stack(weighted_sum, dim=0).sum(0) / weights.sum()
</code></pre>
<p><strong>Questions</strong></p>
<ul>
<li>The above function uses a list <code>weighted_sum</code>, could be this the root of the problem, or it may be related to the large amount of files?</li>
<li>If not, using HDF5 would solve the problem allowing me to used the deafult multiprocessing sharing strategy?</li>
</ul>

ptrblck: <p>You might be running out of shared memory so you could try to increase it. Also, in case you are using containers, make sure you are providing enough shared mem to their runtime (e.g. via <code>--ipc=host</code> for docker containers).</p>
Conceptual doubt with ConvTranspose2D and Stride

Pablo_Garcia_Sant: <p>Hi everyone!</p>
<p>I have doubts mainly about how the strides in the ConvTranspose2D layers work.</p>
<p>My model of an Conv AE is :</p>
<pre><code class="lang-auto">    def __init__(self):
        super(ConvAutoencoder, self).__init__()

        # Encoder
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(in_channels=16, out_channels=4, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)

        # Decoder
        self.t_conv1 = nn.ConvTranspose2d(in_channels = 4, out_channels = 16,kernel_size= 2, stride=2)
        self.t_conv2 = nn.ConvTranspose2d(in_channels = 16, out_channels =1, kernel_size=2, stride=2)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        x = F.relu(self.conv2(x))
        x = self.pool(x)
        x = F.relu(self.t_conv1(x))
        x = self.t_conv2(x)
        return x
</code></pre>
<p>If the input has 1 channel and size 4x8, I understand that after the first pool, the size is 2x4 and after the second one is 1x2. I know that after the two transpose layers, it gets size 4x8 again, but I don’t understand how the kernel size and the strides in the ConvTranspose2d layers work to perform this.</p>
<p>Moreover, for example if the input was of size 4x6, I don’t know how to return it to that size with the transpose layers.</p>

ptrblck: <p><a href="https://arxiv.org/abs/1603.07285"> A guide to convolution arithmetic for deep learning</a> is a great reference to as it visualizes how different convolutions are applied. The relevant <a href="https://github.com/vdumoulin/conv_arithmetic">repository</a> also provides some animations.</p>
Fast Fourier transform (torch.fft.fft), How to mask/ignore some indices in tensor

Dee: <p>I have input tensor :</p>
<pre><code class="lang-auto">x=torch.tensor( [ [5,3,4,2,1,4,0,0],
                  [2,3,4,2,1,0,0,0],
                  [8,3,4,0,0,0,0,0],
                  [1,3,4,2,3,1,2,0] ] )
</code></pre>
<p>Instead of zero , there can be any value. I want to <strong>ignore</strong> those indices.</p>
<p>I want to do <code>torch.fft.fft( x, dim = -1 ) </code>.<br>
How can I apply a mask so It calculates before zero values or ignore those values/indices ? Should I use zeros or any other value  or other solution ?<br>
TORCH.FFT.FFT has a parameter n ( Signal length) but it trims for the whole dimension so it doesn’t work for me unless I do it row by row with zero padding.</p>
Sudden drop in training speed after a few batches on cuda

TMaddox: <p>Versions:<br>
pytorch 1.11.0<br>
pytorch-cluster 1.6.0<br>
pytorch-geometric 2.0.4<br>
pytorch-mutex 1.0<br>
pytorch-scatter 2.0.9<br>
pytorch-sparse 0.6.14<br>
pytorch-spline-conv 1.2.1<br>
torchvision 0.12.0<br>
cuda 11.5<br>
cudnn 8.3.2<br>
windows 10</p>
<p>I am training a custom graph neural net with pytorch and am experiencing a sudden drop in training speed by ~ 6x after the first few batches. I used <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="noopener nofollow ugc">pytorch profiler</a> to profile one “epoch” (amount of data greatly reduced). Everything is on cuda (the data too, I wrote my own dataloader as suggested <a href="https://discuss.pytorch.org/t/how-to-load-all-data-into-gpu-for-training/27609/9">here</a>). I have 5GB dedicated RAM on my GPU (nvidia Quadro P2200) which is never full (by at least a margin of 30%). The GPU load is also only at ~ 35%.</p>
<p>The sudden drop is not specific to any function. Everything needs ~ 6x the time.</p>
<p>Here is the trace I am producing with the torch profiler:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48fbd565273627c048044abce11de289ba4a2bc0.png" data-download-href="https://discuss.pytorch.org/uploads/default/48fbd565273627c048044abce11de289ba4a2bc0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48fbd565273627c048044abce11de289ba4a2bc0_2_690x215.png" alt="image" data-base62-sha1="apDVAnhRlh3vP2dNKjI1gjms5Bm" width="690" height="215" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48fbd565273627c048044abce11de289ba4a2bc0_2_690x215.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48fbd565273627c048044abce11de289ba4a2bc0_2_1035x322.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48fbd565273627c048044abce11de289ba4a2bc0_2_1380x430.png 2x" data-dominant-color="E1E8E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1883×587 75.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What could cause the sudden performance drop and how can I circumvent this? Thanks!</p>
Scatter with two index tensor

Cyu: <p>Dear all,</p>
<p>suppose I have two index Tensor</p>
<p>index_1, size is [N], and index_2 size is also [N].</p>
<p>Now I have a target tensor, and src tensor like [N, C], what I want is</p>
<p><code>target[index_1[i],  index_2[i], j] = src[i, j]</code></p>
<p>i is in range N and j is in range C</p>
<p>The most similar function I found is torch.scatter(),</p>
<p><code>Tensor.</code> <code>scatter_</code> (<em>dim</em> , <em>index</em> , <em>src</em> , <em>reduce=None</em> ) works like</p>
<p><code>target[index[i, j], j] = src[i, j]</code></p>
<p>but it supports only one index tensor. Do you have any alternatives? Because N is quite large, looping is not realistic. I have also tried by indexing</p>
<p><code>target[index_1,  index_2, :] = src</code></p>
<p>It not raise error but just not correct</p>
<p>Thanks</p>

ptrblck: <p>Could you post a minimal code snippet showing the desired result using your “slow” approach?<br>
I’m unsure if this would work:</p>
<pre><code class="lang-python">N, C = 2, 3

target = torch.zeros(N, N, C)
src = torch.randn(N, C)
index1 = torch.randint(0, N, (N,))
index2 = torch.randint(0, N, (N,))

target[index1.unsqueeze(1), index2] = src
</code></pre>
<p>so I would like to compare it to your reference.</p>
Loading data from SQL with IterableDataset

wmpauli: <p>I worked on a project recently where I needed to pull data directly from a SQL table via sqlalchemy/odbc. I found it quite challenging at the time, so I thought I’d share my approach here in case anybody finds themselves in a similar situation, or would like to add.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/Azure/elenchus">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/Azure/elenchus" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/231cef3a1880e2fff5022c59083b0164d7395dcfa67e0469f39bcb9d8b78bbb2/Azure/elenchus" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/Azure/elenchus" target="_blank" rel="noopener nofollow ugc">GitHub - Azure/elenchus: Machine teaching with Azure SQL</a></h3>

  <p>Machine teaching with Azure SQL. Contribute to Azure/elenchus development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The repo demonstrates how to fine-tune a hugging face transformer model for text classification (I chose QNLI for now). It can easily be configured for other subsets of the GLUE benchmark dataset.</p>
<p>For the implementation of the dataset class, I chose to subclass IterableDataset (i.e. torch.utils.data.IterableDataset; <a href="https://pytorch.org/docs/stable/data.html" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.12 documentation</a>). Quoting the PyTorch documentation: “An iterable-style dataset is an instance of a subclass of IterableDataset that implements the <strong>iter</strong>() protocol, and represents an iterable over data samples. This type of datasets is particularly suitable for cases where random reads are expensive or even improbable, and where the batch size depends on the fetched data.” This contrasts with the standard Dataset class, which implements a method “<strong>getitem</strong>(index)” that for selecting a sample from the dataset by its index.</p>
<p>Perhaps a downside of this approach is that this class cannot easily be combined with PyTorch Samplers. Again quoting the PyTorch documentation: “For iterable-style datasets, data loading order is entirely controlled by the user-defined iterable. This allows easier implementations of chunk-reading and dynamic batch size (e.g., by yielding a batched sample at each time).”</p>
<p>The repo is pretty minimal at this point, but might still be useful for some.</p>
Prefetch_factor not prefetching?

wmpauli: <p>I have an iterable dataset with the following <code>__iter__()</code> method.</p>
<pre><code class="lang-auto">def __iter__(self):
   for track in self.selected_tracks:
      samples = self.load_track(track)
      for sample in samples:
         yield sample
</code></pre>
<p>where <code>load_track</code> loads a number of samples, which takes a long time.</p>
<p>Use a standard dataloader, with <code>for sample in dataloader</code>. I’m able to run this with several persistent workers. However, I noticed that in between calls to <code>load_track</code>, all the workers are just sitting there. I thought I could use <code>prefetch_factor</code> with a number larger than the number of samples returned by <code>load_track</code>, but it doesn’t seem to have an effect. Am I understanding prefetch_factor correctly, in that it loads samples for the next iteration on the dataloader?</p>

lkc1: <p>I’m seeing a similar behavior. Changing the prefetch_factor does not change the latency of fetching batches at all.</p>

wmpauli: <p><a class="mention" href="/u/lkc1">@lkc1</a>, I did not dig much deeper into this issue, because I realized that the reason the workers were just sitting there was that I was pulling data from a very large SQL table, and they were all just waiting for the data.</p>
Positional encoding at padded values

gmfm27: <p>Hi there, we are using a transformer architecture for knowledge tracing purposes (as per the implementation described in this paper: <a href="https://arxiv.org/pdf/2010.12042.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2010.12042.pdf</a>)</p>
<p>We are using left padding on sequences shorter than our max sequence length, and were wondering if we need a mask for the positional encoding function too, because the positional encoding is applied to the encoder input, resulting in non zero values before being inputted into the transformer.</p>
<p>Doesn’t this mean that these padded values will have some effect on prediction now, when they should not be affecting the prediction e.g. of values later on in the sequence?</p>
<details>
<summary>
Summary</summary>
<p>This text will be hidden</p>
</details>
Request for comprehensive documentation for Libtorch

SM19: <p>Dear Pytorch Team,</p>
<p>Pytorch’s documentation is probably one of the best I have ever seen.<br>
However, the case with Libtorch is rather sad!!!<br>
There are just some very basic examples which seems pretty random and not well organized.<br>
I understand that it follows Pytorch in terms of use.</p>
<p>However, to really make it a solid framework for C++ usage, I would request a thorough documentation for the C++ frontend as well. At the very least - each API needs at least a 1 line example.</p>
<p>Presently we have to actually see the source code headers and try to use it.<br>
This is not very nice.</p>
<p>I would really love to see Pytorch/Libtorch as a single solution for all deep learning needs.</p>
<p>Best Regards<br>
Sambit</p>

ptrblck: <p>Thanks for the feedback as I agree that the C++ docs could be improved.<br>
Contributions in this area are more than welcome so would you be interested in improving (some of) these docs?</p>

SM19: <p>Yes, surely. I would love to contribute my 2 cents whenever possible.<br>
How may I add/ edit documentation?</p>

ptrblck: <p>The <code>libtorch</code> docs should be <a href="https://github.com/pytorch/pytorch/tree/master/docs/cpp/source">here</a> and you could add more documentation there.<br>
I would recommend to create a GitHub issue first, explain the lack of docs, your interest in fixing (some of) these issues, and then follow up with a PR after discussing these changes.</p>

walkacross: <p>hi there, i launch a <a href="https://github.com/walkacross/pytorch-libtorch-API-translation" rel="noopener nofollow ugc">project</a>, which aims to prodive API translation between Pytorch(python version) and Libtorch(C++ version).</p>
<p>all code lives in jupyter notebook, you can interactively write C++ in notebook.</p>
<p>welcome pull request.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/walkacross/pytorch-libtorch-API-translation">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/walkacross/pytorch-libtorch-API-translation" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/fc0787100a4facec6afa4e2067b1ea151f892032d0207b0b5f2bfb469cad40b4/walkacross/pytorch-libtorch-API-translation" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/walkacross/pytorch-libtorch-API-translation" target="_blank" rel="noopener nofollow ugc">GitHub - walkacross/pytorch-libtorch-API-translation: api translation and...</a></h3>

  <p>api translation and comparsion between pytorch(python) and libtorch(c++) - GitHub - walkacross/pytorch-libtorch-API-translation: api translation and comparsion between pytorch(python) and libtorch(...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Validation loss is much higher than training loss on first epoch and after

Krzysztof_Bieniek: <p>I tried to implement the FinerPCN model in PyTorch (the original code is in TF) for point cloud completion, but modified it to accept tensor size [B, N, 6] instead of [B, N, 3].<br>
The problem is that I’m getting a validation error of 833.99 on the first epoch and a training loss of 0.6.<br>
After that training loss is decreasing, but validation loss jumps higher / lower.<br>
What could be the cause of that?</p>
<p>Here’s my model: <a href="https://github.com/chrisvonbienias/masters-thesis/blob/main/models/model_finerpcn.py" rel="noopener nofollow ugc">https://github.com/chrisvonbienias/masters-thesis/blob/main/models/model_finerpcn.py</a></p>
MPS support for Torchaudio?

antoinerpr: <p>Hi everyone,</p>
<p>My question is simple: Are Torchaudio APIs soon gonna be able to support MPS to perform computations on M1 mac GPUs?</p>
<p>Here is the doc and it doesn’t mention any support thus far, apart from CPU and CUDA:<br>
<a href="https://pytorch.org/audio/main/supported_features.html" rel="noopener nofollow ugc">https://pytorch.org/audio/main/supported_features.html</a></p>
<p>Is there anything in the pipes with a coming release?</p>
<p>Cheers</p>
<p>Antoine</p>
Joint Parameter Multivariate RNN

jayz: <p>Hello, I am trying to extract joint features from H time series based on a single RNN. I have data tensors X of shape N x C x H x L, where N batch size, C is the number of features used as the input_size for the RNN, and L is the sequence length.</p>
<p>I’d like to use a single RNN to extract features from the H parallel time series. For this, I loop through the H rows (N x C x L) of the tensor X and feed them one after the other into the same RNN. The class <code>CustomRNN</code> below defines this RNN and takes care of reshaping the batch correctly (needs to be fed as (N x L x C) to the RNN. The <code>CustomRNN</code> maps from C → C hidden size and outputs a sequence of length L, again (with all the hidden states). The class <code>JointRNN</code> defines a single <code>CustomRNN</code> and loops through the rows of X, feeding them to the RNN and concatenating them to the final tensor. Is this the correct approach to extract joint RNN features from the individual time series?</p>
<pre><code class="lang-auto">class CustomRNN(nn.Module):

    def __init__(self,C): 
        super().__init__()
        self.rnn = nn.RNN(C,C,batch_first=True)
        return 
    
    def forward(self,x): 
        # input:  N x C x L
        # return: N x C x 1 x L
        return self.rnn(x.transpose(1,2))[0].transpose(1,2).unsqueeze(2)

class JointRNN(nn.Module): 
    
    def __init__(self,C,H,**kwargs): 
        super().__init__()
        self.H = H    
        self.rnn = CustomRNN(C)     
        return 
    
    def forward(self,x): 
        out = torch.tensor([],device=device)
        for i in range(self.H): 
            out = torch.cat([out,self.rnn(x[:,:,i,:])],dim=2)
        # return: N x C x H x L
        return out  
</code></pre>
<p>Thanks!</p>
Initialising the weights of the first convolutional layer while changing the number of channels in pretrained RESNET

Adit_Whorra: <p>I am trying to change the number of channels of a pre-trained resnet model in pytorch. Here is the code -</p>
<pre><code class="lang-auto">from torchvision.models import models
resnet50 = models.resnet50(weights="IMAGENET1K_V2")
resnet.conv1 = nn.Conv2d(
    in_channels, out_channels=64, kernel_size=7, stride=2, padding=3, bias=False
)
</code></pre>
<p>This seems to be working. I am verifying it by doing -</p>
<pre><code class="lang-auto">summary(model, (1, 224, 224))
</code></pre>
<p>This is returning the summary correctly.</p>
<p>However, what I am confused about is that currently, the weights of the first convolutional layer are initialised randomly. Is there a way I can initialise the first layer from the weights of the pretrained 3-channel pytorch model? One way to do it would be to take a mean of the weights from the pretrained model.</p>
<p>The shape of the weight matrix of the first conv layer for 3 channels is torch.Size([64, 3, 7, 7]) whereas for 1 channel it is torch.Size([64, 1, 7, 7]). Does taking the mean across the second dimension make sense?</p>

fpozzi: <p>Hi <a class="mention" href="/u/adit_whorra">@Adit_Whorra</a>,<br>
I think you can explore some solutions:</p>
<ol>
<li>Adapt input. You can convert the 1 channel image to 3 channels one (just replicating the channel).</li>
<li>Adapt first model layer. The solution you suggested. To initialize your first layer, you can try to use the <em>mean</em> or the <em>sum</em> over channels of the 3-channel pre-trained weights.</li>
</ol>
<p>Evaluate both solutions on ImageNet dataset, then you decide.</p>
<p>Note: All solutions do not require re-training.</p>

Adit_Whorra: <p>The first solution will unnecessarily add extra parameters so I think the second option makes more sense. Will the resultant weights still retain what the model has learnt from pre-training though after taking the mean or sum?</p>

fpozzi: <blockquote>
<p>The first solution will unnecessarily add extra parameters so I think the second option makes more sense.</p>
</blockquote>
<p>I would not say so. The number of extra parameters is negligible.</p>
<p>Consider the following example:</p>
<pre><code class="lang-auto">from torchvision import models
import torch.nn as nn

resnet50_gray = models.resnet50(weights="IMAGENET1K_V2")
resnet50_gray.conv1 = nn.Conv2d(
    1, out_channels=64, kernel_size=7, stride=2, padding=3, bias=False
)

resnet50_rgb = models.resnet50(weights="IMAGENET1K_V2")
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; print("gray", count_parameters(resnet50_gray))
gray: 25550760
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; print("rgb", count_parameters(resnet50_rgb))
rgb:  25557032
</code></pre>
<hr>
<blockquote>
<p>Will the resultant weights still retain what the model has learnt from pre-training though after taking the mean or sum?</p>
</blockquote>
<p>The main point IMO is to compute how many accuracy points are lost after that operation. That’s why use should try all the alternatives.</p>
Multithreading torchscript

netif121: <p>Hello, I would like to use the same model in multithreading, but when I define the model on each thread separately, the program allocates the same amount of memory in the GPU regardless of the number of threads, and when I try to increase the number of threads the performance drops. It seems that each thread is using the same model, even if they are allocated on different stack. How can I separate them ?</p>
Convert tensor into an array

Dan_Sagher: <p>Hi,<br>
I’m new in Pytorch and I would like to know how to convert an existing tensor into an int array.<br>
I tried to do:</p>
<p>temp_arr = temp_tensor.data&lt; int&gt;();</p>
<p>when “temp_arr” is an (int *), and “temp_tensor” type is int, but i keep getting run time error.<br>
Thank you very much!</p>

Matej_Kompanek: <p>I think data is templated member so I would suggest using .data&lt;int&gt;()</p>
<pre><code class="lang-auto">auto temp_tensor = torch::randn({ 3, 4 }).to(torch::kInt32);

int* temp_arr = temp_tensor.data&lt;int&gt;();
		
// alternative
//int* temp_arr = (int*)temp_tensor.data_ptr();

for (int y = 0; y &lt; temp_tensor.sizes()[0]; ++y)
{
	for (int x = 0; x &lt; temp_tensor.sizes()[1]; ++x)
	{
	    std::cout &lt;&lt; (*temp_arr++) &lt;&lt; " ";
	}

	std::cout &lt;&lt; std::endl;
}

std::cout &lt;&lt; temp_tensor &lt;&lt; std::endl;
</code></pre>

Dan_Sagher: <p>Thank you for your comment!<br>
That’s true, when I put &lt;&gt; in my post it deleted it so now i added space (&lt; int&gt;).</p>
<p>Anyway, I keep getting an error.</p>

Matej_Kompanek: <p>What does the error say ? There’s usually pretty good explaining message. Do you catch those errors ? -&gt;</p>
<pre><code class="lang-auto">try
{
	auto temp_tensor = torch::randn({ 3, 4 }).to(torch::kInt32);
	int* temp_arr = temp_tensor.data&lt;int&gt;();
}
catch (std::runtime_error&amp; e)
{
  std::cout &lt;&lt; e.what() &lt;&lt; std::endl;
}
catch (const c10::Error&amp; e)
{
   std::cout &lt;&lt; e.msg() &lt;&lt; std::endl;
}

system("PAUSE");
</code></pre>

Dan_Sagher: <p>The problem was that after some functions the tensor type was converted to long so i got the massage:</p>
<p>expected scalar type Int but found Long</p>
<p>Thanks again!</p>

fucker007: <p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  expected scalar type Int but found Long<br>
Exception raised from data_ptr at aten/src/ATen/core/TensorMethods.cpp:20 (most recent call first):<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x3e (0x7f66dab3aa1e in /home/nathan/app/libtorch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span> c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&amp;) + 0x5c (0x7f66dab13ce4 in /home/nathan/app/libtorch/lib/libc10.so)<br>
frame <span class="hashtag">#2:</span> int* at::TensorBase::data_ptr() const + 0x112 (0x7f66c3b5b2b2 in /home/nathan/app/libtorch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#3:</span> ./bins/run_test() [0x41e94a]<br>
frame <span class="hashtag">#4:</span> ./bins/run_test() [0x41cf21]<br>
frame <span class="hashtag">#5:</span> ./bins/run_test() [0x41d3d2]<br>
frame <span class="hashtag">#6:</span> ./bins/run_test() [0x41337b]<br>
frame <span class="hashtag">#7:</span> ./bins/run_test() [0x4133d3]<br>
frame <span class="hashtag">#8:</span> __libc_start_main + 0xf0 (0x7f66c01f7840 in /lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag">#9:</span> ./bins/run_test() [0x4130a9]</p>
Just to be clear on feature extraction

SU801T: <p>Just wanted to be clear on the pytorch tutorial for feature extraction using a convnet: <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.12.0+cu102 documentation</a></p>
<p>Am I right to assume that in this set-up, batch norm layers are still active (depending on the model)?</p>
<p>If so, is there a benefit to this?</p>
<p>I have read elsewhere that batch norm layers should also be frozen and thus should use the imagenet global stats, rather than update when doing feature extraction. If this holds true, then why does the tutorial allow for batch norm layers to remain active?</p>

ptrblck: <p>Yes, I think you are right that <code>model.train()</code> is used during the training/fine-tuning.<br>
I guess the author saw benefits in the final accuracy of the fine-tuned model, but you could compare both approaches and check if this holds true.</p>
<p>Could you post the reference you saw as it could also explain by these stats are not trained (maybe the model is more eager to overfit)?</p>

SU801T: <p>Yes, sure. Here is one paper I came across:</p>
<aside class="onebox pdf" data-onebox-src="https://proceedings.mlr.press/v143/kanavati21a/kanavati21a.pdf">
  <header class="source">

      <a href="https://proceedings.mlr.press/v143/kanavati21a/kanavati21a.pdf" target="_blank" rel="noopener nofollow ugc">proceedings.mlr.press</a>
  </header>

  <article class="onebox-body">
    <a href="https://proceedings.mlr.press/v143/kanavati21a/kanavati21a.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://proceedings.mlr.press/v143/kanavati21a/kanavati21a.pdf" target="_blank" rel="noopener nofollow ugc">kanavati21a.pdf</a></h3>

  <p class="filesize">1011.22 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Where the authors state the following:</p>
<blockquote>
<p>Within the context of fine-tuning a model, the batch normalisation (batch norm) (Ioffe and Szegedy, 2015) layer requires taking a few precautions due to how it operates differently between training and inference. During training, the layer uses the current batch mean and standard deviation to normalise the activations, and, at the same time, it updates exponentially moving averages of the mean and standard deviation and stores them as non-trainable weights to use during inference. While fine-tuning a model, it is usually recommended to use the batch norm layer in inference mode to avoid unexpected or poor performance on the validation and test sets from additional updates during training.</p>
</blockquote>
Evaluate pre-trained Faster R-CNN on COCO dataset

dhyeon.samsung: <p>Hello,</p>
<p>I am trying to evaluate pre-trained faster r-cnn from torchvision on COCO dataset.</p>
<p>I first referred to “TorchVision Instance Segmentation Finetuning Tutorial”, but I couldn’t figure out the error.</p>
<p>Is there any way that I could evaluate pre-trained model provided in torch?</p>
<p>Thank you in advance!</p>
Store original data (e.g., text, image) along with tensor data in Pytorch Dataloader

Krishna_Garg: <p>Currently, I am using <code>TensorDataset</code> followed by <code>DataLoader</code> to load my dataset like below:</p>
<pre><code class="lang-auto">tensor_loader = TensorDataset(x_input_ids,x_seg_ids,x_atten_masks,y)
data_loader = DataLoader(tensor_loader, shuffle=True, batch_size=batch_size)
</code></pre>
<p>I now want to also store original (text) data along with the tensor data in the <code>data_loader</code> like below:</p>
<pre><code class="lang-auto">tensor_loader = TensorDataset(x_input_ids,x_seg_ids,x_atten_masks,y, x_input_strs)
</code></pre>
<p>Note: <code>x_input_strs</code> is text data corresponding to <code>x_input_ids</code> but it fails since <code>TensorDataset</code> allows only tensors. I also tried something like this:</p>
<pre><code class="lang-auto">tensor_loader = Dataset(x_input_ids,x_seg_ids,x_atten_masks,y, x_input_strs)
</code></pre>
<p>But it gives the following error:</p>
<pre><code class="lang-auto">TypeError: object.__new__() takes exactly one argument (the type to instantiate)
</code></pre>
<p>Any suggestions are appreciated.</p>

nivek: <p><code>TensorDataset</code> only accepts <code>*tensors: Tensor</code> as argument(s) and <code>Dataset</code> only takes one argument.</p>
<p>One possible solution is to <code>zip</code> your inputs into tuples and pass them to <code>Dataset</code>. Then in your <code>__getitem__</code> you can decide whether you would like to return a tuple including your original data or discard it.</p>
System CUDA version vs torch wheel CUDA version

ender.wieczorek: <p>Does the CUDA version I have installed on my system need to match the CUDA version of the torch wheel?</p>

smth: <p>no, this is not necessary. PyTorch wheels ship with all the CUDA libraries they need.</p>
<p>You only need to have updated NVIDIA driver.</p>

ender.wieczorek: <p>Is it also true when I have a dynamically compiled CUDA kernel (*.cu)? In this case PyTorch calls nvcc installed on the system.</p>

smth: <p>if you have dynamically compiled cu files, then you do have to match pytorch cuda version and system cuda version</p>

ender.wieczorek: <p>Got it, thanks for confirming, Soumith!</p>
The default padding mode seems not to be zero?

ojipadeson: <p>I’m new to pytorch and I found that the padding mode of conv2d is not zero padding in the following code:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
a = torch.tensor([[[[1,2,3,4],[4,5,6,7],[7,8,9,10]]]])
weight = torch.ones((1,1,3,3)).to(torch.long)
F.conv2d(a, weight, stride=1, padding=1)
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">tensor([[[[12, 21, 27, 20],
          [27, 45, 54, 39],
          [24, 39, 45, 32]]]])
</code></pre>
<p>When without padding, the output is:</p>
<pre><code class="lang-auto">tensor([[[[45, 54]]]])
</code></pre>
<p>I read from the document that the default padding mode is “zero”, which means use zero to pad output tensors. So is there anything that changes the mode?</p>

ptrblck: <p>The default padding value is <code>0.</code> and you can manually verify it:</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F

a = torch.tensor([[[[1,2,3,4],[4,5,6,7],[7,8,9,10]]]])
weight = torch.ones((1,1,3,3)).to(torch.long)
ref = F.conv2d(a, weight, stride=1, padding=1)

a_pad = F.pad(a, pad=(1, 1, 1, 1), value=0.)
print(a_pad)

out = F.conv2d(a_pad, weight, stride=1, padding=0)

print((ref - out).abs().max())
# tensor(0)
</code></pre>
<p>The output will of course not contain zeros at the border since your kernel has a spatial size of <code>3x3</code> and will thus process the zero padded border as well as values from the input.</p>
Line width in source code documentation

Connor_Anderson: <p>Hey, when viewing source code through the documentation site, it appears that the setting for the width of a line is less than what is used in the formatting for the code itself, so that many of the lines wrap around making it harder to read the code. Would it be simple to change this setting so that the code doesn’t wrap around?</p>

ptrblck: <p>Could you show an example of this line wrapping, please?</p>

Connor_Anderson: <p>For example, here’s one spot. Look about halfway down, line starting with <code>msg += ...</code>. There are generally some long lines like this that get wrapped in many of the source files that I look at.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4bca91da53cd54d36570bc49c553d8c08c648cf.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a4bca91da53cd54d36570bc49c553d8c08c648cf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4bca91da53cd54d36570bc49c553d8c08c648cf_2_690x429.jpeg" alt="image" data-base62-sha1="nvkvxdXXniDgDdrbGmanQaj9i1x" width="690" height="429" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4bca91da53cd54d36570bc49c553d8c08c648cf_2_690x429.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4bca91da53cd54d36570bc49c553d8c08c648cf_2_1035x643.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4bca91da53cd54d36570bc49c553d8c08c648cf_2_1380x858.jpeg 2x" data-dominant-color="F2F0F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1194 129 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Ah I see, thanks for sharing it.<br>
It seems that the line width is indeed shorter in the docs as the original <a href="https://github.com/pytorch/vision/blob/8068594aecf453840d829e48fa374e568febf76d/torchvision/datasets/folder.py#L101">source code</a> doesn’t break this line.<br>
Could you create a <a href="https://github.com/pytorch/pytorch/issues">GitHub issue</a> for it so that the code owners could take a look at it, please?</p>
Recommender system custom loss with kNN not working

emnlscr: <p>Hi everyone</p>
<p>I’m trying to train a recommender system that takes as input a query (1xN-dim vector), an index(QxN-dim vector) and performs a kNN search to find the k closest (most similar) arrays. For the supervised training, I want compute the mean precision at k score (<a href="http://sdsawtelle.github.io/blog/output/mean-average-precision-MAP-for-recommender-systems.html" rel="noopener nofollow ugc">here a post about the metric</a>). In other words, if the labels of the k recommended indexes are the same as the query the score is higher. The model is a simple MLP that works on the output of another model that extracts some embeddings. I want to use 1 minus the score as loss function to train my model but this doesn’t work.</p>
<p>Here is my loss function and a minimum reproducible example:</p>
<pre><code class="lang-auto">def knn_mpatk_loss(query_arrays, query_labels, index_arrays, index_labels, k=3):
    
    queries = query_arrays.shape[0]
    
    distances = torch.cdist(query_embeds, index_embeds)
    knn = distances.topk(k, largest=False)
    nbrs = knn.indices
    nn_labels = index_labels[nbrs]
    mAP = torch.sum(nn_labels == torch.unsqueeze(query_labels,1))/(k*queries)
    
    return mAP

rnd_query_embeds = torch.tensor(np.random.random((10,100)), requires_grad=True)
rnd_query_labels = torch.tensor(np.random.randint(0,5,(10))/1.0, requires_grad=True)
rnd_index_embeds = torch.tensor(np.random.random((100,100)), requires_grad=True)
rnd_index_labels = torch.tensor(np.random.randint(0,5,(100))/1.0, requires_grad=True)

loss = 1 - knn_mpatk_loss(rnd_query_embeds, rnd_query_labels, rnd_index_embeds, rnd_index_labels, k=3)

print(loss.requires_grad)

</code></pre>
<p>I got <code>False</code> for the <code>print(loss.requires_grad)</code> and of course, if I start the training I got:</p>
<pre><code class="lang-auto">----------
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

</code></pre>
<p>Thank you.</p>
<p>EDIT:<br>
After a more accurate inspection, I think the problem might be in the boolean comparison when generating mAP. Any suggestion on that step?</p>
<p>EDIT2:<br>
The code now is reproducible and it can be seen that the output of <code>print(loss.requires_grad)</code> is <code>False</code>.</p>

AlphaBetaGamma96: <p>Can you share what the <code>distance</code> function is? (And make sure the example code you share is a complete minimal reproducible example? So people can debug it themselves?)</p>
<p>If you want a <code>torch</code> equivalent version to <code>==</code> you can just use <code>torch.equal</code></p>

emnlscr: <p>Hi, thank you for your answer. About the distances I made a typo and corrected it. I tried to use <code>Torch.equal</code> and I get the same issue. To ensure reproducibility, I added the code for 4 random tensors and check if the function output has the <code>requires_grad=True</code></p>
Using gpu_reduce_kernel from a PyTorch extension

Giodiro: <p>I used to be able to compile an CUDA extension which included <code>&lt;ATen/native/cuda/Reduce.cuh&gt;</code> to use the <code>gpu_reduce_kernel</code> function.<br>
In pytorch 1.12 Reduce.cuh started to include <code>ATen/native/cuda/jit_utils.h</code> which in turn includes <code>ATen/cuda/nvrtc_stub/ATenNVRTC.h</code> which does not seem to be included in the headers which are distributed with pytorch.<br>
Is there another way I can use <code>gpu_reduce_kernel</code> or something similar? Am I doing something I’m not supposed to by trying to use it from an extension?</p>

ptrblck: <p>This seems like a newly introduced issue, so could you create a <a href="https://github.com/pytorch/pytorch/issues">GitHub issue</a> for it, so that we can track and fix it, please?</p>

Giodiro: <p>Sure! For reference: <a href="https://github.com/pytorch/pytorch/issues/82408" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/82408</a></p>
How to perform a zero masking for RNN in Pytorch

LXR: <p>I have some time series data padded with 0s in the shape of (Batch, length, features). For more detail, I extracted MFCCs from audio files with (60,40), 60 frames, and 40 MFCCs for each audio file input.</p>
<p>I used to run Tensorflow and applied the <code>Masking</code> layer with the value I wish to mask.</p>
<p>I am trying to do the same thing in Pytorch. I have done some research on this and found people mentioning <code>pack_padded_sequence</code>  from <code>from torch.nn.utils.rnn </code></p>
<p>It appears that <code>pack_padded_sequence</code> is the only way to do a mask for Pytorch RNN.</p>
<p>I have rewritten the dataset preparation codes and created a list containing all the 2D array data. It is a list with a length of 12746 and the 2d array inside is in the form of (x,40); “x” can be any number lower than 60. So basically I am going to prepare data for training in the shape of (12746,60,40)</p>
<p>How should I proceed as the packed sequence cannot be created as a PyTorch dataset?</p>
<pre><code class="lang-auto">class mydata(Dataset):
   def __init__(self, X, y):
        self.X = torch.FloatTensor(X)
        self.y = torch.FloatTensor(y)
                
   def __len__(self):
        return len(self.X)
    
   def __getitem__(self, index):
        y = self.y[index]
        X = self.X[index] 
        return X,y


padded = pad_sequence(data, batch_first=True, padding_value=0.0)
lengths = torch.tensor([len(t) for t in data])
# print('#padded', padded)

print('--------------------------------------------')
packed = torch.nn.utils.rnn.pack_padded_sequence(padded, lengths.to('cpu'), batch_first=True, enforce_sorted=False)

#split them into 0.7 proportion. It was done using Train_test_split.
X_train = packed[0:8900]
y_train = y[:8900]
X_valid = packed[8900:]
y_valid = y[8900:]

train_dataset = mytools.mydata(X_train,y_train)
valid_dataset = mytools.mydata(X_valid,y_valid)
trainloader = DataLoader(train_dataset, batch_size=256, shuffle=True, num_workers=0)
validloader = DataLoader(valid_dataset, batch_size=256, shuffle=False, num_workers=0)
</code></pre>
<p>I was thinking should this <code>pack pad</code> procedure be done after I created the dataloader and right before feeding the input to the rnn?</p>