akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Profiling waveglow by NVIDIA Nsight Compute

username: minty99
cooked: <p>I tried to profile <a href="https://github.com/NVIDIA/waveglow" rel="nofollow noopener"> https://github.com/NVIDIA/waveglow </a> by this command:</p>
<pre><code class="lang-auto">nv-nsight-cu-cli --export ./nsight_output ~/.virtualenvs/waveglow/bin/python3 inference.py -f &lt;(ls mel_spectrograms/*.pt) -w waveglow_256channels.pt -o . --is_fp16 -s 0.6
</code></pre>
<p>Python command is from instruction of <a href="https://github.com/NVIDIA/waveglow#generate-audio-with-our-pre-existing-model" rel="nofollow noopener">https://github.com/NVIDIA/waveglow#generate-audio-with-our-pre-existing-model</a> ,<br>
and it works with Nsight System, not Nsight Compute.</p>
<p>Profiling doesn’t end printing this log; so I pressed Ctrl+C.<br>
Also, It profiles only one kernel but I have more kernels. (checked by Nsight Systems)</p>
<pre><code class="lang-auto">...
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 286: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 287: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 288: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 289: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 290: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 291: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 292: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 293: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 294: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 295: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 296: 0%....50%...^C
==PROF== Received signal, trying to shutdown target application
 - 43 passes
==ERROR== Failed to profile kernel "weight_norm_fwd_first_dim_ker..." in process
==ERROR== An error occurred while trying to profile.
==ERROR== An error occurred while trying to profile
==PROF== Report: nsight_compute_result.nsight-cuprof-report
</code></pre>
<p>OS:  CentOS Linux 7, Nsight Compute (2019.3.1, Build 26317742)<br>
GPU: Tesla V100-PCIE-32GB</p>
<p>How can I fix this?</p>
T >= 0 && t < n_classes error

username: J_Na
cooked: <p>I’m trying to load EMNIST from pytorch datasets which have 26 classes.<br>
So I modify vgg16() model to have 26 num_classes (1 to 26). but this error occurs.<br>
What`s wrong?</p>
<pre><code class="lang-auto">/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [3,0,0] Assertion `t &gt;= 0 &amp;&amp; t &lt; n_classes` failed.

</code></pre>

username: InnovArul
cooked: <p>Print the targets coming out of your dataset and make sure they are within [0,25]. Are they?</p>
RuntimeError: RuntimeError: output with backend CPU and dtype Long doesn't match the desired backend CPU and dtype Float

username: se7en
cooked: <p>Has anyone met this problem before?</p>

username: Deepali
cooked: <p>Sounds similar to - <a href="https://discuss.pytorch.org/t/gpytorch-runtimeerror-expected-backend-cpu-and-dtype-double-but-got-backend-cpu-and-dtype-float/44309" class="inline-onebox">Gpytorch: RuntimeError: expected backend CPU and dtype Double but got backend CPU and dtype Float?</a></p>
Avoid getting caught in loops with LSTM

username: JohnCwok
cooked: <p>Hello !</p>
<p>I recently implemented a name generating RNN frms scratch which was doing ok but far from perfect. So I thought about trying my luck with LSTM cells to see if it makes a difference. Indeed it does and the outpus looks way better for the first 7 ~ 8 characters. But then the networks gets caught in a loop and outputs things like “laulaulaulau” or “rourourourou” (it is supposed the generate french names).</p>
<p>Is it a often occuring problem ? If so do you know a way to fix it ? I’m concern about the fact the network doesn’t produce EOS tokens…</p>
<p>here is the model :</p>
<pre><code class="lang-auto">class pytorchLSTM(nn.Module):
    def __init__(self,input_size,hidden_size):
        super(pytorchLSTM,self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size)
        self.output_layer = nn.Linear(hidden_size,input_size)
        self.tanh = nn.Tanh()
        self.softmax = nn.LogSoftmax(dim = 2)
        
    def forward(self, input, hidden)
            out, hidden = self.lstm(input,hidden)
            out = self.tanh(out)
            out = self.output_layer(out)
            out = self.softmax(out)
        return out, hidden
</code></pre>
<p>training loop :</p>
<pre><code class="lang-auto">
def train_lstm(model):
    start = time.time()
    criterion = nn.NLLLoss()
    optimizer = torch.optim.Adam(model.parameters())
    n_iters = 20000
    print_every = 1000
    plot_every = 500
    all_losses = []
    total_loss = 0
    for iter in range(1,n_iters+1):
        line = randomChoice(category_line)
        input_line_tensor = inputTensor(line)
        target_line_tensor = targetTensor(line).unsqueeze(-1)
        optimizer.zero_grad()       
        loss = 0
        output, hidden = model(input_line_tensor)
        for i in range(input_line_tensor.size(0)):
            l = criterion(output[i], target_line_tensor[i])
            loss += l
        loss.backward()
        optimizer.step() 
</code></pre>
<p>the sampling function :</p>
<pre><code class="lang-auto">def sample():
    max_length = 20
    input = torch.zeros(1,1,n_letters)
    input[0][0][n_letters - 2] = 1
    output_name = ""
    hidden = (torch.zeros(2,1,lstm.hidden_size),torch.zeros(2,1,lstm.hidden_size)) 
    
    for i in range(max_length):
        output, hidden = lstm(input)
        output = output[-1][:][:]
        l = torch.multinomial(torch.exp(output[0]),num_samples = 1).item()
        if l == n_letters - 1:
            break
        else:
            letter = all_letters[l]
            output_name += letter
        input = inputTensor(letter)
    return output_name
</code></pre>
<p>a typical sample :</p>
<pre><code class="lang-auto">Laurayeerauerararauo
Leayealouododauodouo
Meeaueeulauodalauouo
</code></pre>
<p>Do you know how I can improve that ?</p>
Why GoogLeNet has BatchNorm layers?

username: lugiavn
cooked: <p>2 years ago I was looking for a GoogLeNet model</p><aside class="quote" data-post="1" data-topic="7845">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lugiavn/48/1799_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/where-is-pretrained-googlenet-model/7845">Where is pretrained GoogLeNet model</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hey, I want to ask if anybody has the GoogLeNet model (trained on ImageNet) 
I’m trying to reproduce some deep metric learning result and all papers use GoogLeNet, so for fair comparison and analysis… 
Training the network myself or even converting it from tensorflow is too much work <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title="frowning" alt="frowning" class="emoji">
  </blockquote>
</aside>

<p>Recently somebody has added a GoogLeNet model to torchvision</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/vision/blob/main/torchvision/models/googlenet.py">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/googlenet.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/main/torchvision/models/googlenet.py" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/main/torchvision/models/googlenet.py</a></h4>


      <pre><code class="lang-py">import warnings
from collections import namedtuple
from functools import partial
from typing import Any, Callable, List, Optional, Tuple

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch import Tensor

from ..transforms._presets import ImageClassification
from ..utils import _log_api_usage_once
from ._api import register_model, Weights, WeightsEnum
from ._meta import _IMAGENET_CATEGORIES
from ._utils import _ovewrite_named_param, handle_legacy_interface


__all__ = ["GoogLeNet", "GoogLeNetOutputs", "_GoogLeNetOutputs", "GoogLeNet_Weights", "googlenet"]


</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/googlenet.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
The original paper is cited and also it says there that the model is ported from tensorflow.<br>
I want to check if this really follows the original GoogLeNet, how come it has BatchNorm layers?</p>
How to pass weights parameter inside nn.BCEWithLogitsLoss

username: n0obcoder
cooked: <p>Let’s say i have to train an image classifier an a highly unbalanced dataset. Then I would like to penalize the losses belonging to the dominating classes less and vice versa !</p>
<p>Can you pls show with a few lines of code how exactly  <em>weights</em>  in nn.BCEWithLogitsLossis passed ?</p>
<p>Imagine we have a dataset in which we have three classes with the following number of examples:<br>
classA: 900<br>
classB: 90<br>
classC: 10<br>
now how would you define ur loss function and how would you pass the weights argument?</p>
<p>would it be like<br>
loss_fn = nn.CrossEntropyLoss(weight = [1- 900/1000, 1- 90/1000, 1 - 10/1000]) ???</p>

username: Balamurali_M
cooked: <p>loss_fn = nn.CrossEntropyLoss(weight = [1- 900/1000, 1- 90/1000, 1 - 10/1000])</p>
<p>This will work</p>

username: KFrank
cooked: <p>Hello Gurgaon!</p>
<aside class="quote no-group" data-username="n0obcoder" data-post="1" data-topic="49388" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/n0obcoder/48/16606_2.png" class="avatar"> n0obcoder:</div>
<blockquote>
<p>Imagine we have a dataset in which we have three classes with the following number of examples:<br>
classA: 900<br>
classB: 90<br>
classC: 10<br>
now how would you define ur loss function and how would you pass the weights argument?</p>
<p>would it be like<br>
loss_fn = nn.CrossEntropyLoss(weight = [1- 900/1000, 1- 90/1000, 1 - 10/1000]) ???</p>
</blockquote>
</aside>
<p>You are correct that the named <code>weight</code> argument is the way to<br>
provide class weights to <code>nn.CrossEntropyLoss</code>.</p>
<p>But the weights you use in your example don’t make a lot of<br>
sense to me.  The weights you give for classes B and C are<br>
not very different – they are 0.91 and 0.99, respectively, even<br>
though you have nine times as many class-B samples as<br>
class-C samples.</p>
<p>If you want your weights to fully compensate for the unbalanced<br>
number of samples per class, you would want relative weights of:</p>
<pre data-code-wrap="python"><code class="lang-python">    weight = [ 1/900, 1/90, 1/10 ]
</code></pre>
<p>Furthermore , if you want your weights to have – on average – an<br>
overall scale of 1, you would want:</p>
<pre data-code-wrap="python"><code class="lang-python">    weight = (1000 / 3) * [ 1/900, 1/90, 1/10 ]
</code></pre>
<p>(Also, could you edit the title and body of your post to change<br>
“nn.BCEWithLogitsLoss” to “nn.CrossEntropyLoss” where it<br>
occurs.)</p>
<p>Best regards.</p>
<p>K. Frank</p>
How to send and receive from cuda simultaneously from the same process?

username: pybald
cooked: <p>I’m new to pytorch / cuda … trying to figure this out. Let us say I have two tensors x_gpu and y_gpu. Their corresponding cpu tensors are x_cpu and y_cpu. I’d like to execute this transfer:<br>
x_gpu = x_cpu.cuda() # PCI-e down link<br>
y_cpu = y_gpu.cpu()   # PCI-e up link</p>
<p>It’d be great to have them non-blocking so they can operate in parallel. So, I set non_blocking=True attribute: x_cpu.cuda(non_blocking = True). That didn’t make any difference to the measured bandwidth which is pretty low (some 192 GB/sec). If I only write in one direction, I get 288 GB/sec (which is also really low, another mystery at this point). Thanks!</p>
Draw images in batches without index shuffle

username: Nimrod_Daniel
cooked: <p>I have a dataset class that gets images and ground truth rectangles that draws random images (indexes) in each batch. The thing is, in my case I need to draw the images by the same order in input lists. The following code is using  np.random.permutation in self.index for random draws. I’m not sure how should I change self.index (It appears in <em>init</em> and _next)?<br>
It’s a numpy/data.Dataset question.<br>
p.s - you can ignore all the ops.</p>
<blockquote>
<p>self.index = np.random.permutation(len(self.img_list))</p>
</blockquote>
<pre><code>class RegionDataset(data.Dataset):
def __init__(self, img_list, gt, opts): # seq['image'], seq['gt']
    self.img_list = np.asarray(img_list)
    self.gt = gt

    self.batch_frames = opts['batch_frames'] # 8
    self.batch_pos = opts['batch_pos']
    self.batch_neg = opts['batch_neg']
    self.overlap_pos = opts['overlap_pos']
    self.overlap_neg = opts['overlap_neg']
    self.crop_size = opts['img_size']
    self.padding = opts['padding']
    self.flip = opts.get('flip', False)
    self.rotate = opts.get('rotate', 0)
    self.blur = opts.get('blur', 0)

    self.index = np.random.permutation(len(self.img_list))
    self.pointer = 0

    image = Image.open(self.img_list[0]).convert('RGB')
    self.pos_generator = SampleGenerator('uniform', image.size,
            opts['trans_pos'], opts['scale_pos'])
    self.neg_generator = SampleGenerator('uniform', image.size,
            opts['trans_neg'], opts['scale_neg'])
    
def __iter__(self):
    return self

def __next__(self):
    next_pointer = min(self.pointer + self.batch_frames, len(self.img_list))
    idx = self.index[self.pointer:next_pointer]
    
    if len(idx) &lt; self.batch_frames:
        self.index = np.random.permutation(len(self.img_list))
        next_pointer = self.batch_frames - len(idx)
        idx = np.concatenate((idx, self.index[:next_pointer]))
    self.pointer = next_pointer
    (.....)
</code></pre>
Contrast difference in Autoencoder output images

username: Saikat_Dutta
cooked: <p>I trained an Autoencoder network for images. Input image transform is :</p>
<pre><code class="lang-auto">transforms.Compose([transforms.ToPILImage(),transforms.ToTensor(),
										transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
</code></pre>
<p>I used tanh in final layer and I am using torchvision.utils.save_image with normalize=True, but I observe a contrast difference between original image and generated one. What might be the problem here?</p>
Doubt in calculating output shape of convolution operation

username: ASHUTOSH_CHANDRA
cooked: <p>I’ve 2 doubts in the official formula for calculating shape of output after applying 2d convolutional operation :</p>
<ol>
<li>Why do we add 1 and take floor, why don’t just take ceiling ?</li>
<li>Why do we add 1 in the numerator ?</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p><code>floor</code> + 1 is different to <code>ceil</code> for the case of a zero, which is needed in the calculation:<br>
<code>floor(0) + 1 = 1</code>, while <code>ceil(0) = 0</code></p>
</li>
<li>
<p>We subtract <code>1</code> in the numerator. Have a look at the <a href="http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html#dilated-convolutions" rel="nofollow noopener">Conv arithmetic</a> to derive the formula. Basically if you reformulate the dilated conv output shape formula, you’ll get to the one used in the PyTorch docs.</p>
</li>
</ol>
Any idea on how to slice an (N,D) tensor with minibatches?

username: ZhangTianrong
cooked: <p>I was asked to add a GNN module to an existing intent/slot-tagger co-training network. I was given a bipartite graph bearing connections between intent and slot-tags and was supposed to update the embeddings for intent and slot-tags several times per minibatch. I am not very familiar with Pytorch Geometric and what I am trying to do is to generate new embeddings of all nodes by passing in the same graph every time and slice from it. However, the new embeddings is of size (N,D) and the indices have the size (batch_size, L). How can I generate a (batch_size, L, D) tensor accordingly? May I use the from_pretrained method with freeze=False to manuallt update the embeddings each time?</p>
<p>I am sorry for my wierd discription. The problem is about slicing but I also would like to hear suggestions over my approach of implementation, which I am highly difident.</p>

username: ptrblck
cooked: <p>I’m not sure to understand the question completely, so please correct me, if I’m misunderstanding something.</p>
<p>If you would like to finetune an embedding layer, you can use the <code>from_pretrained</code> method and set <code>freese=False</code>, as you suggested.<br>
The shapes should work out of the box:</p>
<pre><code class="lang-python"># Create embedding
N, D = 10, 100
emb = nn.Embedding(num_embeddings=N, embedding_dim=D)

# Create input
batch_size = 5
L = 7
x = torch.randint(0, N, (batch_size, L))

# Forward pass
output = emb(x)
print(output.shape)  # batch_size, L, D
</code></pre>
Policy gradients, reinforce with baselines loss function

username: mansur
cooked: <p>I am having trouble with the loss function corresponding to the REINFORCE with Baseline algorithm as described in Sutton and Barto book:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9e17eb15999dd87a9845c7f36ef8727970f3c950.png" data-download-href="https://discuss.pytorch.org/uploads/default/9e17eb15999dd87a9845c7f36ef8727970f3c950" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9e17eb15999dd87a9845c7f36ef8727970f3c950.png" alt="image" data-base62-sha1="myyGtkjm3y1ytMGZkGMTdWSc2w8" width="690" height="348" data-dominant-color="DADADA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">907×458 24.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The last line is the update for the policy net.<br>
Let gamma=1 for simplicity…<br>
Now I want to construct loss function for the policy net output, so that I could backpropagate through it after playing one episode.<br>
I am guessing it should be P_loss = - sum_over_episode(log_p*delta). The minus comes from the fact that in pytorch we minimize a loss function, but the update rule in the book tells us how to do gradient ascent for maximizing the objective.<br>
I implemented training at each episode as follows:</p>
<pre><code class="lang-auto"># network
class MLP(nn.Module):
    def __init__(self, input_dim, output_dim, hidden=256):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, hidden)
        self.fc2 = nn.Linear(hidden, hidden)
        self.fc3 = nn.Linear(hidden, output_dim)
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        q = F.relu(self.fc3(x))
        q = F.log_softmax(q, dim=-1)
        return q

def policy_loss(returns, baselines, log_p_taken):
  loss = -torch.sum((returns - baselines)*log_p_taken)
  return loss
</code></pre>
<p>when training:</p>
<pre><code class="lang-auto">for ep in range(N_ep):
  ...
  self.optimizer_p.zero_grad()
  log_P = P_net(ep_states)  # P_net is an instance of MLP net declared above
  log_P_taken = torch.gather(log_P, 1, ep_taken_actions.view(-1, 1)).squeeze()
  P_loss = policy_loss(ep_returns, ep_baselines, log_P_taken)
  P_loss.backward()
  self.optimizer_p.step()
  ...

</code></pre>
<p>However, training seems to never converge with that loss, but trainin with P_loss_negative = - P_loss gives good results. Training with P_loss_negative passes cartpole_v0 and almost passes cartpole_v1, while training with P_loss always diverges.<br>
Please help me find what is wrong. Thank you!!</p>
Flatten before Sigmoid()?

username: oasjd7
cooked: <p>I want to build classification model with global average pooling.<br>
Should I flatten before using <code>Sigmoid()</code> ?</p>
<pre><code class="lang-auto">        self.main = nn.Sequential(
            nn.Conv2d(in_channels=out_channel, out_channels=1, kernel_size=1),
            nn.AvgPool2d(28, 1)
            # flatten -&gt; sigmoid
            # sigmoid
        )
</code></pre>

username: JuanFMontesinos
cooked: <p>Flatten is an elemebt-wise operation so it doesn’t really matter if you just apply spatial transformations afterwards</p>

username: Oli
cooked: <p>As Juan mentioned, you don’t need to flatten before sigmoid. You do need to flatten it so it will fit in the last fully connected layer since you are doing classification</p>
How to load Images in C++ (DataLoader etc.)

username: RuFAI
cooked: <p>Hi, I am searching for a way to load my images as I do it in Python. I just want to instantiate a DataLoader object and give a directory as a parameter. Why is it so complicated in C++? I found some code but it is kind of hard for me to understad it. Here it is…</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;

// You can for example just read your data and directly store it as tensor.
torch::Tensor read_data(const std::string&amp; loc)
{
    torch::Tensor tensor = ...

    // Here you need to get your data.

    return tensor;
};

class MyDataset : public torch::data::Dataset&lt;MyDataset&gt;
{
    private:
        torch::Tensor states_, labels_;

    public:
        explicit MyDataset(const std::string&amp; loc_states, const std::string&amp; loc_labels) 
            : states_(read_data(loc_states)),
              labels_(read_data(loc_labels) {   };

        torch::data::Example&lt;&gt; get(size_t index) override;
};

torch::data::Example&lt;&gt; MyDataset::get(size_t index)
{
    // You may for example also read in a .csv file that stores locations
    // to your data and then read in the data at this step. Be creative.
    return {states_[index], labels_[index]};
} 
</code></pre>
<p>What does this do? <code>torch::data::Example&lt;&gt; get(size_t index) override;</code> and where does the <code>size_t index</code> come from? Is it really necessary to build my own Dataset class? How do I push mit imaga data into the tensor in the <code>read_data</code> Method?</p>
<p>Thank you all</p>

username: dambo
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/mhubii/libtorch_custom_dataset" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/26366414?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/mhubii/libtorch_custom_dataset" target="_blank" rel="nofollow noopener">mhubii/libtorch_custom_dataset</a></h3>

<p>Contribute to mhubii/libtorch_custom_dataset development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: RuFAI
cooked: <p>Thank you for your post. To me it is inacceptable that there is no core functionality for using custom  Datasets as intuatively as in the python version. One more reason for me to go back to Tensorflow. I mean…they want to have lots of users for this framework and build it like ‘We don dont care about any documentation for the c++ version. So please be a pro when you want more than python. And pleas don’t ask the forum because we won’t anwer you’.</p>
<p>Thank you for your help</p>

username: dambo
cooked: <p>I have already complained about this.</p>
pytorch hub PermissionError

username: Quebradawill
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/6389d94666ec38c65fc76ccd2171247fa17e7499.png" data-download-href="https://discuss.pytorch.org/uploads/default/6389d94666ec38c65fc76ccd2171247fa17e7499" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6389d94666ec38c65fc76ccd2171247fa17e7499.png" alt="image" data-base62-sha1="ecyBLrKgvcLB1hkCqEml4loG9CN" width="690" height="325" data-dominant-color="FEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">812×383 21.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Do you get the same error if you set <code>force_reload=True</code>?<br>
Also, could you change the folder for the same of debugging using <a href="https://pytorch.org/docs/stable/hub.html#torch.hub.set_dir" rel="nofollow noopener">torch.hub.set_dir()</a> to some other location and reload the list?</p>

username: Quebradawill
cooked: <p>Thank you for your reply, I tried your advices but get the same errors, what the problems are?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7e74dbf7f43a2b2712613ff61908cee2bde8c5fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/7e74dbf7f43a2b2712613ff61908cee2bde8c5fb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7e74dbf7f43a2b2712613ff61908cee2bde8c5fb_2_416x500.png" alt="image" data-base62-sha1="i2GwVqNB2vovRxHNtbyJ1kxx4Yr" width="416" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7e74dbf7f43a2b2712613ff61908cee2bde8c5fb_2_416x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/7e74dbf7f43a2b2712613ff61908cee2bde8c5fb_2_624x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/7/7e74dbf7f43a2b2712613ff61908cee2bde8c5fb_2_832x1000.png 2x" data-dominant-color="FBEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">840×1008 77.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ailzhang
cooked: <p>Hi this is a known issue that was fixed by <a href="https://github.com/pytorch/pytorch/pull/20782" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/20782</a>. It’s should be available in pytorch nightly.</p>
Model loading error

username: Rajamannar_Krishnamo
cooked: <p>I get this error when I try to load the model the model is getting saved, when I try to load it it’s throwing a error.<br>
The syntax which I used is the one below:</p>
<p><code>model_scratch.load_state_dict(torch.load('saved_models/model_scratch.pt'))</code></p>
<p>and the error which I got is this</p>
<p><code>IncompatibleKeys(missing_keys=[], unexpected_keys=[])</code></p>

username: ptrblck
cooked: <p>This message just states, that no incompatible keys were found, so you are good to go.</p>

username: Rajamannar_Krishnamo
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> I thought its was some error</p>

username: smth
cooked: <p><a class="mention" href="/u/rajamannar_krishnamo">@Rajamannar_Krishnamo</a> dont tag or cross-post. if ptrblck has time he will reply, you dont need to flag it.</p>
Confusing part in torch.nn.conv2d

username: Htut_Lynn_Aung
cooked: <p>Hello guys,</p>
<p>I am now building a feature extraction model using convolutional neural network for the OCR system. While building convolution layers in the model, there is one thing that I don’t understand. As far as my knowledge goes, the output size of the tensor matrix from torch.nn.conv2d is calculated using this equation : (height/width - kernel size) / stride + 1.<br>
<code>input = torch.zeros((1,3,32,320))</code><br>
<code>self.conv = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=2 , bias=False)</code><br>
If I feed the input matrix into above convolution layer, then the calculation would be<br>
new_height = (320 - 3)/2 +1 : 159.5<br>
new_width  = (32 - 3)2 + 1 : 15.5<br>
Since the new height and width are float values, the convolution shouldn’t work but still it returns new height and width values: torch.Size([1, 64, 13, 157])<br>
Can you please explain why it returns such values?</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">import torch
import torchvision
from torch import nn
from config import nh

class Net(nn.Module):
    def __init__(self, in_channels):
        super(Net, self).__init__()
        
        self.conv = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=2 , bias=False)
        # self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

    def forward(self, x):
        x = self.conv(x)
        # x = self.maxpool(x)
        return x

if __name__ == "__main__":
    device = torch.device('cpu')
    a = torch.zeros((1,3,32,320)).to(device)
    net = Net(3)
    net.eval()
    b = net(a)
    print(b.size())
</code></pre>
<p>Sincerely,<br>
Htut Lynn Aung</p>

username: MariosOreo
cooked: <aside class="quote no-group quote-modified" data-username="Htut_Lynn_Aung" data-post="1" data-topic="39060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/htut_lynn_aung/48/6027_2.png" class="avatar"> Htut_Lynn_Aung:</div>
<blockquote>
<p>returns new height and width values: torch.Size([2, 64, 13, 157])</p>
</blockquote>
</aside>
<p>Hi,<br>
Why does <code>batch_size</code> mismatch the input 1-&gt;2?<br>
And I just have a try and it works well, the output.shape is [1, 64, 15, 159]</p>

username: Htut_Lynn_Aung
cooked: <p>Hi <a class="mention" href="/u/mariosoreo">@MariosOreo</a>,</p>
<p>It is a mistake from my part. Yes, the <code>batch_size</code> is 1 in this place.  I will correct the details. But still the output in my side is also <code>torch.Size([1, 64, 15, 159])</code>  which is different from the results gained from the equations. It is not supposed to output this results right? or am I missing something in this equation?  If my knowledge goes correctly, subtraction of kernel size from height or width of the image must be divisible by the stride but in this example, it is not divisible. I did not do any padding here too. That is where I am confused.</p>
<p>sincerely,<br>
Htut Lynn Aung</p>
How to compute l2 norm between every pixel and its 8-pixel neighborhood?

username: dean
cooked: <p>I want to compute a pixel-wise loss by summing up distances to pixels neighbors. Is there a “sliding window” manner to do that with Pytorch?</p>

username: ptrblck
cooked: <p>You could probably use <code>unfold</code> to get the image patches and calculate the loss on these.<br>
Have a look at <a href="https://discuss.pytorch.org/t/how-to-extract-smaller-image-patches-3d/16837/4">this post</a> for an example.</p>

username: dean
cooked: <p>Thank u so much! :))</p>
Problem when installing the latest pytorch using pip3

username: Yuan_Fang
cooked: <p>I was trying to install pytorch:<br>
pip3 install <a href="https://download.pytorch.org/whl/cu100/torch-1.1.0-cp36-cp36m-linux_x86_64.whl" rel="nofollow noopener">https://download.pytorch.org/whl/cu100/torch-1.1.0-cp36-cp36m-linux_x86_64.whl</a></p>
<p>but I got this error. Could anyone help? Thanks!<br>
ERROR: Exception:<br>
Traceback (most recent call last):<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py”, line 178, in main<br>
status = self.run(options, args)<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/commands/install.py”, line 352, in run<br>
resolver.resolve(requirement_set)<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/resolve.py”, line 131, in resolve<br>
self._resolve_one(requirement_set, req)<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/resolve.py”, line 294, in _resolve_one<br>
abstract_dist = self._get_abstract_dist_for(req_to_install)<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/resolve.py”, line 242, in _get_abstract_dist_for<br>
self.require_hashes<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/operations/prepare.py”, line 347, in prepare_linked_requirement<br>
progress_bar=self.progress_bar<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/download.py”, line 886, in unpack_url<br>
progress_bar=progress_bar<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/download.py”, line 750, in unpack_http_url<br>
unpack_file(from_path, location, content_type, link)<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/utils/misc.py”, line 628, in unpack_file<br>
flatten=not filename.endswith(’.whl’)<br>
File “/home/yuan/.local/lib/python3.6/site-packages/pip/_internal/utils/misc.py”, line 524, in unzip_file<br>
shutil.copyfileobj(fp, destfp)<br>
File “/usr/lib/python3.6/shutil.py”, line 79, in copyfileobj<br>
buf = fsrc.read(length)<br>
File “/usr/lib/python3.6/zipfile.py”, line 872, in read<br>
data = self._read1(n)<br>
File “/usr/lib/python3.6/zipfile.py”, line 948, in _read1<br>
data = self._decompressor.decompress(data, n)<br>
zlib.error: Error -3 while decompressing data: invalid stored block lengths</p>

username: Yuan_Fang
cooked: <p>I solved this problem by<br>
sudo pip3 install --no-cache-dir <a href="https://download.pytorch.org/whl/cu100/torch-1.1.0-cp36-cp36m-linux_x86_64.whl" rel="nofollow noopener">https://download.pytorch.org/whl/cu100/torch-1.1.0-cp36-cp36m-linux_x86_64.whl</a></p>
How to load ResNeXt101_64x4d in Pytorch

username: AmardeepGanguly
cooked: <p>How to load models like resnext101_64x4d as whenever i  write</p>
<pre><code class="lang-auto">model=torchvision.models.resnext101_64x4d(pretrained=True,process=True)
</code></pre>
<p>I get an attribute error like torchvision.models has no attribute resnext101_64x4d although the pretrained weights for these models are available,how can i load such a model?</p>

username: ptrblck
cooked: <p>This model doesn’t seem to be listed in <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">torchvision.models</a>.<br>
Are you maybe referring to <a class="mention" href="/u/cadene">@Cadene</a>’s <a href="https://github.com/Cadene/pretrained-models.pytorch" rel="nofollow noopener">repo</a>?</p>

username: AmardeepGanguly
cooked: <p>Yes ,i was wondering whether we can load pretrained models other than the  ones listed in torchvision .models</p>

username: ptrblck
cooked: <p>Sure, once you have the model definition and the checkpoint file, you can load whichever model you like.<br>
I’m not sure, if the linked repo is already integrated into our hub, but you can use it following their instructions in the repo.</p>
Maskrcnn_resnet50_fpn wrapped in DataParallel still using single GPU

username: soulmachine
cooked: <p>I wrap <code>maskrcnn_resnet50_fpn</code> in <code>DataParallel</code> as the following:</p>
<pre><code class="lang-python"># create mask rcnn model
num_classes = 2
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model_ft = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
in_features = model_ft.roi_heads.box_predictor.cls_score.in_features
model_ft.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
in_features_mask = model_ft.roi_heads.mask_predictor.conv5_mask.in_channels
hidden_layer = 256
model_ft.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, num_classes)
model_ft.to(device)

NUM_GPUS = torch.cuda.device_count()
if NUM_GPUS &gt; 1:
    model_ft = torch.nn.DataParallel(model_ft)
model_ft.to(device)
</code></pre>
<p>During the training I see only one GPU is busy and other GPUs are idle and empty.</p>
Extracting weights from trained GRU model

username: dinesh_mareedu
cooked: <pre><code>I am implementing c++ code for GRU.
</code></pre>
<p>Trained using:<br>
nn.GRU(input_size=32, hidden_size=32, num_layers=1, dropout=0, batch_first=True)</p>
<p>For that I have extracted weights from model using model.weight_ih_l0  and model.weight_hh_10. They are coming as a matrix with 96x32. How can I extract reset and update gates weights from them to calculate equations? Hope I get a solution.</p>

username: tom
cooked: <p>The Variables section of the <a href="https://pytorch.org/docs/master/nn.html#torch.nn.GRU" rel="nofollow noopener">GRU documentation</a> has the order relating to the equations at the top.</p>
<p>Best regards</p>
<p>Thomas</p>
GPU cached memory increase when training different models in a loop

username: pytorch_coder
cooked: <p>Hello all,<br>
I’m training several models one after the other and my GPU cached memory is being increased during training of the second model.<br>
Note that when using the same code but train only one model with many epochs this doesnt happen.<br>
So it somehow related to replacing trained model in the GPU.<br>
For simplicity I used the same architecture, optimizer, etc  for all trained model.</p>
<p>My code for is:</p>
<pre><code class="lang-auto">device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
for model_name, training_obj in all_models.items():
    num_epochs = len(training_obj['lr_array'])
    log_message('--------------Training model: ' + model_name + ' -------------------------------')
    optimizer = training_obj['optimizer']
    train_loader = training_obj['train_loader']
    criterion = training_obj['criterion']
    model = training_obj.get('model')
    model.to(device)
    model.train()
    epochs_to_run = range(num_epochs)
    # Loop over epochs to run and train the model
    for epoch in epochs_to_run:
        torch.cuda.empty_cache()
        # Loop over mini batches
        for i_batch, (images, labels) in enumerate(train_loader):
            images = images.to(device)
            labels = labels.to(device)
            # Forward pass
            outputs = model(images).squeeze()
            optimizer.zero_grad()
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
</code></pre>

username: pytorch_coder
cooked: <p>Found the problem.<br>
I had to delete objects of models after training so that GC can clean them before moving to next training next model.</p>
Reproducibility in C++

username: Thomas_Johnson
cooked: <p>The PyTorch docs say that you need to set <code>torch.backends.cudnn.deterministic = True</code> for reproducible runs.</p>
<p>What’s the equivalent setting in the C++ front-end?</p>

username: tom
cooked: <p>Hi Thomas,</p>
<p>This is part of the <code>globalContext</code> (<code>ATen/Context.h</code>): Would <code>at::globalContext().setDeterministicCuDNN(true)</code> help?</p>
<p>Best regards</p>
<p>Thomas</p>
Cifar10 preparing dataset for binary classification

username: Cagri_Kaplan
cooked: <p>Hello, I want to modify labels of CIFAR10 dataset as 0s and 1s.<br>
For a specific class (say cc), I need to assign labels as 1 and 0 for the rest of all and then train the network.</p>
<p>trainset = torchvision.datasets.CIFAR10(root=’./data’, train=True, download=True, transform=transform_train)</p>
<p>transit has 50000 tuples each consists of training image and its label. How can I solve this problem?</p>
Class weights semantic segmentation

username: Nishanth_Sasankan
cooked: <p>Hi Everyone,<br>
I am having a binary classification for an image with 2 labels.  I realized that if I count the number of pixels corresponding to each class, one class has way more pixels compared to the other. I wanted to know how I can find the weights for my image classes and apply them to semantic segmentation. I am pretty new to this, any and all help is greatly appreciated</p>
<p>Thanks<br>
Nishanth</p>
Autograd consumes large GPU memory

username: coder_coder
cooked: <p>Hi all,</p>
<p>I have an equation like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0efc101e788beef8d6e2aac89ccfffc103e0473a.png" alt="1561858456(1)" data-base62-sha1="28yJ4zZjSojgXEAA8ostjEN5P9o" width="393" height="93"><br>
<strong>v</strong>, <strong>h</strong> and <strong>M</strong> are learnable parameters, and the size of <strong>v</strong> is <code>n*k*1</code>. And I implemented this as follows (b is the batch_size):</p>
<pre><code class="lang-auto">(v * x).transpose(-1, -2).sum(dim=1).matmul(( # b*1*k
(h * v).matmul( # b*n*k*1
v.transpose(-1, -2).matmul( # b*n*1*1
M.matmul(embed_deep))) # b*n*k*1
* x).sum(dim=1)  # b*1*1
</code></pre>
<p>However, I found this snippet consumed large GPU memory (e.g., 20G when I set <code>k=512</code>). In my view, this is just some matrix multiplication and summations, how could it possible to use so much GPU memories?<br>
Any help, please.</p>
Changing data drive

username: Tavados
cooked: <p>I noticed that Pytorch is writing and deleting a lot of data to my main drive and I couldn’t find a way, how to change the drive, that the data is written to. Is that even possible?</p>
<p>In my case for each epoch it generates and deletes about 8 GB of data, which can really hurt my SSD’s lifespan when things run for hours and there are multiply multi-hundred epoch trainings daily. I’d like to save that data either in a RAM disk (if there are speed benefits) or to an HDD.</p>
<p>By the way: I can’t make out, why there is even 8 GB of new data generated per epoch, when the dataset is just about 20 MB big in form of about 2500ish images. I do random flips, rotations and crops, but that can’t cause the temporary data to be bigger by a factor of 400, can it?</p>

username: ptrblck
cooked: <p>Are you storing the checkpoints frequently on your drive?<br>
Besides e.g. the pretrained weights from <code>torchvision</code>, PyTorch shouldn’t store anything on the drive, if you have enough RAM.<br>
However, if you are running out of RAM, your OS might use your SSD as the swap file (but again this is not PyTorch-specific).</p>
<p>Could you narrow down, which files are being saved?</p>

username: Tavados
cooked: <p>Weird, it seems to be solved after disabling the swap file mechanism.<br>
I don’t understand however, why a swap file would’ve been created in the first place, when of 32 GB of RAM only 15-22 were used.</p>
<p>Thanks for your help!</p>

username: ptrblck
cooked: <p>The swap won’t necessarily be only used when your RAM is full (at least not on Ubuntu using the default settings).<br>
You can change the tendency of the kernel to use the swap using the <code>swappiness</code> setting as described <a href="https://help.ubuntu.com/community/SwapFaq" rel="nofollow noopener">here</a> (scroll down to swappiness).</p>
<p>These settings might be OS dependent, so I’m not sure how other Linux distributions handle it.</p>
Regarding training using convnet transfer learning

username: vijaytida
cooked: <p>I got this error when adding one layer using classifier1 to pretrained model vgg16 here is the modified code<br>
model_conv = torchvision.models.vgg16(pretrained=True)<br>
for param in model_conv.parameters():<br>
param.requires_grad = False</p>
<p>num_ftrs = model_conv.classifier[6].out_features<br>
-&gt;model_conv.add_module(“classifier1”,nn.ReLU(inplace=True))<br>
-&gt;model_conv.classifier1=nn.Sequential(model_conv.classifier1,nn.Dropout(0.5),nn.Linear(num_ftrs,2))<br>
model_conv.classifier.requires_grad=True<br>
model_conv = model_conv.to(device)<br>
criterion = nn.CrossEntropyLoss()<br>
optimizer_conv = optim.SGD(model_conv.classifier1.parameters(), lr=0.001, momentum=0.9)<br>
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)<br>
model_conv = train_model(model_conv, criterion, optimizer_conv,<br>
exp_lr_scheduler, num_epochs=25)</p>
<p>I got this error while training<br>
Traceback (most recent call last):<br>
File “/home/gputest/PycharmProjects/test1/transfer_learning_tutorial.py”, line 289, in <br>
exp_lr_scheduler, num_epochs=25)<br>
File “/home/gputest/PycharmProjects/test1/transfer_learning_tutorial.py”, line 178, in train_model<br>
loss.backward()<br>
File “/home/gputest/.local/lib/python3.6/site-packages/torch/tensor.py”, line 107, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/gputest/.local/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 93, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>

username: ptrblck
cooked: <p>Answered <a href="https://discuss.pytorch.org/t/model-not-training/5055/12">here</a>.</p>
Better way to find max tensor element

username: NOP
cooked: <p>This will at the very end give me the max element in 3D tensor.<br>
Is there a better way?</p>
<pre><code class="lang-auto">values = torch.randn(32, 32, 32)
values, indices = values.max(0)
print(values, indices)
values, indices = values.max(0)
print(values, indices)
values, indices = values.max(0)
print(values, indices) #tensor(4.0173) tensor(31)
</code></pre>

username: ptrblck
cooked: <p>If you just skip the <code>dim</code> argument, you’ll get the max value:</p>
<pre><code class="lang-python">a = values.max()

# if you need the index also
b = torch.argmax(values)
idx = np.unravel_index(b, values.shape)
</code></pre>
How to accelerate many matrics mulitiplication

username: 082a83b8eb9177051864
cooked: <p>I have to carry out a lot of matrics mulitiplication at same time. It is a part of my own neural network.<br>
On matlab, i can do things like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a097259cbd7af3556c59b8c0437bf1e121a1d193.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a097259cbd7af3556c59b8c0437bf1e121a1d193" title="8AE4B52E-6911-4D75-85F2-6522CC7FD587"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a097259cbd7af3556c59b8c0437bf1e121a1d193_2_690x315.jpeg" alt="8AE4B52E-6911-4D75-85F2-6522CC7FD587" data-base62-sha1="mUEe0YATnhdLmaPYSuuvOYh7lAL" width="690" height="315" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/a097259cbd7af3556c59b8c0437bf1e121a1d193_2_690x315.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/a/a097259cbd7af3556c59b8c0437bf1e121a1d193.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/a097259cbd7af3556c59b8c0437bf1e121a1d193.jpeg 2x" data-dominant-color="F7F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">8AE4B52E-6911-4D75-85F2-6522CC7FD587</span><span class="informations">968×442 46.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I can just use the keyword parfor<br>
However, when it comes to pytorch, anthing becomes different.<br>
I implemented the same code like this:</p>
<pre><code class="lang-auto">    def forward(self, x):
        """
        :param x: train or test with the dimension of [N ,D_in, D_in, num, frame]
        :return:
        """
        for i in range(x.shape[0]):
            for j in range(x.shape[4]):
                for z in range(x.shape[3]):
                    x[i, :, :, z, j] = self.w[:, :, z].mm(x[i, :, :, z, j])
        return x
</code></pre>
<p>It is every slow!!<br>
I cannot come up with any idea to accelerate this process.<br>
Could you give me a hand? I will appreciate it!</p>