akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to apply multiprocessing with cuda?

username: yihe_Young
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89e02bb60faf7718525ee65024301e50875acea3.png" data-download-href="https://discuss.pytorch.org/uploads/default/89e02bb60faf7718525ee65024301e50875acea3" title="TIM截图20200730214049"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89e02bb60faf7718525ee65024301e50875acea3.png" alt="TIM截图20200730214049" data-base62-sha1="jFHHdvgn8uf5L2eoodn3AYLqSYz" width="652" height="500" data-dominant-color="F3F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">TIM截图20200730214049</span><span class="informations">960×736 22.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I want to apply multiprocessing on some simple cuda-related problem as above(Jupyter Notebook). It ought to print out the tensor if everything goes well. However, nothing happens(even without any errors…) after running out all cells. And the ‘traditional’ method-----‘set_start_method(‘spawn’)’ didn’t help… I’m pretty eager to figure out the real problem.</p>
Detnet implementation issues

username: promach
cooked: <p>I am trying to implement <a href="https://arxiv.org/pdf/1804.06215.pdf#page=7" rel="nofollow noopener">detnet backbone</a> using code from a book, but I am bit confused with how the convolution blocks are arranged for each bottleneck…</p>
<p>The random number generator output tensor (<code>input</code>) is of size (1, 1024), not sure what the parameters value 14 are for.</p>
<p>I am also confused with how the <code>input</code> is used to interface with <code>bottleneck_b</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://i.imgur.com/jIfYxrZ.png" title="" rel="nofollow noopener"><img src="https://i.imgur.com/jIfYxrZ.png" alt="" width="550" height="500"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename"></span><span class="informations">977×888</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>detnet.py</strong></p>
<pre><code class="lang-auto">import torch
from detnet_bottleneck import DetBottleneck

# 完成一个Stage 5,即B-A-A的结构,Stage 4输出通道数为1024
bottleneck_b = DetBottleneck(1024, 256, 1, True).cuda()

bottleneck_a1 = DetBottleneck(256, 256).cuda()
bottleneck_a2 = DetBottleneck(256, 256).cuda()

input = torch.randn(1, 1024, 14, 14).cuda()

# 将input作为某一层的特征图,依次传入Bottleneck B、A1与A2三个模块
output1 = bottleneck_b(input)
output2 = bottleneck_a1(output1)
output3 = bottleneck_a2(output2)

print(output1.shape)
print(output2.shape)
print(output3.shape)
</code></pre>
<p><strong>detnet_bottleneck.py</strong></p>
<pre><code class="lang-auto">from torch import nn
class DetBottleneck(nn.Module):
    # 初始化时extra为False时为Bottleneck A,为True时则为Bottleneck B
    def __init__(self, inplanes, planes, stride=1, extra=False):
        super(DetBottleneck, self).__init__()
        # 构建连续3个卷积层的Bottleneck
        self.bottleneck = nn.Sequential
        (
            nn.Conv2d(inplanes, planes, 1, bias=False),
            nn.BatchNorm2d(planes),
            nn.ReLU(inplace=True),
            nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=2, dilation=2, bias=False),
            nn.BatchNorm2d(planes),
            nn.ReLU(inplace=True),
            nn.Conv2d(planes, planes, 1, bias=False),
            nn.BatchNorm2d(planes),
        )

        self.relu = nn.ReLU(inplace=True)
        self.extra = extra
        # Bottleneck B的1×1卷积
        if self.extra:
            self.extra_conv = nn.Sequential(
                nn.Conv2d(inplanes, planes, 1, bias=False),
                nn.BatchNorm2d(planes)
            )

    def forward(self, x):
        # 对于Bottleneck B来讲,需要对恒等映射增加卷积处理,与ResNet类似
        if self.extra:
            identity = self.extra_conv(x)
        else:
            identity = x
        out = self.bottleneck(x)
        out += identity
        out = self.relu(out)
        return out
</code></pre>
Where is momentum in Adam method in the PYTORCH?

username: John1231983
cooked: <p>In the CAFFE, it has<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d058975e25fdf004daa552195ad1cd1b0fc52548.png" data-download-href="https://discuss.pytorch.org/uploads/default/d058975e25fdf004daa552195ad1cd1b0fc52548" title="Screenshot%20from%202018-06-12%2010-39-46"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d058975e25fdf004daa552195ad1cd1b0fc52548_2_690x341.png" alt="Screenshot%20from%202018-06-12%2010-39-46" data-base62-sha1="tJ77quMgOAlxaRv7cyGMrgMshle" width="690" height="341" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d058975e25fdf004daa552195ad1cd1b0fc52548_2_690x341.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d058975e25fdf004daa552195ad1cd1b0fc52548_2_1035x511.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/d058975e25fdf004daa552195ad1cd1b0fc52548.png 2x" data-dominant-color="F3F2EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202018-06-12%2010-39-46</span><span class="informations">1167×578 62.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We can find the setting in <a href="https://github.com/BVLC/caffe/blob/master/examples/mnist/lenet_solver_adam.prototxt#L12" class="inline-onebox" rel="noopener nofollow ugc">caffe/examples/mnist/lenet_solver_adam.prototxt at master · BVLC/caffe · GitHub</a></p>
<p>While in pytorch, the Adam method is</p>
<pre><code class="lang-auto">class torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)
</code></pre>
<p>I did not find any params for momentum. How to set it in pytorch? Thanks</p>

username: John1231983
cooked: <p>Sorry. The last sentence mentioned it. beta is same as momentum in CAFFE</p>

username: crcrpar
cooked: <p>hi</p>
<ul>
<li>
<code>lr</code> → <code>alpha</code>
</li>
<li>
<code>betas</code> → (beta1, beta2)</li>
</ul>

username: John1231983
cooked: <p>Are you sure? So what is means “Caffes uses the value of momentum, momentum2 and delta for beta1,beta2 …” If I want to set momentum as 0.95, what should I do in PYTORCH with Adam method?</p>

username: krioux5
cooked: <aside class="quote no-group" data-username="crcrpar" data-post="3" data-topic="19536">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/crcrpar/48/24766_2.png" class="avatar"> crcrpar:</div>
<blockquote>
<p>→ <code>alpha</code></p>
</blockquote>
</aside>
<p>Any solution here? Is there a momentum equivalent for Adam?</p>
I want to print error messages only once when using multi-GPU

username: mcorange
cooked: <p>Hi everyone, when I train the model with 4 GPUs using pytorch distributed, I see error messages 4 times if there is any, how to make it only appear once? Thanks</p>

username: siheming
cooked: <p>If you start the training using a function that gets a <code>rank</code> argument passed, I used this:</p>
<pre><code class="lang-python">if rank == 0:
    print("error message")
</code></pre>

username: mrshenli
cooked: <p>If you want to suppress both stdout and stderr outputs, you can try this:</p>
<pre><code class="lang-python">import sys, os

f = open(os.devnull, "w")
if rank != 0:
    sys.stdout = f
    sys.stderr = f
</code></pre>

username: mcorange
cooked: <p>Thanks. It is exactly what I wanted.</p>
IndexError: too many indices for tensor of dimension 1

username: be7f984b2f2e66ba7969
cooked: <p>Hi,</p>
<p>this is very weird.</p>
<p>I have two tensors, inds and scores</p>
<p>where inds.ndim=1 and scores.ndim=1</p>
<p>and inds.shape=(28), scores.shape=(300)</p>
<p>the content is:</p>
<pre><code class="lang-auto">inds tensor([  1,   2,  18,   3,   0,  11,  12,  16,  14, 163,  41, 104,  87, 139,
        176,  74, 304, 143, 261, 265, 191, 257, 316, 106, 267, 241, 118, 253],
       device='cuda:0')
</code></pre>
<p>when calling</p>
<p>a=scores[inds]</p>
<p>pytorch would throw</p>
<p>IndexError: too many indices for tensor of dimension 1</p>
<p>but calling</p>
<p>torch.gather(scores, 0, inds) is fine</p>
<p>any idea?</p>

username: be7f984b2f2e66ba7969
cooked: <p>ok. this indeed no pytorch issue. but torch2trt issue (<a href="https://github.com/NVIDIA-AI-IOT/torch2trt/issues/262" rel="nofollow noopener">https://github.com/NVIDIA-AI-IOT/torch2trt/issues/262</a>)</p>
Dimension Error in seq2seq Decorder model.!

username: sgaur
cooked: <p>Hi All,</p>
<p>I am completely noob in NLP domain. I am trying to implement seq2seq model.</p>
<p>Below is the code for LSTM Decorder</p>
<pre><code class="lang-auto">class Decoder(nn.Module):
    def __init__(self, output_dim, emb_dim, hid_dim, n_layers):
        super().__init__()
        
        self.emb_dim = emb_dim
        self.n_layers =  n_layers
        self.hid_dim = hid_dim
        
        self.embed = nn.Embedding(output_dim,emb_dim)
        self.RNN = nn.LSTM(emb_dim,hid_dim,n_layers)
        self.fc = nn.Linear(hid_dim,output_dim)
        
    def forward(self, x,hidden,cell): # 
        
    
        x = x.unsqueeze(0)
        embedding = self.embed(x)
        # Now the output is (1,N,Hidden_siize)
        ## https://buomsoo-kim.github.io/attention/2020/01/25/Attention-mechanism-4.md/
        o,(ht,ct) = self.RNN(embedding,(hidden,cell))
        pred = self.fc(o.squeeze(0))
        return pred,ht,ct
</code></pre>
<p>I like to understand what is the use of <strong>x = x.unsqueeze(0)</strong> and if it should be use it convert 3 dim into 4-dim where nn.LSTM only takes 3-dim.</p>
<p>Sorry for stupid question.!!</p>
How to specify only single module in the Per-parameter options?

username: falmasri
cooked: <p>I have a very nested model with many Prelu. It is not possible to specify all the parameters in the optimizer as a default and respecify all the Prelu as the specific condition.</p>
<p>I want to apply zero weight decay to all Prelu parameters and keep all the rest default.</p>

username: falmasri
cooked: <p>it says in the documentation</p>
<blockquote>
<p>You can still pass options as keyword arguments. They will be used as defaults, in the groups that didn’t override them. This is useful when you only want to vary a single option while keeping all others consistent between parameter groups.</p>
</blockquote>
<p>Which means I can only pass the parameters that I want to make them specific and keep the rest following the main conditions. how this can be applied?</p>

username: tom
cooked: <aside class="quote no-group" data-username="falmasri" data-post="1" data-topic="91008">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/falmasri/48/11265_2.png" class="avatar"> falmasri:</div>
<blockquote>
<p>I want to apply zero weight decay to all Prelu parameters and keep all the rest default.</p>
</blockquote>
</aside>
<aside class="quote no-group">
<blockquote>
<p>Which means I can only pass the parameters that I want to make them specific and keep the rest following the main conditions. how this can be applied?</p>
</blockquote>
</aside>
<p>Just like for the <code>lr</code> in the <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options" rel="noopener nofollow ugc">snippet below the note you cite</a>, you pass <code>weight_decay</code> globally set to what you need and add a <code>weight_decay : 0.0</code> entry to the dictionary for the param group with prelu parameters.</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.nn.Sequential(torch.nn.Linear(3, 4), torch.nn.PReLU())

prelus = {n for n, m in model.named_modules() if isinstance(m, torch.nn.PReLU)}

prelu_param_names = {n for n, _ in model.named_parameters() if n.rsplit('.', 1)[0] in prelus}

torch.optim.SGD([
                {'params': [p for n, p in model.named_parameters() if n not in prelu_param_names]},
                {'params': [p for n, p in model.named_parameters() if n in prelu_param_names], 'weight_decay': 0.0}
            ], lr=1e-2, momentum=0.9, weight_decay=1e-5)
</code></pre>
<p>I’m sure you could make the filtering more terse, but it would seem that names are something easy to debug if things go wrong.</p>
<p>Best regards</p>
<p>Thomas</p>
Tutorial: Object Detection Finetuning

username: maren11
cooked: <p>Hi there,<br>
i am new to Objetct Detection and i tried the PyTorch Turorial Object Detection Finetuning. At the end of the Tutorial I can choose an image of the test dataset and I see the prediction. I checked out some pictures and looked at the prediction with the result that only one of the Person is detected. Is there any possibility to change the code that all person with a score higher than 80% will be shown?</p>
<p>Thanks for your help.</p>

username: tom
cooked: <p>At the end the tutorial uses something like <code>prediction[0]['masks'][0, 0]</code> to get a mask.</p>
<p>So taking the boolean selector tensor <code>preds_to_keep = prediction[0]['scores'] &gt; 0.8</code><br>
and then indexing the masks as <code>prediction[0]['masks'][preds_to_keep, 0]</code>. should do the trix. use <code>.max(dim=0).values</code> or so to take the maximum over all masks.</p>
<p>Best regards</p>
<p>Thomas</p>

username: maren11
cooked: <p>Thanks for help.<br>
If I type this:</p>
<pre><code class="lang-auto">preds_to_keep = prediction[0]['scores'] &gt; 0.8
Image.fromarray(prediction[0]['masks'][preds_to_keep, 0].mul(255).byte().cpu().numpy())
</code></pre>
<p>I get the error message: TypeError: Cannot handle this data type: (1, 1, 512), |u1</p>

username: tom
cooked: <p>I think you need to throw in a <code>.max(dim=0).values</code>  before the <code>mul</code>.</p>

username: maren11
cooked: <p>Thanks, now it works.</p>
Got CUDNN_STATUS_NOT_INITIALIZED when calling backward() through a module written in C++ API

username: wctu
cooked: <p>I got the error msg when calling backward() through a module written in C++ extension.<br>
I wrote the module based on this tutorial<br>
<a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html</a><br>
The module works fine when calling the forward or backward alone.<br>
It also works fine when calling forward along with other torch layers in python.<br>
The following error only occurs when calling backward() and when the module is connected with other layers.</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/home/wctu/Project/semseg/spn/ptsemseg/models/spn_2way.py”, line 554, in <br>
out.backward()<br>
File “/home/wctu/.local/lib/python3.6/site-packages/torch/tensor.py”, line 185, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/wctu/.local/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 127, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED<br>
Exception raised from createCuDNNHandle at /pytorch/aten/src/ATen/cudnn/Handle.cpp:9 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fc019acd1e2 in /home/wctu/.local/lib/python3.6/site-packages/torch/lib/libc10.so)</p>
</blockquote>
<p>According to a previous post, the pytorch installation has already taken care of the cuda, cudnn installation. Does the raised error mean I am using mismatched CUDA driver version?<br>
I am using PyTorch 1.6 and CUDA 10.2 on a Ubuntu 18.04 machine. The CUDA driver is 440.95.01.</p>
<p>Update:<br>
I found the issue has nothing to do with cudnn. It is similar to the post <a href="https://discuss.pytorch.org/t/cudnn-status-not-initialized-error-when-using-batch-size-1/26114">26114</a>, where the issue was raised by wrong indexing in my CUDA kernel. Once I fix the bug, it works fine now.</p>

username: ptrblck
cooked: <p>Good to hear you narrowed down the indexing issue.<br>
For the sake of completeness: due to the asynchronous execution of CUDA kernels, you might see errors from different libs or code places. To get a better stack trace, which should point to the failing operation, you could run the script via: <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code>.</p>
Getting nan gradient with custom loss function

username: kgoyal40
cooked: <p>Hi,</p>
<p>I am creating a custom cross entropy function and the aim to is get the gradients for some model parameters. I get ‘nan’ grad for the parameters. What is incorrect here? Following is the code I am using.</p>
<pre><code class="lang-auto">weigths = an input vector of model parameters
X = A dataset with # features = len(weights) - 1
sumproduct = a function sumproduct(X, W) = w0 + w1x1 + ... wnxn
weight_tensors = torch.tensor(weights, requires_grad=True)
y_hat = [sumproduct(list(X.iloc[i, :]), weight_tensors) for i in range(X.shape[0])]
prob = [1 / (1 + torch.exp(-1 * y_hat[i])) for i in range(len(y_hat))]

loss = -sum([torch.log(prob[i] + torch.exp(torch.tensor([-10], dtype=torch.float32)))
                     if y[i] == 1 else torch.log(1 - prob[i] + torch.exp(torch.tensor([-10], 
                     dtype=torch.float32)))
                     for i in range(len(y))]) / len(y)

loss.register_hook(print)        
loss.backward()
gradients = weight_tensors.grad
print(gradients)
</code></pre>
<p>Thanks</p>

username: ptrblck
cooked: <p>My first guess would be a negative value in <code>torch.log</code> is creating the NaNs.<br>
Could you check your code, if this might be the case?</p>
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:511

username: LZX
cooked: <p>THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument<br>
Traceback (most recent call last):<br>
File “/home/rixi/lzx/WaveGlow/train.py”, line 188, in <br>
train(num_gpus, args.rank, args.group_name, **train_config)<br>
File “/home/rixi/lzx/WaveGlow/train.py”, line 125, in train<br>
outputs = model((mel, audio))<br>
File “/home/rixi/lzx/WaveGlow/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/rixi/lzx/WaveGlow/glow.py”, line 233, in forward<br>
audio, log_det_W = self.convinv<a>k</a><br>
File “/home/rixi/lzx/WaveGlow/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/rixi/lzx/WaveGlow/glow.py”, line 100, in forward<br>
log_det_W = batch_size * n_of_groups * torch.logdet(W)<br>
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:511</p>
<p>CUDA=10.0  TORCH=1.0</p>

username: ptrblck
cooked: <p>Is your code working fine on the CPU?<br>
If that’s the case could you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the complete stack trace here, please?</p>
Using Nvidia tensor core for float mm computation

username: Hongzhang_Shan
cooked: <p>On Nvidia V100, cuda supports to compute the float32 mm computation using tensor cores instead of FP32 cores. Does pytorch support this ? I mean, using tensor core to compute torch.mm for float32 dat type.</p>

username: ngimel
cooked: <p>On v100 cuda (cublas) does not support using tensor cores instead of fp32 cores on fp32 data, so pytorch does not support it either.</p>

username: Hongzhang_Shan
cooked: <p>I read through <a href="https://docs.nvidia.com/cuda/cublas/index.html" rel="nofollow noopener">https://docs.nvidia.com/cuda/cublas/index.html</a> and found<br>
CUBLAS_COMPUTE_32F_FAST_16F is used for<br>
cublasGemmEx and cublasLtMatmul,<br>
“Allows the library to use Tensor Cores with automatic down-conversion and 16-bit half-precision compute for 32-bit input and output matrices.”</p>
<p>Thought the conversion is supported. Maybe I am wrong ? Thanks.</p>
Custom loss with one hot encode

username: Labib_Chowdhury
cooked: <p>Hi all,<br>
I am new to PyTorch. I need some help to implement a custom loss. So, my model output shape is [256,462](256 is minibatch size) and my target shape is [256], where the target is just label encoder. I want to write a function that receives target shape as [256] and convert to [256,462] one hot encode tensor. My number of class is 462. Then I want to do some calculations in GPU.<br>
<strong>How can I convert the target shape of [256] to [256,462] with one hot encoding?</strong></p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>If the target contains the class indices, you could use <code>F.one_hot</code>:</p>
<pre><code class="lang-python">target = torch.randint(0, 462, (256,))
target = F.one_hot(target)
</code></pre>
How to rewrite Dataset to load a pair of data

username: skyraker
cooked: <p>Hi,<br>
The input of my model is sequence data(saved as .txt), and the label is Image(saved as .png). I don’t know how load them at the same time and ensure that they correspond one to one.</p>

username: ptrblck
cooked: <p>It’s a bit hard to help here, as we also don’t know how these files correspond to each other. <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>Generally, the <code>torch.utils.data.Dataset</code> just needs a method to load a single sample.<br>
This can be a plain Python function, which would return tensors.</p>
<p>I would recommend to start with this plain Python method and try to load each sample given an <code>index</code>.<br>
Based on your description I guess you could use the file names to load the pairs or create any lookup table such as a <code>list</code> of <code>tuples</code> containing the file paths to the data and label.<br>
Once your method can load a single sample and return the tensors, you could just paste it into the <code>Dataset.__getitem__</code> method.</p>
Load input images from one folder, and output images from another folder, input-output pair are indicated by filename

username: Stanley_C
cooked: <p>I may not know the correct term, but what is it called when the input/output dataset are in 2 different folders, and each input output pair is linked by name, and how do I load it into pytorch? I plan to load point clouds or images, for either both input and output, so is the usage the same for both point clouds or images?</p>

username: ptrblck
cooked: <p>If the file names of the input and output files use the same file pattern, you could try to sort them and load the pair in a <code>Dataset</code>.<br>
On the other hand, if that’s not possible, I would recommend to create a lookup table (e.g. using a <code>list</code> of <code>tuples</code>, <code>dict</code> or any other container) to get the right file paths for the current <code>index</code> in the <code>__getitem__</code> of your <code>Dataset</code>.</p>
Loss function of auto-encoder is not decreased

username: Ryanz
cooked: <p>I am build an auto-encoder for graph embedding.<br>
This is my autoencoder file: <a href="https://github.com/hxtruong/dynamic-graph-embedding/blob/master/src/static_ge.py" rel="nofollow noopener">https://github.com/hxtruong/dynamic-graph-embedding/blob/master/src/static_ge.py</a></p>
<p>And this is class which is using Autoencoder to build a model and train: <a href="https://github.com/hxtruong/dynamic-graph-embedding/blob/master/src/static_ge.py" rel="nofollow noopener">https://github.com/hxtruong/dynamic-graph-embedding/blob/master/src/static_ge.py</a></p>
<p>I stuck at loss function that doesn’t decrease. I have tried to different learning rate [0.003, 0.001, 0.0005] which seems not improve model.</p>
<p>Note: I am not sure about layer in auto-encoder and loss function as my code are correct.</p>
Package name passed to 'find package handle standard args' (torch) does not match the name of the calling package(Torch)

username: tazz
cooked: <p>I have installed pytorch in virtual environment and am trying to convert a pytorch model to C++ by converting it to TorchScript via tracing and then using cmake to convert the model into C++. When try to run the command<br>
<strong>cmake -DCMAKE_PREFIX_PATH=$PWD/…/libtorch …</strong><br>
I am getting the following error -&gt;</p>
<pre><code class="lang-auto">-- Caffe2: CUDA detected: 10.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda-10.1/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda-10.1
-- Caffe2: Header version is: 10.1
-- Found cuDNN: v7.6.5  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- Autodetected CUDA architecture(s):  6.1
-- Added CUDA NVCC flags for: -gencode;arch=compute_61,code=sm_61
CMake Warning (dev) at /usr/local/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:272 (message):
  The package name passed to `find_package_handle_standard_args` (torch) does
  not match the name of the calling package (Torch).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  /local/mnt2/workspace2/Virtual_Env/venv-glow/lib/python3.7/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:130 (find_package_handle_standard_args)
  CMakeLists.txt:4 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
-- Generating done
</code></pre>
<p>The CMakeLists.txt contains the following code-&gt;</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(Pytorch_test_model)
find_package(Torch REQUIRED)
add_executable(Pytorch_test_model main.cpp)
target_link_libraries(Pytorch_test_model "${TORCH_LIBRARIES}")
set_property(TARGET Pytorch_test_model PROPERTY CXX_STANDARD 11)
</code></pre>
<p>How can I resolve this error?</p>

username: tazz
cooked: <p>There seems to be an issue when you try to compile it with GCC I was able to pass the following flags when passing the cmake commands and able to compile it</p>
<p>cmake -DCMAKE_PREFIX_PATH=$PWD/…/libtorch -DCMAKE_C_COMPILER="/usr/bin/clang" -DCMAKE_CXX_COMPILER="/usr/bin/clang++" … -Wno-dev</p>
<p>Can someone let me know what the issue is?</p>

username: seeker
cooked: <p>For sure the “solution” is not even a real solution. Warning issued when GCC as compiler means there must be something wrong somewhere. I think this should be an issue on github repo if reproduction is possible on the latest release.</p>
Which coreference chains does each sentence relate to in NeuralCoref?

username: Mahsa_Afsharizade
cooked: <p>Hi, I am using <a href="https://github.com/huggingface/neuralcoref" rel="nofollow noopener">neuralcoref</a> for the task of coreference resolution in a text.</p>
<p>I want to know each sentence has mentions from which coreference clusters. For example, sentence1 has mentions from coreference clusters 1, and 4; sentence 2 has mentions from coreference clusters 10 , 14.</p>
<p>How can I do this?</p>
<p>Thanks in advance.</p>
Do evaluation without pausing the main training process

username: Naruto-Sasuke
cooked: <p>Usually, we train model and eval it in the same procedure.</p>
<pre><code class="lang-python">...
for epoch_idx range(max_epoch):
    net.train()
    # do training with batchsize = N
    for train_img, label in train_loader:
        ...
        pred = net(train_img)
        loss = criterion(pred, label)
        optim.zero_grad()
        loss.backward()
        optim.step()
   # eval
   if epoch_idx % eval_step == 0:
       net.eval()
       # do evaluation with batchsize =1
</code></pre>
<p>However, for quite a lot tasks that are favor of cropped images with fixed size when performing training, yet when it comes to evaluation,  each image varies with each other in size and so the batchsize has to be 1.</p>
<p>In my case, the evaluation time is about 10x longer than that of training, which makes the whole procedure slow. The model is large and I do not want to store it for each epoch.</p>
<p>I wonder whether it is practicable: create a new thread and perform the evaluation, so that the main thread continues trianing without suspending.</p>
<p>I am not familar with multi-processing, so any help will be appreciating.</p>
Creating custom point cloud dataset by passing in csv file with path to input point cloud and path to output point cloud

username: Stanley_C
cooked: <p>Currently, I have a csv file, where one column is the input paths to point clouds, and the other column is outputs to points clouds. Here is the dataset: <a href="https://drive.google.com/drive/folders/1HLj8trab5uigxVBu5gzkeWQK1W0HTsG0?usp=sharing" rel="nofollow noopener">https://drive.google.com/drive/folders/1HLj8trab5uigxVBu5gzkeWQK1W0HTsG0?usp=sharing</a>.<br>
I wrote this code to load in the input point cloud and output point cloud for each single pair, but currently, for each pair, the array is overwritten, so I might need to append to the array, instead of overwriting. Each point has its x, y, and z coordinates, which are stored in the 3 separate layers, though, I hope to reshape to work with the data in 3d convolutions.</p>
<pre><code class="lang-auto">import pandas as pd
import numpy as np

df=pd.read_csv('train.csv', sep=',', usecols = ['input','output'])
shape = df.shape

for current in range (shape[0]):	
	input_pc = np.loadtxt(df.iloc[current,0], delimiter=' ')
	print(input_pc.shape)
	output_pc = np.loadtxt(df.iloc[current,1], delimiter=' ')
	print(output_pc.shape)
</code></pre>
<p>Currently, here is my dataloader code:</p>
<pre><code class="lang-auto">train_loader = DataLoader(MyDataset("./train.csv"), batch_size=16, shuffle=False, num_workers=0, worker_init_fn=None)
</code></pre>
<p>How would I adapt my code so that I can load a txt file, convert it to a 3d matrix, which each separate file being a new point cloud, use the csv file to determine the path off input and output, and load it in so that each input output pair is one batch?  Effectively, I’m trying to create a standard autoencoder, but, I’m trying to use point cloud instead of the standard image, I would like to be able to specify a different output file than input file.</p>
Dev Loss decreases along with training loss then it becomes constant

username: Vaibhav_Sundharam
cooked: <p>Hello,<br>
I am working on a computer vision project related to face anti-spoofing. My neural network is inspired by ResNet50 architecture. During training, the Dev loss and training loss decreases but at some point it almost become constant and the plots becomes almost parallel to each other. I have attached the image for reference. I have tried various regularization techniques like weight decay in the optimizer as well as dropouts in the neural network.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/534b254f9d0dd5d2d62b074f81a4904e41470a3d.png" data-download-href="https://discuss.pytorch.org/uploads/default/534b254f9d0dd5d2d62b074f81a4904e41470a3d" title="Replay Attack HTER 4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/534b254f9d0dd5d2d62b074f81a4904e41470a3d_2_675x500.png" alt="Replay Attack HTER 4" data-base62-sha1="bSQD07XN38cqjyCoUYV2ftCnXDn" width="675" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/534b254f9d0dd5d2d62b074f81a4904e41470a3d_2_675x500.png, https://discuss.pytorch.org/uploads/default/original/3X/5/3/534b254f9d0dd5d2d62b074f81a4904e41470a3d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/3/534b254f9d0dd5d2d62b074f81a4904e41470a3d.png 2x" data-dominant-color="FBF9FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Replay Attack HTER 4</span><span class="informations">950×703 56.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am not able to understand the reason behind this behavior and what can I do to overcome this problem?</p>
Torch.clamp() does not work properly

username: YJSNPI
cooked: <p>torch.clamp() does not work properly<br>
The environment I used was google colaboratory</p>
<p>I ran torch.clamp() according to the official documentation<br>
<a href="https://pytorch.org/docs/master/generated/torch.clamp.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.clamp.html</a></p>
<p>working program</p>
<pre><code class="lang-auto">print(torch.__version__)
a = torch.randn(4)
print(a)
torch.clamp(a, min=-0.5, max=0.5)
print(a)
</code></pre>
<p>The results obtained</p>
<pre><code class="lang-auto">1.5.1+cu101
tensor([-0.8323, -0.0443,  0.4641,  0.0824])
tensor([-0.8323, -0.0443,  0.4641,  0.0824])
</code></pre>
<p>If it works like the document, the original result is tensor([-0.5, -0.0443, 0.4641, 0.0824])</p>
<p>What’s wrong with that?<br>
Is this a bug?</p>
<p>I want to set constraints on weight parameters like keras.constraints<br>
I can’t do it because torch.clamp() doesn’t work properly.</p><aside class="quote" data-post="1" data-topic="23620">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/4da419/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/set-constraints-on-parameters-or-layers/23620">Set constraints on parameters or layers</a> 
  </div>
  <blockquote>
    Hi, are there any ways in Pytorch to set the range of parameters or values in each layer? For example, is it able to constrain the range of the linear product Y = WX to [-1, 1]? If not, how about limiting the range of the weight? 
I noticed in Karas, user can define this by setting constraints. Any equivalence in Pytorch? 
Thanks!
  </blockquote>
</aside>


username: sniklaus
cooked: <p>Change</p>
<pre><code class="lang-auto">torch.clamp(a, min=-0.5, max=0.5)
</code></pre>
<p>To</p>
<pre><code class="lang-auto">a = torch.clamp(a, min=-0.5, max=0.5)
</code></pre>
<p>Or</p>
<pre><code class="lang-auto">a = a.clamp(-0.5, 0.5)
</code></pre>
<p>Or</p>
<pre><code class="lang-auto">a.clamp_(-0.5, 0.5)
</code></pre>
Pytorch and cuda not compatible

username: Saifeddine_Barkia
cooked: <p>I have got this error when I tried a code implemented in pytorch, I’m using the GPU mode in colab. What commands should I use to upgrade.<br>
I tried this but didn’t do much:</p>
<pre><code class="lang-auto">!conda install pytorch==1.5.0 torchvision==0.6.0 cudatoolkit=10.2 -c pytorch

</code></pre>
<pre><code class="lang-auto">File "train_bap.py", line 210, in &lt;module&gt;
    train()
  File "train_bap.py", line 145, in train
    train_prec, train_loss = engine.train(state, e)
  File "/content/WS_DAN_PyTorch-master/utils/engine.py", line 41, in train
    target = label.cuda()
  File "/usr/local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 186, in _lazy_init
    _check_driver()
  File "/usr/local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 77, in _check_driver
    of the CUDA driver.""".format(str(torch._C._cuda_getDriverVersion())))
AssertionError: 
The NVIDIA driver on your system is too old (found version 10010).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.
</code></pre>
<pre><code class="lang-auto">import torch 
print(torch.__version__)
1.5.0+cu101

</code></pre>
<p>Here are the version that I’m using , what should I update?</p>
<pre><code class="lang-auto">!nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
</code></pre>
<p>I would appreciate your help ! Thank you</p>

username: michaelklachko
cooked: <p>Choose cudatoolkit=10.1</p>
<p>Also, why are you using Pytorch 1.5?</p>

username: Saifeddine_Barkia
cooked: <aside class="quote no-group" data-username="Saifeddine_Barkia" data-post="1" data-topic="90991">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saifeddine_barkia/48/27005_2.png" class="avatar"> Saifeddine_Barkia:</div>
<blockquote>
<pre><code class="lang-auto">!conda install pytorch==1.5.0 torchvision==0.6.0 cudatoolkit=10.2 -c pytorch
</code></pre>
</blockquote>
</aside>
<p>I don’t know, The github code didn’t provide any requirement , so I used pytorch 1.5. I will try 10.1.</p>

username: michaelklachko
cooked: <p>Don’t use 1.5. Copy the command from <a href="http://pytorch.org" rel="nofollow noopener">pytorch.org</a></p>
What video formats are supported by torchvision.io.read_video?

username: braindotai
cooked: <p>Title is self explanatory… <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=9" title=":expressionless:" class="emoji" alt=":expressionless:">.</p>

username: JuanFMontesinos
cooked: <p>It uses pyav by default as backend (unless you compile from source). Everything is based on ffmpeg so I’d say formats are really extense. I’ve tried myself mkv, avi, mp4 end webm.</p>
Execute a model on CPU and then GPU

username: rahul003
cooked: <p>After moving a model to CPU to execute a step, I would like to move it to GPU for the next steps. But when I do this, the weight remains on CPU. I am using .to() method. I see the error ‘RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same’</p>
<p>What might be going on?</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Hi,<br>
You are most probably just moving the inputs to CUDA but not the model --aka the weights. It might also be that the model initialization is faulty as you have used nn.parameters inside the ‘forward’ function. Can you please edit your original question with the model architectural code and the place where you are sending over the model to CUDA ?</p>
(Computationally) Optimizing highly dynamic graphs

username: Ending_Credits
cooked: <p>Hi all,</p>
<p>I’ve been working on a system for automatically embedding arbitrary JSON with for training with grad decent via backprop through structure. The way it works is by treating JSON as a tree-structure and using a special recursive neural network. However instead of using just a single recursive functions, different functions are applied in different places.</p>
<p>I have this all working (see: <a href="https://github.com/EndingCredits/json2vec" rel="nofollow noopener">https://github.com/EndingCredits/json2vec</a>) and it works pretty well from a model perspective, but understandably it’s very slow. Obviously the basic model is more complicated than a typical MLP, but I supect most of this is due to other overheads.</p>
<p>I was wondering if people with more understanding of the baremetal pytorch would be able to give some pointers for optimising esoteric models. (Of course, if anyone has any other suggestings regarding implementation, that would also be welcome.)</p>
<p>I’ve had a basic poke around with cProfiler, but I didn’t really spot anything big, and I don’t really know what to look for. One thing I did try was move all the raw data into a single tensor, and point the individual operations to different slices, but that didn’t seem to have a huge impact.</p>
Most accurate, quickest, and efficient way to calculate the 3 number mean and std of a dataset for normalization?

username: ProGamerGov
cooked: <p>At the moment I’ve been using a method that I found here: <a href="https://discuss.pytorch.org/t/computing-the-mean-and-std-of-dataset/34949/4" class="inline-onebox">Computing the mean and std of dataset</a></p>
<pre><code class="lang-auto">    mean = 0.0
    for images, _ in loader:
        images = images.view(images.size(0), images.size(1), -1)
        mean += images.mean(2).sum(0)
    mean = mean / len(loader.dataset)

    std = 0.0
    for images, _ in loader:
        images = images.view(images.size(0), images.size(1), -1)
        std += ((images - mean.unsqueeze(1))**2).sum([0,2])
    std = torch.sqrt(std / (len(loader.dataset)*224*224))


    print('\n' + str(round(mean[0].item(), 4)) + ',' + str(round(mean[1].item(), 4)) + ',' + str(round(mean[2].item(), 4)) + \
          '  ' + str(round(std[0].item(), 4)) + ',' + str(round(std[1].item(), 4)) +',' +  str(round(std[2].item(), 4)))
</code></pre>
<p>I also found another slightly different method here that’s quicker than the method above, but it produces a different set of numbers for the dataset STD: <a href="https://stackoverflow.com/questions/60101240/finding-mean-and-standard-deviation-across-image-channels-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/60101240/finding-mean-and-standard-deviation-across-image-channels-pytorch</a></p>
<pre><code class="lang-auto">    nimages = 0
    mean = 0.
    std = 0.
    for batch, _ in loader:
        # Rearrange batch to be the shape of [B, C, W * H]
        batch = batch.view(batch.size(0), batch.size(1), -1)
        # Update total number of images
        nimages += batch.size(0)
        # Compute mean and std here
        mean += batch.mean(2).sum(0) 
        std += batch.std(2).sum(0)

    # Final step
    mean /= nimages
    std /= nimages

    print(mean)
    print(std)
</code></pre>
<p>Which method is more accurate? Is there a better way to perform these calculations?</p>
How do we decide where to add the Dropout layer,Batch Normalization and Activation Layer in CNN?

username: harshini_sewani
cooked: <p>When and where should I use the these layers in my CNN model? How much should be the dropout rate?</p>

username: ptrblck
cooked: <p>I think there is no single best answer to it and you could refer to some “standard” models, e.g. ResNet-like models, or any newer architectures. While the nonlinearity was often applied directly after the conv layers, you will also see some models, where it’s applied after the batchnorm layer.</p>
<p>The drop rate can be treated as a hyperparameter and you chould use the validation loss to tune it.</p>

username: harshini_sewani
cooked: <p>Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Kornia SIFT Descriptor output size not consistent

username: ajitkr1994
cooked: <p>Hey,<br>
I’m using the SIFT descriptor from Kornia.</p>
<p>I should get an output of size 1x128, but I get 1x200, and I’m unable to understand why.</p>
<p>img = torch.randn([1, 1, 10, 10])<br>
sift = kornia.SIFTDescriptor(10, 8, 4)<br>
sift(img).size() -&gt; torch.Size([1, 200])</p>
<p>Can someone please explain?</p>

username: ptrblck
cooked: <p>I’m not sure, why you get this output shape, as the docs explain dim1 should be computed as:</p>
<pre data-code-wrap="python"><code class="lang-python"> num_ang_bins * num_spatial_bins ** 2
</code></pre>
<p>which would be <code>8*4**2=128</code>.</p>
<p>CC <a class="mention" href="/u/edgarriba">@edgarriba</a><br>
It also seems that the inplace docs might not be up to date, as the <code>input</code> is given as:</p>
<blockquote>
<p>Input: (B, 1, num_spatial_bins, num_spatial_bins)</p>
</blockquote>
<p>while I would assume it should be:</p>
<blockquote>
<p>Input: (B, 1, patch_size, patch_size)</p>
</blockquote>

username: ducha-aiki
cooked: <p>Hi, <a class="mention" href="/u/ajitkr1994">@ajitkr1994</a></p>
<p>Sorry for so late response. The problem is that 10 patch size is too small for the even division into 4 spatial bins, therefore the requested descriptor cannot be created.</p>
<p>Addressed in <a href="https://github.com/kornia/kornia/pull/598" rel="nofollow noopener">https://github.com/kornia/kornia/pull/598</a>, where kornia shows an error when requested parameters are not allowed.</p>
Train model components, and reuse them later

username: Odded
cooked: <p>Hi,<br>
Say I have created the above model:</p>
<p>first_seq =  nn.Sequential( do stuff…)<br>
second_seq =  nn.Sequential( do more stuff…)</p>
<p>class My_First_Model (nn.Module):<br>
def <strong>init</strong>(self, first_seq, second_seq ):<br>
super(My_First_Model, self).<strong>init</strong>()</p>
<pre><code>    self.first = first_seq
    self.second = second_seq 
        
def forward(self, x):
    x = self.first(x)
    x = self.second(x)
    return x
</code></pre>
<p>model_1 = My_First_Model()</p>
<h3></h3>
<p>train model</p>
<h3></h3>
<p>No I want to use the models “self.first” in another model, for example:</p>
<p>class My_second_Model (nn.Module):<br>
def <strong>init</strong>(self, input_Sequential):<br>
super(My_second_Model, self).<strong>init</strong>()</p>
<pre><code>    self.first = input_Sequential
    self.second = nn.Sequential( do stuff...)
        
def forward(self, x):
    x = self.first(x)
    x = self.second(x)
    return x
</code></pre>
<p>model_2 = My_second_Model ( model_1.self.first )</p>
<h3></h3>
<p>train model_2</p>
<h3></h3>
<p>The question is: will the input sequential into model_2 (model_1.self.first) will contain the wights learned during the first training? and not the initial weights assigned when defined (first_seq =  nn.Sequential( do stuff…)) ?</p>
<p>I hope my question is understandable,</p>
<p>Thanks!</p>
Overlapping CPU and GPU workload

username: bfeeny
cooked: <p>I am trying to speedup data loading.  Here is a common flow I have found:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader

# some code

loader = DataLoader(your_dataset, ..., pin_memory=True)
data_iter = iter(loader)

next_batch = data_iter.next() # start loading the first batch
next_batch = [ _.cuda(non_blocking=True) for _ in nex_batch ]  # with pin_memory=True and async=True, this will copy data to GPU non blockingly

for i in range(len(loader)):
    batch = next_batch 
    if i + 2 != len(loader): 
        # start copying data of next batch
        next_batch = data_iter.next()
        next_batch = [ _.cuda(non_blocking=True) for _ in next_batch]
</code></pre>
<p>Is there a better way of accomplishing this? It just seems like it could be done more elegantly.</p>
Sliding window inside batch created with DataL batch_first=?

username: pipcia
cooked: <p>I have following data; sequence of 20653 points (time sequence) with 18 futures/labales each.<br>
I created sliding window of size 8 and splitted into test and training.<br>
Received [14386, 8, 18] and [6161,8,18] (X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, random_state=RANDOM_SEED, shuffle=False)<br>
Now decided to use DataLoader to create batches of size 288.<br>
Received 50 tesnors of size [288,8,18] (loader=DataLoader(my_data, batch_size=288, shuffle=False, drop_last=Flase)).<br>
Before creating batches i have been using the following lstm -&gt; self.lstm = nn.LSTM(input_size, hidden_size, num_layers, bidirectional=True, dropout=0.2, batch_first=False) Now my question is, since DataLoader returns batch size as first variable [288,8,18] should i use batch_first=True in lstm. It is not clear to me since i use also sliding window here. Just wanted to add that in cause batch_first=True the network does not learn - no improvement in results…so i will stick to batch_first=false however my understanding is at beast weak</p>