akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why are the dimensions of input/output in lstm oriented in the way they are?

lollicoco: <p>Hi! I’m a newbie and I’m still learning about pytorch. I noticed that the input and output of lstm in pytorch have their dimensions in a specific order (sequence length, batch size, input size). May I ask why it is defined in this way?</p>
<p>I’m more familiar with the conv modules and there the batch size is usually at the beginning (e.g. batch_size x channel x (D) x (H) x W). So it feels a bit unintuitive to me that the orientation is changed for lstm.</p>
<p>I do realize you can set a <code>batch_first</code> argument when setting nn.lstm, but since the default of that arg is false, I’m just wondering if there is any specific reason behind this orientation of dims.</p>

googlebot: <p>RNNs use a loop, unrolling the time dimension, so time-major RNNs process contiguous sequential memory blocks. How important that is, depends on implementation, device and tensor size (i.e. whether it fits in cache).</p>
Pre-load tensors to GPU

AdrianKs: <p>I want to hide the time needed to load tensors from CPU to GPU. This can in general be done by using gpu_tensor = tensor.cuda(non_blocking=True).<br>
This will block as soon as I access the gpu_tensor. But let’s suppose I do some work only after I access the tensor the first time on the GPU. To still be able to hide the CPU-GPU-movement time, I would like to load the tensor one iteration ahead.<br>
Here is a little code example:</p>
<pre><code class="lang-auto">    tensors = [torch.rand((20000, 800), dtype=torch.float32).pin_memory(),
               torch.rand((20000, 800), dtype=torch.float32).pin_memory()]

    gpu_tensors = [torch.rand((20000, 800), dtype=torch.float32, device="cuda"),
                   torch.rand((20000, 800), dtype=torch.float32, device="cuda")]

    if pre_load:
        gpu_tensors[0].copy_(tensors[0], non_blocking=False)
    for batch_index in range(num_batches):
        to_gpu_time -= time.time()
        if pre_load:
            gpu_tensors[(batch_index+1)%2].copy_(tensors[(batch_index+1)%2], non_blocking=True)
        else:
            gpu_tensor = tensors[0].to(device, non_blocking=True)
        to_gpu_time += time.time()

        tensor_access_time -= time.time()
        if pre_load:
            gpu_tensor = gpu_tensors[batch_index%2]
        sum_result = gpu_tensor.sum().item()
        tensor_access_time += time.time()

        # do some work
        total_sleep_2_time -= time.time()
        if sleep_2:
            time.sleep(sleep_time)
        total_sleep_2_time += time.time()

        if write_back:
            write_back_time -= time.time()
            cpu_tensor = gpu_tensor.cpu()
            write_back_time += time.time()
</code></pre>
<p>In theory it should be possible to hide the CPU-GPU-movement time in the sleep time after the tensor access if we are pre-loading one tensor ahead. But unfortunately this seems not to work this way.<br>
Do you have an idea why I am still losing time in the tensor_access part if I pre-load a tensor or do you know how I could solve this problem?</p>

ptrblck: <p>Host to device copies should be executed asynchronous, if the source tensor is pinned.<br>
I would recommend to use e.g. <code>Nsight</code> and inspect the kernel calls visually.</p>
Pooling output of fully connected layers?

Wesley_Neill: <p>Hi there!</p>
<p>I’ve just read the ResNet paper and I decided to implement a version that works on tabular data.</p>
<p>The only hold-up is figuring out how to map the input to a residual block down to the same size as the block’s output. I figured a nice way to do this might be to use some form of pooling. I thought maybe <code>MaxPool1d</code> might do the trick, but I looked at the docs and saw that this is still a pooling method designed for a 3d tensor.</p>
<p>Before I try to go create some silly version of my own, is there anything built into PyTorch that does this for you?</p>

ptrblck: <p>If you want to use the pooling kernel on the feature dimension of the output of the linear layer, you could <code>unsqueeze</code> a channel dimension, apply the pooling, and <code>squeeze</code> the channel dimension again.<br>
Note that the kernel would pool neighboring features. The output activation of linear layers would not contain spatially correlated features, so I would recommend to also test other strategies such as keeping the largest outputs or calculate some form or correlation factor.</p>
<p>Let us know how your experiments went! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Wesley_Neill: <p>Thanks, I actually ended up doing almost exactly that except I used adaptive pooling. Super cool feature!</p>
<p>I’ve posted some very mixed results here: <a href="https://discuss.pytorch.org/t/how-to-use-residual-learning-applied-to-fully-connected-networks/98708" class="inline-onebox">How to use residual learning applied to fully connected networks?</a></p>
<p>It’s not pretty, but maybe a little tweaking will fix it?</p>
Is there an inverse of rnn.pack_sequence?

Trump: <p>When we feed sentences into LSTM</p>
<pre><code class="lang-auto"># Variable length input sequence 'a' (without pad)
# 10 sentences, embedding size 5
a = [torch.randn(random.randint(1, 4), 5), 10]
a = torch.nn.utils.rnn.pack_sequence(a, enforce_sorted=True)
...
# here `out` is packed_sequence..
out, _ = model.lstm(a)

</code></pre>
<p>To unpack <code>out</code>,<br>
Pytorch has <code>torch.nn.utils.rnn.pad_packed_sequence</code> which is an inverse operation to <code>pack_padded_sequence</code>. It gives unpacked and also padded sequences…<br>
sure we can easily remove pads…<br>
but sometimes people want nice and simple way… like an inverse of <code>pack_sequence</code>.<br>
is there anything like this?</p>
<p>thanks</p>

Shandilya21: <p>instead of sorting the sequences : using torch.nn.utils.rnn.pad_packed_sequences<br>
you can try <strong>sorted(key, ‘reverse’)</strong> for sequence sorting without changing it actually.<br>
and you can pass it directly into the input model and no need to reverse it accordingly</p>

shaabhishek: <p>I was faced with this usecase and ended up writing a function to convert a packed sequence into a list of (unpadded) sequences. Sharing it here just in case it’s useful for someone else:</p>
<pre><code class="lang-auto">import torch.nn.utils.rnn as rnnutils

def unpack_sequence(packed_sequence, lengths):
    assert isinstance(packed_sequence, rnnutils.PackedSequence)
    head = 0
    trailing_dims = packed_sequence.data.shape[1:]
    unpacked_sequence = [torch.zeros(l, *trailing_dims) for l in lengths]
    # l_idx - goes from 0 - maxLen-1
    for l_idx, b_size in enumerate(packed_sequence.batch_sizes):
        for b_idx in range(b_size):
            unpacked_sequence[b_idx][l_idx] = packed_sequence.data[head]
            head += 1
    return unpacked_sequence
</code></pre>

braekevelt: <p>I expanded on the code of <a class="mention" href="/u/shaabhishek">@shaabhishek</a> to allow sequences of length zero, but ended up not using it anyway. Hope this might help someone else.</p>
<pre><code class="lang-auto">from typing import List, Tuple
import torch
import torch.nn.utils.rnn as rnn


def pack(sequence: List[torch.Tensor]) -&gt; Tuple[rnn.PackedSequence, List[int]]:
    lengths = list(map(len, sequence))
    tensors = [tensor for length, tensor in zip(lengths, sequence) if length &gt; 0]
    packed_sequence = rnn.pack_sequence(tensors)
    return packed_sequence, lengths


def unpack(packed_sequence: rnn.PackedSequence, lengths: List[int]) -&gt; List[torch.Tensor]:
    trailing_dims = packed_sequence.data.shape[1:]
    unpacked_sequence = []
    idx_map = {}
    head = 0
    for b_idx, length in enumerate(lengths):
        unpacked_sequence.append(torch.zeros(length, *trailing_dims))
        if length &gt; 0:
            idx_map[head] = b_idx
            head += 1
    head = 0
    for l_idx, b_size in enumerate(packed_sequence.batch_sizes):
        for b_idx in range(b_size):
            unpacked_sequence[idx_map[b_idx]][l_idx] = packed_sequence.data[head]
            head += 1
    return unpacked_sequence

</code></pre>
Language Translation Tutorial outdated

petz2000: <p>Hi,</p>
<p>I just wanted to execute the “Language Translation with TorchText” tutorial on Colab. It throws an exception "TypeError: <strong>init</strong>() got an unexpected keyword argument ‘tokenizer_language’ in the second cell (seems the like the Field constructor does not want “tokenizer_language” anymore).</p>
<p>Is there a chance that the tutorial might be updated soon (and where is now the right place to put the language)?</p>
<p>cheers</p>
Does .to(device) detach the tensor

rouniuyizu: <p>I’m still not 100% sure on this, and would like to use following snippet as example:</p>
<p><em>import torch</em><br>
<em>a =  torch.randn(4,5)</em><br>
<em>a.requried_grad = True (just to make it more complicated)</em><br>
<em>b = a.to(“cuda:0”)</em></p>
<p>Is b detached from a assuming they’re not sharing the same memory in nature (one on CPU and another on GPU)? My impression is yes.<br>
If that’s the case, does it save to use following code style:<br>
<em>result1 = net1(a.to(“cuda:0”))</em><br>
<em>result2 = net2(a.to(“cuda:1”))</em><br>
and the backprop won’t mess up because two network inputs are both detached from a?</p>
<p>Also, any API that I could use to check whether two tensors are *tached*?<br>
Thank you,</p>

albanD: <p>Hi,</p>
<p>The <code>Tensor.to()</code> op is <strong>not</strong>  detaching and gradients will flow back to a.</p>
<p>When you do</p>
<pre><code class="lang-auto">result1 = net1(a.to(“cuda:0”))
result2 = net2(a.to(“cuda:1”))
</code></pre>
<p>Then all the gradients will flow back all the way to <code>a</code> properly.</p>
<blockquote>
<p>Also, any API that I could use to check whether two tensors are <em>tached</em>?</p>
</blockquote>
<p>I’m afraid there is not.<br>
You can check by side effect by doing: <code>autograd.grad(out, inp)</code>. This will try to compute the gradient and raise an error if inp is not attached to out.</p>

rouniuyizu: <p>Thanks a lot for this clarification!</p>
Add custom computation to gradient calculation of loss function

Ameet_Rahane: <p>Hi,</p>
<p>I’m trying to modify my loss function to add a custom regulizer on the weights of the neural network. Right now, I just have a series of linear layers, but I want to do elementwise mulitplication and division with the weights of those layers in the loss function and I want the differentiation to account for this. I tried doing this by doing torch.mul(layer.weight.data, constant), but this doesn’t seem to account that computation in the gradient for the weight. How would I do this same manipulation, but have the calculation included in autograd?</p>

albanD: <p>Hi,</p>
<p>You should never use <code>.data</code> when using pytorch anymore.<br>
In this case, it will break the graph and prevent gradient to be properly computed. Just remove the <code>.data</code>.</p>

Ameet_Rahane: <p>This makes me feel stupid, thanks!</p>
Cuda is not available on ipyhon but available on python console

whisht: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aae98acee77c54fca741971751fbef8d7698f1dc.png" data-download-href="https://discuss.pytorch.org/uploads/default/aae98acee77c54fca741971751fbef8d7698f1dc" title="Snipaste_2020-10-09_14-24-13"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aae98acee77c54fca741971751fbef8d7698f1dc.png" alt="Snipaste_2020-10-09_14-24-13" data-base62-sha1="onXwYwvXRc4ex01cnYsviCrcrso" width="690" height="305" data-dominant-color="060606"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Snipaste_2020-10-09_14-24-13</span><span class="informations">1080×478 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am trying to use pytorch with gpu, but it is so weird that torch can detect gpu(cuda) on ipython console, but not on python console.</p>
<p>It seems both of them are using the same python interpreter, but the environment is different.</p>

BramVanroy: <p>I’m not very familiar with ipython/conda but doesn’t it say right there that they are different environments? The first python executable is just in miniconda3 but the second one is in its own /env.</p>

pytorchtester0: <aside class="quote no-group" data-username="whisht" data-post="1" data-topic="98758">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e274bd/48.png" class="avatar"> whisht:</div>
<blockquote>
<p>I am trying to use pytorch with gpu, but it is so weird that torch can detect gpu(cuda) on ipython console, but not on python console.</p>
</blockquote>
</aside>
<p>Which CUDA version are you using?</p>

whisht: <p>CUDA 10.2.<br>
I was figure out why this situation happened. There are two version pytorch installed on my PC, the pytorch  <code>pytorch=1.6.0=py3.8_cuda102_cudnn7_0</code> and <code>pytorch=1.6.0=cpu_py38h538a6d7_0</code> in different conda environment. But I stiil don’t know why this happend when I use different console in the same conda environment. Is there any differernt when using <code>IPython</code> with <code>Python</code> ?</p>

pytorchtester0: <aside class="quote no-group" data-username="whisht" data-post="4" data-topic="98758">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e274bd/48.png" class="avatar"> whisht:</div>
<blockquote>
<p>CUDA 10.2.</p>
</blockquote>
</aside>
<p>One of your environment is running a CPU only version of PyTorch. It seems that Ipython is using it.</p>
Libtorch c++ GPU forward failed, but CPU succeed

asa008: <p>I install pytorch by source compiling to get libtorch on window10, CUDA10.0, VS 2017, RTX 2080.<br>
Thou Libtorch C++ work with cpu successfully, it will crash when libtorch c++ work with GPU.<br>
Actually, it can load model successfully, but it will crash when execute forward(.) function.</p>
<h2>To Reproduce</h2>
<p>compile and execute following codes, the error will occur:</p>
<p><span class="hashtag">#include</span> &lt;torch/script.h&gt; // One-stop header.</p>
<p><span class="hashtag">#include</span> <br>
<span class="hashtag">#include</span> </p>
<p>int main(int argc, const char* argv[]) {</p>
<pre><code class="lang-auto">std::cout &amp;lt;&amp;lt; "ok 1\n";
if (argc != 2) {
std::cerr &amp;lt;&amp;lt; "usage: example-app \n";
return -1;
}
std::cout &amp;lt;&amp;lt; "ok 2\n" &amp;lt;&amp;lt; argv[1] &amp;lt;&amp;lt; std::endl;
// Deserialize the ScriptModule from a file using torch::jit::load().
std::shared_ptrtorch::jit::script::Module module = torch::jit::load(argv[1]);
std::cout &amp;lt;&amp;lt; "ok 3\n";
assert(module != nullptr);
std::cout &amp;lt;&amp;lt; "ok\n";


// Create a vector of inputs.
std::vector&lt;torch::jit::IValue&gt; inputs;

inputs.push_back(torch::ones({ 1, 3, 224, 224 }));
std::cout &lt;&lt; "ok 4\n";
// Execute the model and turn its output into a tensor.
auto output = module-&gt;forward(inputs).toTensor();
std::cout &lt;&lt; "ok 5\n";
std::cout &lt;&lt; output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) &lt;&lt; '\n';

// Create a vector of inputs.
module-&gt;to(at::kCUDA);
std::cout &lt;&lt; "ok 6\n";
std::vector&lt;torch::jit::IValue&gt; inputs2;
inputs2.push_back(torch::ones({ 1, 3, 224, 224 }).to(at::kCUDA));
std::cout &lt;&lt; "ok 7\n";
	// Execute the model and turn its output into a tensor.
auto output2 = module-&gt;forward(inputs2).toTensor();
std::cout &lt;&lt; "ok 8\n";
std::cout &lt;&lt; output2.slice(/*dim=*/1, /*start=*/0, /*end=*/5) &lt;&lt; '\n';

getchar();
</code></pre>
<p>}</p>
<h2>Expected behavior</h2>
<p>it should output printing: “ok 8”</p>
<p>however , “ok 8” didnot appear:</p>
<p>F:\Projects\LibtorchTestPrj\test5\predict\x64\Release<br>
λ .\predict.exe model.pt<br>
ok 1<br>
ok 2<br>
model.pt<br>
ok 3<br>
ok<br>
ok 4<br>
ok 5<br>
-0.4035 0.4926 -0.0028 0.0435 -0.0177<br>
[ Variable[CPUFloatType]{1,5} ]<br>
ok 6<br>
ok 7</p>
<p>F:\Projects\LibtorchTestPrj\test5\predict\x64\Release</p>
<h2>Environment</h2>
<ul>
<li>PyTorch Version (e.g., 1.0): pytorch master 1.0.1</li>
<li>OS (e.g., Linux): win10</li>
<li>How you installed PyTorch ( <code>conda</code> ,  <code>pip</code> , source): source</li>
<li>Build command you used (if compiling from source):</li>
<li>Python version: python 3.6.6</li>
<li>CUDA/cuDNN version: CUDA 10.0</li>
<li>GPU models and configuration: RTX 2080</li>
<li>Any other relevant information: VS 2017</li>
</ul>
<h2>Additional context</h2>
<p>I compiled the newest pytorch source<br>
I use in Release x64 edition<br>
can anybody help me? I am very comfused!<br>
thanks very much!</p>

Jelly123456: <p>I am facing the same problem now. Do you have any solution yet?</p>
Use Feedback to Improve a Model

smi222: <p>I have developed a ML model for a classification (0/1) NLP task and deployed it in production environment. The prediction of the model is displayed to users, and the users have the option to give a feedback (if the prediction was right/wrong).</p>
<p>How can I continuously incorporate this feedback in my model ? From a UX stand point you dont want a user to correct/teach the system more than twice/thrice for a specific input, system shld learn fast i.e. so the feedback shld be incorporated “fast”. (Google priority inbox does this in a seamless way)</p>
<p>How does one build this “feedback loop” using which my system can improve ?</p>
<p>How can  retain the last layer of my model, keeping the weights of all other layers intact?</p>
ONNX export failing with INTERNAL ASSERT failure to do with node->kind()

Peter_Featherstone: <p>I’m trying to ONNX export a yolo model with dynamic input shape while not having to apply grid offsets and anchors post processing. I’m using torch.jit.script to help ONNX export do this.<br>
The following function works fine during normal execution, but fails during ONNX export with the error:</p>
<pre><code class="lang-auto">RuntimeError: !node-&gt;kind().is_aten() &amp;&amp; !node-&gt;kind().is_prim() &amp;&amp; !node-&gt;kind().is_attr() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/serialization/export.cpp":453, please report a bug to PyTorch. 
</code></pre>
<pre><code class="lang-auto">@torch.jit.script
def yolo_predict(pred, anchors, stride: int, scale_x_y: float):
    #transpose
    nA = anchors.size(0)
    nB, nC, nH, nW = pred.size()
    nattr = int(nC / nA)
    pred = pred.view(nB, nA, nattr, nH, nW).permute(0, 1, 3, 4, 2).contiguous()

    #make grid
    grid_x = torch.arange(nW).view(1, 1, 1, nW)
    grid_y = torch.arange(nH).view(1, 1, nH, 1)
    anchor_w = anchors[:, 0].view(1, nA, 1, 1).float() / (nW * stride)
    anchor_h = anchors[:, 1].view(1, nA, 1, 1).float() / (nH * stride)

    #apply grid
    x = pred[..., 0]
    y = pred[..., 1]
    w = pred[..., 2]
    h = pred[..., 3]
    p = pred[..., 4:]
    bx = ((torch.sigmoid(x) - 0.5) * scale_x_y + 0.5 + grid_x) / nW
    by = ((torch.sigmoid(y) - 0.5) * scale_x_y + 0.5 + grid_y) / nH
    bw = torch.exp(w) * anchor_w
    bh = torch.exp(h) * anchor_h
    bp = torch.sigmoid(p)
    preds = torch.stack([bx, by, bw, bh], -1)
    preds = torch.cat([preds, bp], -1)
    preds = preds.view(nB, -1, nattr)

    return preds
</code></pre>
<p>Any ideas what’s causing this error?</p>

ptrblck: <p>Could you create an issue on GitHub please, so that we can track this error?</p>

Peter_Featherstone: <p>I have already, but maintainers are being a bit slow at responding. The issue is <a href="https://github.com/pytorch/pytorch/issues/45816" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/45816</a></p>
Running model on GPU with C++ on Windows

InfiniteLife: <p>I have this code which runs on CPU(if I remove kCUDA device placement).</p>
<blockquote>
<pre><code>torch::NoGradGuard no_grad;
torch::jit::script::Module module;

try {
    // Deserialize the ScriptModule from a file using torch::jit::load().
    module = torch::jit::load(path, torch::kCUDA);
}
catch (const c10::Error&amp; e) {
    std::cerr &lt;&lt; "error loading the model\n";
    return -1;
}

std::cout &lt;&lt; "ok\n";

torch::Tensor tensor = torch::ones({ 1, 3, 384, 640 }).to(at::kCUDA);

// Execute the model and turn its output into a tensor.
auto output = module.forward({ tensor }).toTuple();
</code></pre>
</blockquote>
<p>But with</p>
<blockquote>
<p>torch::kCUDA</p>
</blockquote>
<p>forward pass fails inside get_method in module.h:</p>
<blockquote>
<p>IValue forward(std::vector inputs) {<br>
return get_method(“forward”)(std::move(inputs));<br>
}</p>
</blockquote>
<p>Here is call stack:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97dd111a11b721a72ebdb8561df1c8302ac16cd4.png" data-download-href="https://discuss.pytorch.org/uploads/default/97dd111a11b721a72ebdb8561df1c8302ac16cd4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97dd111a11b721a72ebdb8561df1c8302ac16cd4.png" alt="image" data-base62-sha1="lFrJlOqap1GVjDCLt6aluU5WPHu" width="690" height="127" data-dominant-color="313031"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1309×242 8.75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
What could be the problem here?</p>
<p>CUDA version 10.1</p>
Setting indices to be ignored in new/custom segmentation losses

ash95: <p>I’ve been experimenting with loss functions for semantic segmentation and seem to have hit a wall regarding this. In cross entropy and nll loss implementation, we can set a parameter to ignore a class value. How do we go about doing this for custom loss functions? What’s the prevailing technique?</p>
<p>Do you simply drop the ignore values? In my case, the ignore values are present for padding.</p>
<p>Appreciate any pointers.</p>

ptrblck: <p>You could mask the loss output by multiplying it with zeros for the ignored classes and ones for others.</p>
How to prevent "CUDA Ouft of Memory" when using large batch size?

sofuncheung: <p>Hi there!</p>
<p>I have a question that might be simple to you: when trying to use a very large batch size (say 5000 for cifar10), what can I set in PyTorch in order to prevent the GPU from out of memory?<br>
Like using a dtype that taking less memory or setting smaller num_of_worker?</p>
<p>Cheers!</p>

ptrblck: <p>You could use <a href="https://pytorch.org/docs/stable/amp.html">automatic mixed-precision</a> to use <code>float16</code>, where applicable, or apply <code>torch.utils.checkpoint</code> to trade compute for memory.</p>
<p>Changing the data type manually to <code>float16</code> or another (low-precision) format might work, but you would have to take care of the stability of all operations in your model, e.g. since the values might over-/underflow more easily.</p>
<p>The number of workers in a <code>DataLoader</code> doesn’t change the GPU memory usage.</p>
"RuntimeError: input must have 3 dimensions, got 2" for BiLSTM forward function

kingstontan: <p>Hi all,</p>
<p>This is my first time using PyTorch and would like to train a CNN -&gt; BiLSTM -&gt; CTC output.</p>
<p>Below is my code for the model, is it implemented correctly? I’m getting the following error.</p>
<p><code>RuntimeError: input must have 3 dimensions, got 2</code> for the line <code>out, _ = self.lstm1(out)</code></p>
<p>Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">class ConvNetToBiLSTM(nn.Module):
    def __init__(self):
        super(ConvNetToBiLSTM, self).__init__()
        # 32 by 256
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        # 16 by 128
        self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        # 8 by 64
        self.fc1 = nn.Linear(8 * 64 * 64, 1024)

        self.layer_norm = nn.LayerNorm(1024)
        self.lstm1 = nn.LSTM(input_size=1024, hidden_size=100, num_layers=1, bidirectional=True)
        self.dropout = nn.Dropout()

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)

        out = self.layer_norm(out)
        out = F.gelu(out)
        out, _ = self.lstm1(out)
        out = self.dropout(out)
        return out
</code></pre>

KarthikR: <p>out = out.reshape(out.size(0), -1)   =&gt; You are reshaping here, causing dimensions to reduce.</p>
<p>Try adding a dimension like below but check how your model performs:</p>
<p>out = F.gelu(out)<br>
<strong>out = out.unsqueeze(dim=0)</strong><br>
<strong>print(out.shape)</strong><br>
out, _ = self.lstm1(out)</p>
GAN is generating Bluish images

bobobo-prog: <p>So i started learning how GANS work from the pytorch DCGAN tutorial , i managed to train the model for 300 epochs but after i generate an image with noise its all bluish and not RGB , im really new to this so please help me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b4bcb0084bd6c1d9d47146b1d2ed15ecafcb928.png" alt="image" data-base62-sha1="fjbts3ILCUB2fwgLuoSe90yx9Qk" width="466" height="463"></p>

pytorchtester0: <aside class="quote no-group" data-username="bobobo-prog" data-post="1" data-topic="98704">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bobobo-prog/48/29505_2.png" class="avatar"> bobobo-prog:</div>
<blockquote>
<p>m the pytorch DCGAN tutorial , i managed to train the model for 300 epochs but after i generate an image with noise its all bluish and not RGB , im really new to this so please help me</p>
</blockquote>
</aside>
<p>Hello, Wow indeed that does not seem normal, are you sure you used the Celeb-A Faces dataset which is downloadable in googledrive, as indicated in the DCGAN tutorial?</p>

bobobo-prog: <p>No i used a different dataset this one <a href="https://www.kaggle.com/ikarus777/best-artworks-of-all-time" rel="noopener nofollow ugc">https://www.kaggle.com/ikarus777/best-artworks-of-all-time</a></p>
<p>lmk if u want to see my code</p>
<p>thanks for helping</p>

pytorchtester0: <p>Have you made any adjustments to the original source code for the tutorial?</p>

bobobo-prog: <p>no i havent , i wanted to save every image from the grid in separate files and i somehow ended up with this bluish thing .</p>
Opset_version 1 for ONNX export

americast: <p>Hi everyone,</p>
<p>I wish to export a model to ONNX with opset version 1. It seems Pytorch 1.6.0 supports version 9 only. It would be great if I could be informed which version of Pytorch supports exporting to version 1.</p>
<p>My final aim is to convert the ONNX models to DLC using SNPE. SNPE supports this conversion only for version 1 wrt most common functions (<a href="https://developer.qualcomm.com/docs/snpe/supported_onnx_ops.html" rel="noopener nofollow ugc">https://developer.qualcomm.com/docs/snpe/supported_onnx_ops.html</a>).</p>
<p>Gramercy!</p>
Taking too much training time for CycleGAN

Arpit_Garg: <p>I am trying to reimplement CycleGAN from this GitHub: (<a href="https://github.com/eriklindernoren/PyTorch-GAN#cyclegan" rel="noopener nofollow ugc">https://github.com/eriklindernoren/PyTorch-GAN#cyclegan</a>) but it’s taking showing estimated time in days. I am using GPU, is there any way to optimize this or can get pretrained weights.</p>

ptrblck: <p>Based on <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix">this repository</a> and <a href="https://colab.research.google.com/github/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/CycleGAN.ipynb">this Colab notebook</a> one epoch should take approx. 90 seconds on 4x V100s.<br>
Assuming your GPUs are slower and you are using a single GPU, the estimation of days is still quite large.</p>
<p>You could try to run the notebook on Colab and compare the speed with your local system.<br>
If training on your system is indeed that slow, I would recommend to narrow down the bottleneck and check if it’s might be the data loading (are you storing the data on a local SSD or network drive etc.) or the model itself.</p>
How to program this learning schedule

sahiluppal2k: <p>lr = x^0.5 * min(step_num^0.5, step_num*warmup_steps^-1.5)<br>
where warmup_steps = 5000</p>
<p>I’m unable to formulate this.</p>

ptrblck: <p>Where are you stuck at the moment, i.e. what is not working?<br>
Note that Python uses <code>**</code> to calculate powers, while <code>^</code> is an XOR.</p>

sahiluppal2k: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Let me reframe my question<br>
How to formulate this learning rate</p>
<pre><code class="lang-bash">lrate = d_model ^ 0.5 * min( step_num ^ 0.5, step_num * warmup_steps ^ -1.5)
</code></pre>

ptrblck: <p>Replace thr <code>^</code> operations with <code>**</code> and if all inputs are defined it should work fine.<br>
I don’t know what <code>d_model</code> is but it might be a constant or any other value you could pass to the scheduler.<br>
You should be able to define this schedule in e.g. <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.LambdaLR"><code>lr_scheduler.LambdaLR</code></a>.</p>
I want to know the type of the return value of "bitmapToFloat32Tensor"

ZeTyS_TOMe: <h1>Question about “bitmapToFloat32Tensor” in “pytorch_mobile”.</h1>
<p>The input is a bitmap and the output is a tensor.<br>
I would like to know more about this tensor sequence, which I would like to have machine learning to reason about.</p>
<h2>Idealized array</h2>
<pre><code>                   [ [ [ [blue, blue, blue........blue],
                         [blue, blue, blue........blue],
                                           ~ 
                         [blue, blue, blue........blue],
                       ],
                       [ [green, green, green.......green],
                                            ...
</code></pre>
<p>.shape = [1, 3, height, width]</p>

ptrblck: <p>Based on <a href="https://github.com/pytorch/pytorch/blob/89256611b5ae5ce76b5de83e42fcc2220e762048/android/pytorch_android_torchvision/src/main/java/org/pytorch/torchvision/TensorImageUtils.java#L99-L112">this code</a> it seems the returned tensor would have the shape <code>[1, 3, height, width]</code> and thus the channels are in <code>dim1</code>.</p>

ZeTyS_TOMe: <p>Thank you!!<br>
You’ve solved the problem.</p>
Why I am getting "AttributeError: 'Tensor' object has no attribute 'train_img' "

akib62: <p>I am trying to write a program for <code>MNIST Digit Recognition</code>. I am taking help from this link  <a href="https://www.kaggle.com/franklemuchahary/mnist-digit-recognition-using-pytorch/notebook" rel="noopener nofollow ugc">Kaggle Link</a>.</p>
<p>When I am training my model it is showing <code>AttributeError: 'Tensor' object has no attribute 'train_img'</code> I am getting the error at the time of <code>print</code> However, the code is full the same and I just changed the variable name like <code>train_img</code> instead of <code>data</code>. My code is given below:</p>
<pre><code class="lang-auto">
# module packages
from .. import config
from . import preprocess, my_model

# Loding train Data from preprocess.py
train_loader = preprocess.load_train_data()

# Loding Model from my_model.py
model = my_model.get_model()

# Define a Loss function and optimizer
optimizer = optim.Adam(params=model.parameters(), lr=0.003)
criterion = nn.CrossEntropyLoss()

# Train the network
for epoch in range(config.nb_epocs):
    running_loss = 0.0
    for batch_idx, (train_img, train_labels) in enumerate(train_loader):
        
        # get the inputs; data is a list of [inputs, labels]
        train_img = train_img.unsqueeze(1)
        train_img, train_labels = train_img, train_labels
        
        # zero the parameter gradients
        optimizer.zero_grad()
        
        # forward + backward + optimize
        output = model(train_img)
        loss = criterion(output, train_labels)
        loss.backward()
        optimizer.step()
        
        # print statistics
        running_loss += loss.item()
        if (batch_idx + 1)% 100 == 0:    
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                config.nb_epocs, (batch_idx + 1) * len(train_img), len(train_loader.dataset),
                100. * (batch_idx + 1) / len(train_loader), loss.train_img[0]))
            running_loss = 0.0
print('Finished Training')
</code></pre>
<p>Could you tell me what I have to do?</p>
<p>Another thing is, can I do <code>validation_split</code> at the same time of training like <code>Keras model_fit</code>?</p>

Keyv_Krmn: <p><code>loss</code> doesn’t have any attribute name <code>train_img</code>. If you want to get the value of the loss, simply use <code>loss.item()</code></p>

Keyv_Krmn: <p>For splitting you data into train, validation and test, you can use Dataset and DataLoader. Please see <a href="https://discuss.pytorch.org/t/torch-utils-data-dataset-random-split/32209/3" class="inline-onebox">Torch.utils.data.dataset.random_split</a> for example.</p>

akib62: <p>Thank you Keyv_krmn. I will try this</p>
Pytroch tutorials : From torch.utils.tensorboard import SummaryWriter

784682065: <p>ImportError: cannot import name ‘SummaryWriter’ from ‘torch.utils.tensorboard’ (D:\dev\python\Anaconda3\envs\pytroch\lib\site-packages\torch\utils\tensorboard_<em>init</em>_.py)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea3b240cf7717e76819df2142c8a472e6dfe67e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea3b240cf7717e76819df2142c8a472e6dfe67e0" title="error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea3b240cf7717e76819df2142c8a472e6dfe67e0_2_690x319.png" alt="error" data-base62-sha1="xq6r4J5UXfCutrXqmZdVzswx00E" width="690" height="319" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea3b240cf7717e76819df2142c8a472e6dfe67e0_2_690x319.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea3b240cf7717e76819df2142c8a472e6dfe67e0_2_1035x478.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea3b240cf7717e76819df2142c8a472e6dfe67e0_2_1380x638.png 2x" data-dominant-color="333436"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1894×877 122 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
it’s my environment:</p>
<ol>
<li>python 3.7 ,</li>
<li>pytroch 1.6</li>
<li>tensorboard 2.2.1</li>
</ol>

pascal_notsawo: <p>Here are the versions that work for me:</p>
<pre><code class="lang-auto">torch : 1.6.0
tensorboard : 2.2.1 or 2.3
python : 3.6.9
</code></pre>

784682065: <p>I sitll have some questions , <img src="https://discuss.pytorch.org/images/emoji/apple/sleepy.png?v=9" title=":sleepy:" class="emoji" alt=":sleepy:"></p>

pascal_notsawo: <p>Go ahead. If it’s at my level.</p>

784682065: <p>Thank you very much, I have solved the problem now, although I don’t know the specific reason. I copy my code to other folder, and change num_workers which in testloader =torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=0) .I don’t even know what happened,it’s work. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
Bfloat16 + transformers

Sam_Shleifer: <h3>Context</h3>
<p>In huggingface transformers, the pegasus and t5 models overflow during beam search in half precision.</p>
<p>Models that were originally trained in fairseq work well in half precision, which leads to be believe that  models trained in bfloat16 (on TPUS with tensorflow)  will often fail to generate with less dynamic range.</p>
<p>I was considering starting a project to further train the models with a penalty for having large activations (to discourage overflow in fp16), but was wondering whether this is duplicative with the pytorch team’s current efforts.</p>
<h3>Specific Questions</h3>
<ul>
<li>(a) is the snippet below likely to work in a current nightly version?</li>
<li>(b) are the various kernel implementations “in the works” (and my proposed project won’t be useful in a few months)?</li>
<li>© Is bfloat16 + cuda a possibility?</li>
</ul>
<h3>Failing Snippet</h3>
<p>The following snippet with tries to run a transformer forward pass in bfloat16:</p>
<pre><code class="lang-auto">from transformers import BartForConditionalGeneration
import torch
model = BartForConditionalGeneration.from_pretrained("sshleifer/distilbart-xsum-12-3")
model = model.to(torch.bfloat16)
input_ids = torch.tensor([[0, 31414, 232, 328, 740, 1140, 12695, 69, 46078, 1588, 2]], dtype=torch.long)
model(input_ids)
# RuntimeError: "LayerNormKernelImpl" not implemented for 'BFloat16'
</code></pre>

jerryzh168: <p>Currently quantization supports int8/uint8/int32 and bfloat16 is not covered by pytorch quantization.</p>
<p><a class="mention" href="/u/izdeby">@izdeby</a> can you answer the above questions?</p>
Is there any way I could get bit value of my weight

sunyoung: <p>I quantize my model and I want the bit value of all weights in my model. But I don’t how to do it</p>

supriyar: <p>Could you elaborate on what you mean by bit value of the weights?<br>
If you would like to access quantized weights you can try &lt;quantized_module&gt;.weight(). This returns a quantized tensor of weights and you can use int_repr() to get the int8_t values of weights.</p>

111179: <p>Hi.<br>
Based on your advice, when I try to print the weight of the quantized model, I get FP32 weights, scaling, and zero_point, but I can’t get the int weight?</p>

jerryzh168: <p>yeah you can try:</p>
<pre><code class="lang-python">quantized_weight.int_repr()
</code></pre>
<p>to get the int weight, for more details about quantized tensor apis, please see: <a href="https://github.com/pytorch/pytorch/wiki/Introducing-Quantized-Tensor" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/wiki/Introducing-Quantized-Tensor</a></p>
Problem about training with int8

zfonemore: <p>Hello everyone,</p>
<p>Recently, we are focusing on training with int8, not inference on int8. Considering the numerical limitation of int8, at first we keep all parameters in fp32 and only quantize convolution layer (conduct int8 operation) as it is the most compute-intensive part of a model. During the past months, we have achieved some progress (such accuracy comparable to fp32 training and faster than fp32 training), and our paper is accepted by CVPR2020 (<a href="https://arxiv.org/pdf/1912.12607.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1912.12607.pdf</a>).</p>
<p>Now, we want to further explore quantization of other layers like ReLU, Pooling and BatchNorm, and keep dataflow in int8 in order to save memory.<br>
(1) However, we could not use int8 tensor as input &amp; output of a layer in PyTorch due to the autograd mechanism. Would you consider supporting int8 tensor in the future?<br>
(2) Moreover, we want to pass quantization parameters like scale from layer to layer, but there are some problems at the shortcut connection. During backward, the autograd mechanism will add the gradient from main path and the gradient from the shortcut connection automatically. As we map quantization parameters to tensor, the quantization parameters are lost after the add operation. Could you provide some suggestions?</p>
<p>Many thanks.</p>

hx89: <p>I think there are some work planned for int8 training, cc <a class="mention" href="/u/raghuramank100">@raghuramank100</a> for more details.</p>

Forwil_Yu: <p>hi <a class="mention" href="/u/raghuramank100">@raghuramank100</a> <a class="mention" href="/u/hx89">@hx89</a> , I am also one of the authors of INT8 training paper on CVPR2020.<br>
Firstly, I want to clarify that we are focus on <strong>using INT8 computation to speed up training process</strong> , not for quantization aware training. It means that we need to quantize gradient on convolution backward not only for forward.</p>
<p>we are now achieve nearly no accuracy (&lt; 1% TOP1) decent on ImageNet of ResNet / MobileNet/ Inception/ ShuffleNet, Even on detection task on Pascal VOC and COCO with RetinaNet and FasterRCNN we got only ~1% mAP drop. We can check accuracy table on paper:</p>
<p>(see table 7,8 and 9)</p>
<p>And we also considering about INT8 computation implement and overhead reducing (see Section 3.6. General Purpose Training Framework)</p>
<p>We implement with DP4A on GTX1080TI and finally get 1.6x(forward) and 1.9x(backward) speed on convolution over cuDNN:</p>
<p>(see Figure 8.)</p>
<p>We are now plan INT8 Training V2 about quantization all CNN layer include ReLU, Pooling, Add. But We found pytorch not yet support INT8 gradient backward, So we need Pytorch team to give some help.</p>
<p>Feel great to see your response!</p>

111179: <p>Hi.<br>
We are interested in your research, could you share your code?</p>

jerryzh168: <p>Thanks, quantized training is one of our future directions.</p>
<p>what do you mean by “pytorch not yet support int8 gradient backward”? can you elaborate in more detail?</p>
Not Getting an Expected Error

Paritosh: <p>I am trying a two-way classification using a linear layer: <code>nn.Linear(4096, 2)</code> to get the final predictions; and deliberately using target from 50 classes. I think I should be getting an error when using CrossEntropy loss, since target has classes outside of final linear layer. However, I am not getting the any error. I am using version 1.5.0. Any insights? Am I missing something?</p>
<p>Output (batch-size of 5):</p>
<pre><code class="lang-auto">Target:  tensor([ 6, 12,  8, 29,  3], device='cuda:0')
Pred cls:  tensor([[ 5.7777, 11.3648],
        [ 0.4722,  2.3781],
        [ 2.2436, -1.5575],
        [-4.2130,  1.2372],
        [ 9.2907,  4.8596]], device='cuda:0', grad_fn=&lt;AddmmBackward&gt;)
CE loss:  tensor(1.0909, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
</code></pre>

albanD: <p>Hi,</p>
<p>For performance reasons, we cannot do as many checks on the cuda side as we can do on the cpu side <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
Running this on the CPU should raise a nice error.<br>
Also I think we improved the GPU behavior in master so you can try with a nightly build.</p>

Paritosh: <p>Yes, that’s correct – in CPU mode, it raises an error. If I remember correctly, in version 1.0.0, GPU mode was also raising error, that’s why I was a little concerned. Thank you very much!</p>

ptrblck: <aside class="quote no-group" data-username="Paritosh" data-post="1" data-topic="98411">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/45deac/48.png" class="avatar"> Paritosh:</div>
<blockquote>
<p>I am using version 1.5.0.</p>
</blockquote>
</aside>
<p>PyTorch <code>1.5.0</code> had a bug where device assert statements were ignored. It’s fixed in <code>1.6.0</code> and later, so I recommend to use the latest version.</p>
Network doesn't learn with num_workers=0, works fine otherwise

Stefo: <p>As the title says, I have been going crazy over my simple network not working well and I finally managed to identify the culprit: the num_workers in my DataLoader.<br>
A bit more details: it’s a simple fully connected network that takes 1d inputs and regresses a set of 1d outputs, trained with an MSE loss. In my initial experiments, copying code from my other networks, I used num_workers=4, trained and got decent results. Now one week later I tried replicating those results but setting num_workers=0 since my dataset fits all in RAM and there’s no real need for multiple workers since there is no intensive loading/data augmentation for now. After that change, training didn’t work anymore, the loss behaved like this (edit: every spike in the plot is a new epoch):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e0910f69658face9270fa22c93d56d75f17f3a99.png" alt="Screen Shot 2020-10-06 at 9.44.32 AM" data-base62-sha1="w2BK5Kb5nnl93721qPB3X3GP2I1" width="561" height="288"><br>
while if I set num_workers to 0 the MSE loss behaves much more like you would expect (going down and converging to a low value, can’t attach a second screenshot as a new account).<br>
Furthermore, the average error on my test set after 10 epochs is ~17 (high) for the training with num_workers=0, and ~8 (what I would expect) for num_workers=4. I ran multiple trainings with the same exact hyperparameters, including numpy and torch random seeds, with the only change being the num_workers and results are consistent across runs.<br>
The only thing I can think of right now is some kind of bug somewhere in pytorch, do you guys have any clue about what’s going on? Thank you!</p>

ptrblck: <p>Could you post an executable code snippet so that we could try to reproduce it?</p>
Pytorch Dataloader with variable sequence lengths inputs

Imran_Rashid1: <p>Hi, I am trying to create a Dataloader that takes in variable input lengths however I get an error when doing this for a batch_size greater than 1.</p>
<p>I have heard that there are ways around this using  <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pack_sequence.html#torch.nn.utils.rnn.pack_sequence" rel="noopener nofollow ugc">torch.nn.utils.rnn.pack_sequence</a>  and creating a <code>collate_func</code> however it is unclear to me how this works so I have created a very simple example of a Pytorch <code>Dataset</code> that creates random sequences of variable lengths below.</p>
<p>I am unsure how to fiddle with the collate_func together with the <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pack_sequence.html#torch.nn.utils.rnn.pack_sequence" rel="noopener nofollow ugc">torch.nn.utils.rnn.pack_sequence</a> to create Dataloader that takes accepts variable input lengths.</p>
<p>For clarity, I intend to use this with an LSTM and so the <code>rnn.pack_sequence</code> function looks relevant as well.</p>
<pre><code class="lang-auto">import numpy as np
from numpy.random import rand
from random import randint

import torch
from torch.utils.data import DataLoader, Dataset
</code></pre>
<pre><code class="lang-auto">class SequenceFactory(Dataset):
   """ 
   A Dataset that spits out arrays with a random size
         between 1 and 8. 
   """

    def __init__(self):

        max_len = 8
        no_of_sequences = 100

        #create list of arrays of variable lengths between 1 and 8
        self.data = [rand(randint(1,max_len)) for seq in range(no_of_sequences)] 

    def __getitem__(self, index):

        return self.data[index]

    def __len__(self):
        return len(self.data) #100

data = SequenceFactory()
dataloader = DataLoader(data, batch_size=2,
                        shuffle=True)

next(iter(dataloader))

RuntimeError: stack expects each tensor to be equal size, but got [3] at entry 0 and [7] at entry 1
</code></pre>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/13">This post</a> provides some implementations to deal with variable input shapes. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>