akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom DataLoader using .mat file

Nosin_Ibna_Mahbub: <p>Hello, I am trying to segment medical image and i need help on creating a DataLoader to take into a CNN .mat images. My main image and mask are saved in a same mat file. So, I need help to create custom dataloader to read the main image and mask from the same mat file. Thank you</p>

nivek: <p>Hi, please have a look at our <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html" rel="noopener nofollow ugc">tutorial</a> and let us know what you have tried.</p>

Nosin_Ibna_Mahbub: <p>Thanks for your reply.<br>
I have around 4000 mat files. Each mat file contains main image and mask which is shown in the following image. I have to make custom dataloader for this dataset to train the model.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27cc7a4afbe6786f9be1fb619bbbaf782708001d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/27cc7a4afbe6786f9be1fb619bbbaf782708001d" title="mat_file_read"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27cc7a4afbe6786f9be1fb619bbbaf782708001d_2_690x314.jpeg" alt="mat_file_read" data-base62-sha1="5G4Ic3r3tOG7otElj3kvuzr1TwV" width="690" height="314" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27cc7a4afbe6786f9be1fb619bbbaf782708001d_2_690x314.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/2/7/27cc7a4afbe6786f9be1fb619bbbaf782708001d.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/7/27cc7a4afbe6786f9be1fb619bbbaf782708001d.jpeg 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mat_file_read</span><span class="informations">724×330 50.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Writing a custom <code>Dataset</code> as mentioned by <a class="mention" href="/u/nivek">@nivek</a> sounds like the right approach and you could use e.g. <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html"><code>scipy.io.loadmat</code></a> to load each <code>.mat</code> file.</p>
Does autograd on var give a quadratic runtime?

peppermint: <p><strong>TL;DR</strong>, does computing <code>torch.var(x)</code> on a 1D tensor of n values have a quadratic runtime with autograd? Or does autograd it have a linear runtime in n?</p>
<p><strong>Longer version:</strong><br>
If I have n values in a 1D tensor, and I compute var within a nn.Module, will autograd compute an n by n Jacobian in quadratic runtime?</p>
<p>For example, if we have values<br>
<code>x = torch.tensor([1.,2.,4.,8.,16.,32.,64.], requires_grad=True)</code><br>
then <code>y = torch.var(x)</code> will set <code>grad_fn</code> in <code>y</code>.</p>
<p>The variance can be conceptualized as computing all pairwise distances (with a quadratic number of operations) and shifting and scaling:<br>
<code>torch.sum( (x.view(-1,1) - x.view(1,-1))**2 / (2*len(x)*(len(x)-1)) )</code></p>
<p>Or the varaince can be conceptualized as caching the mean (with a linear number of operations), computing all distances to that mean (also linear) and then scaling the result:<br>
<code>torch.sum( (x - torch.mean(x))**2 / (len(x)-1) )</code></p>
<p>In the first way, it’s clear that all pairs of values in <code>x</code> affect <code>torch.var</code>, and so an n by n matrix will be computed by autograd.</p>
<p>In the second (more standard way), the mean is cached, with autograd depicting a dependency of the mean on each value from <code>x</code>. Then, the mean collides with every value from <code>x</code> to compute the variance. So in this case, it appears the gradients will be quadratic as well. Is that the case?</p>
<p>Also, how is there a way that I could’ve delved into <code>y.grad_fn</code> to determine this for myself without asking and without empirically measuring whether the runtimes seem to grow quadratically? Thank you!</p>

peppermint: <p>Just bumping this in case it got lost during summer holidays. Any thoughts are appreciated!</p>
RuntimeError: Expected to mark a variable ready only once on multi-GPU setting

sajastu: <p>I’m trying to run a Huggingface model on multi-GPU. The problem is that when I’m processing multiple inputs which are bound to each other from a single class (shared-weights), I’m getting <code>RuntimeError: Expected to mark a variable ready only once.</code>. While if I use the module only once, for processing one input, I won’t get this error.</p>
<p>To make it clearer, here is the structure:</p>
<pre><code class="lang-auto">class Model():
    
    def __init__(self, ...)
	   self.encoder = ...


    def forward(input_ids, ...):

	   encoder_outputs = self.encoder(input_ids, ...)

	   # filter encoder_outputs and construct another tensor called 'input_ids_selected'

	   encoder_outputs = self.encoder(input_ids_selected, ...)


	   return encoder_outputs
</code></pre>
<p>If I remove this line: <code>encoder_outputs = self.encoder(input_ids_selected, ...)</code>, I will not run into this error. Should say that to filter encoder_outputs from the first pass of encoder, I’m using other modules (linear layers) to find important <code>input_ids</code>, retaining those in <code>input_ids_selected</code>. You can see this as a two-step summarizer.</p>
What is the proper way to create a tensor from this object?

Alio: <p>When I try putting the entire object into a tensor I seem to be getting different numbers somehow from a list of numbers that should be the same.</p>
<p>I’m using a library called MuJoCo and an object <code>d-&gt;qpos</code>, which I have been running a loop over to get the doubles from. It contains 28 doubles and have been accessing the doubles like so,</p>
<pre><code class="lang-auto">            int i;
            for (i = 0; i &lt; 28; i++)
            {
                std::cout &lt;&lt; d-&gt;qpos[i] &lt;&lt; " ";
            }

This shows the numbers are ...  
</code></pre>
<p>1.57335e-15 0.0776263 0.72953<code> -2.25124e-16 -0.683949 -6.29293e-17 1.34495e-14 -0.486588 4.16118e-15 -0.0461098 0.00221961 -0.0907435 -2.7992 -0.708316 -0.0272932 -0.0461098 0.00221961 -0.0907435 -2.7992 -0.708316 0.0272932 0.514133 -0.646928 -1.57181 -0.514133 0.646928 -1.57181  1.1801e+17</code></p>
<p>When I try putting the whole entire object into a tensor without any indexing, for what should be the exact same sequence of numbers, I get…</p>
<pre><code class="lang-auto">
            torch::Tensor a = torch::from_blob(d-&gt;qpos, {28});  

-1.7520e+00
-1.5310e+24
 2.6897e-02
-1.4977e-36
 1.4053e+00
-8.6043e+18
 1.8074e+00
 5.1051e-21
-2.1511e-02
 3.1887e-37
-1.7960e+00
-1.0073e-10
-1.7839e-02
 1.6544e+38
 3.4738e-02
 7.4359e-09
-1.7433e+00
-1.3105e+24
 2.9631e-02
 4.4223e-22
-1.3094e+00
 1.4495e-36
 8.8353e-01
 4.5674e-36
-1.4315e+00
 5.7768e-37
-2.0999e+00
[ CPUFloatType{28} ]
</code></pre>
<p>Is there a problem with the data type? Should there be another option or is Libtorch not able to grab this entire object?<br>
If I print this <code>d-&gt;qpos</code> object without Libtorch or anything else I get a memory address.</p>
<pre><code class="lang-auto">
std::cout &lt;&lt; d-&gt;qpos;  

0x55c510d887c0
</code></pre>
<p>I think I would rather grab the entire object at once since it would probably be faster than iterating over every number in the object. About that, if that is what I have to do, how should I go about it? Should I make a tensor out of each number and torch::cat them into a tensor? Because</p>

beakin: <p>Since you’re using C++, my guess is that this is a type issue.  <code>from_blob</code> lets you pass an arbitrary pointer, and since you aren’t specifying a data type, it seems to be assuming that it should be interpreted as float (single-precision), but your writeup says the values are actually double-precision.  If that’s the case, I’d change your call to:</p>
<p><code>torch::Tensor a = torch::from_blob(d-&gt;qpos, {28}, torch::kFloat64)</code></p>
<p>and see if that resolves your issue.</p>
Transformers for image prediction

Eugenia_Sol: <p>Hey!</p>
<p>I was wondering if Transformers can be use for image prediction?<br>
For example: you’ve got a sequence of images and you want to predict what the next image in the sequence will be.</p>
<p>I’ve found plenty of examples for image classification (ViT), but not for prediction. (If you have some you’re welcome to share)</p>
<p>Thanks</p>

JuanFMontesinos: <aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/THUDM/CogVideo">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/THUDM/CogVideo" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/d22fcfd774afebb7df9858ea24aefd00703d9fe0ecc0050498f35d84c24a5b55/THUDM/CogVideo" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/THUDM/CogVideo" target="_blank" rel="noopener nofollow ugc">GitHub - THUDM/CogVideo: Text-to-video generation.</a></h3>

  <p>Text-to-video generation. . Contribute to THUDM/CogVideo development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
This is text to video using transformers so I guess you can find something similar in the literature review</p>
Accuracy drop after loading model

A_Ebrahimi: <p>hi i’m training a model for semantic segmentation . in each epoch model and two optimizer are saved and we have all the weights . i’m using resnet50 but each time i stop and continue training accuracy drops . i also don’t change the dataset or batch size and batches are normalized before training. when i continuously trained model , got 85% accuracy but after stop and resuming training accuracy dropped to 70% .</p>

JuanFMontesinos: <p>How are you saving the model?<br>
Note that the optimizers have an state too (which have to be saved)<br>
Otherwise, when you resume the statistics are reset (typically happens with Adam).</p>

A_Ebrahimi: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/574ead555536efbecb3b81b2f52b8c5d63e65611.png" alt="Screenshot from 2022-08-02 18-31-11" data-base62-sha1="csm6uRpDaJDXnE28cmVwIMz8XOp" width="632" height="209"></p>
<p>this is how . can it be because of the file format ? i saw some where that its better to use .pt format instead of .pth for making checkpoints</p>
DDP + Torchdata

kiedanski: <p>Hi,</p>
<p>I have been trying to use torch data + DDP but I am running into the following error:</p>
<p>ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=&lt;torch.utils.data.distributed.DistributedSampler object at 0x7fdcbf3afa30&gt;</p>
<p>What is the correct approach for doing this?</p>

mrshenli: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> and <a class="mention" href="/u/ejguan">@ejguan</a> for dataloader questions</p>

ejguan: <p>Since you are using <code>IterableDataset</code>, you should specify your sharding inside your dataset class rather than using <code>DistributedSampler</code>. You need to shard data based on the worker id and rank in distributed environment.</p>
<pre data-code-wrap="py"><code class="lang-nohighlight">class CustomDS(IterableDataset):
    def __init__(self, ...):
        ..,

    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        worker_id = worker_info.id
        total_workers = worker_info.num_workers
        if dist.is_available() and dist.is_initialized():
            world_size = dist.get_world_size()
            rank = dist.get_rank()
        else:
            world_size = 1
            rank = 0
        total_workers *= world_size
        global_worker_id = worker_id * world_size + rank_id
        for i, d in enumerate(your_data):
            if i % total_workers == global_worker_id:
                yield d
</code></pre>
Issue while tracing LayoutLMv2 model

Ankush2k: <p>Model I am using (UniLM, MiniLM, LayoutLM …): LayoutLMv2</p>
<p>The problem arises when using:</p>
<ul>
<li>the official example scripts:</li>
<li>Following NielsRogge demo for implementation,<br>
The model is working fine when deploying it with normal save and load torch functions but to optimise the inference time trying to compile the model using AWS torch neuron sdk to deploy it over inferentia it is causing issue as the compilation of model uses torch.jit.trace() function to compile the model.<br>
When tried implement it over normal torch environment still facing the same issue.</li>
</ul>
<p><strong>To Reproduce</strong><br>
Steps to reproduce the behavior:</p>
<ol>
<li>
<p>Following NielsRogge demo for implementation till the encoded inputs generated in inference, <a href="https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD.ipynb" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a> .</p>
</li>
<li>
<p>Add the below code after encoded inputs are saved.<br>
<a href="https://user-images.githubusercontent.com/34794620/182364303-8f933510-72dc-4476-b77d-f7752e4c4632.png" rel="noopener nofollow ugc"><img src="https://user-images.githubusercontent.com/34794620/182364303-8f933510-72dc-4476-b77d-f7752e4c4632.png" alt="image" width="690" height="252"></a></p>
</li>
</ol>
<p><strong>Expected behavior</strong><br>
A traced model should be created for storing and deploying it to AWS inferentia machine for compilation.</p>
<ul>
<li>Platform:</li>
<li>Python version: Python 3.7.13</li>
<li>PyTorch version (GPU?):1.12.0+cu113 , GPU - Yes</li>
</ul>
<p>If there is any other information required, please drop a note.</p>
<p>Thanks</p>
Convolutional AE always overfitting time series - what's wrong?

Chai: <p>Hi there,</p>
<p>I’ve build a CAE for anomaly detection in time series, but it is always overfitting. I’ve tried data augmentation, short/long inputvector, dropout rates… I don’t know what I’m doing wrong, may be you can give me a hint?</p>
<p>This is my architecture</p>
<pre><code class="lang-auto">class CAE(nn.Module):
    def __init__(self):
        super(CAE, self).__init__()

        self.encoder = Encoder()
        self.decoder = Decoder()

        self.encoder.to(get_device())
        self.decoder.to(get_device())

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()

        self.conv_enc = nn.Sequential(
            nn.AlphaDropout(p=0.5),
            nn.Conv1d(
                in_channels=1,
                out_channels=16,
                kernel_size=5,
                stride=2,
                padding=0
            ),
            nn.SELU(),
            nn.AlphaDropout(p=0.5),
            nn.Conv1d(
                in_channels=16,
                out_channels=8,
                kernel_size=5,
                stride=2,
                padding=0
            ),
            nn.SELU()
        )

        self.conv_enc.apply(self.init_weights)
            
    def init_weights(self, m):
        if type(m) == nn.Conv1d or type(m) == nn.ConvTranspose1d:
            torch.nn.init.xavier_normal_(m.weight)
            m.bias.data.fill_(0.01)

    def forward(self, x):
        x = self.conv_enc(x)
        return x

class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()

        self.conv_dec = nn.Sequential(
             nn.ConvTranspose1d(
                in_channels=8,
                out_channels=16,
                kernel_size=5,
                stride=2,
                padding=0
            ),
            nn.SELU(),
            #nn.AlphaDropout(p=0.5),
            nn.ConvTranspose1d(
                in_channels=16,
                out_channels=1,
                kernel_size=6,
                stride=2,
                padding=0
            )
        )

        self.conv_dec.apply(self.init_weights)
            
    def init_weights(self, m):
        if type(m) == nn.Conv1d or type(m) == nn.ConvTranspose1d:
            torch.nn.init.xavier_normal_(m.weight)
            m.bias.data.fill_(0.01)

    def forward(self, x):
        x = self.conv_dec(x)
        return x
</code></pre>
<p>I’m using following setup</p>
<ul>
<li>optimizer - AdamW with lr=0.0001</li>
<li>scheduler - OneCyleLR</li>
<li>Normalized Data with MinMaxScaler()</li>
<li>loss - L1Loss</li>
<li>batchsize - 250</li>
<li>trainingsize - 12000 sample</li>
<li>inputsize - 800</li>
</ul>
<p>Thank you guys</p>

jayz: <p>Probably weigh too many pars; decrease channels, also you could add weight decay. Try a much larger dataset also. What kind of data aug did you try? What is long short vector?</p>
Facing performance issue for large batch of small matrix multiplication

Aaron_Black: <p>I am recently facing performance issue for large batch of small matrix multiplication using PyTorch.  The running time of the following scripts seems not affected by the channel number. Is there any hint to optimize the running time ?</p>
<p>I have try torch.matmul, torch.bmm, torch.einsum, all the them give the similar results. Would custom cuda kernel help ?</p>
<pre><code class="lang-python">import torch
import torch.cuda
import torch.nn

C = 16
# x = torch.randn((4, 32, 31, 64, 64)).cuda()
x = torch.randn((64*64*4, 31, C)).cuda()
# x = torch.randn((4, 64, 64, 31, 32)).cuda()

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

steps = int(1e3)
start.record()
for j in range(steps):
    # w1 = torch.matmul(x, x.transpose(-2,-1))
    w2 = torch.bmm(x, x.transpose(-2,-1))
    # w = torch.einsum('bhwqc,bhwkc-&gt;bhwqk', x, x)
    # print(torch.allclose(w1,w2))
    # torch.einsum("bhqd,bhkd-&gt;bhqk", test_mat, test_mat)
end.record()

torch.cuda.synchronize()

print('Time: {} ms'.format(start.elapsed_time(end)/steps))
print(torch.cuda.memory_allocated()/(1024))
</code></pre>
<pre><code class="lang-auto">c=16
Time: 1.3880045166015624 ms
94272.0
c=8
Time: 1.3616466064453125 ms
77888.0
c=2
Time: 1.278853759765625 ms
65472.0
</code></pre>

JuanFMontesinos: <p>There is no relevant increment in the amount of channels from 2 to 16.</p>
<pre><code class="lang-auto">import torch.nn

C = 1000

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

steps = int(100)
with torch.no_grad():
    for C in [10, 100, 1000]:
        x = torch.ones((20000, 31, C), device='cuda:0')
        y = x.transpose(-2, -1)
        start.record()
        for j in range(steps):
            w2 = torch.bmm(x,y )
        end.record()

        torch.cuda.synchronize()
        print(f'Channels: {C}')
        print('\t Time: {} ms'.format(start.elapsed_time(end) / steps))
</code></pre>
<pre><code class="lang-auto">Channels: 10
	 Time: 1.2575846099853516 ms
Channels: 100
	 Time: 1.9832627868652344 ms
Channels: 1000
	 Time: 9.910529174804687 ms
</code></pre>
<p>Also, you need to check the workload of your GPU is 100%<br>
Lastly, your script measures the time it requires to transpose.</p>
Multiprocessing of models in C++

Johnny_Cash: <p>HI, I’m quite new to C++, but would like to rewrite my python code to C++, because of speed and better deployment.</p>
<p>Now what I do in Python I use multiprocessing, each process holds a model and I do inference on each model for the images that are loaded into a queue.</p>
<p>My question: can you guide me to how I would do that in C++ with Pytorch? What would be the best practices? Would multithreading be an option? Any pointers are appreciated!</p>
The compatibility of libtorch with different graphics cards

Exlunear: <p>I use libtorch load the .pt file which pytorch have saved. The previous program worked well on gtx1650, gtx1050ti, titanblack and rtx4000. With cuda10.0, pytorch1.1.0 and libtorch1.1.0. But when I change the card to the newest rtx a4000, the program start to have problems in loading module and the speed of computation  is extremely slow. Should I upgrade the version of cuda and pytorch?</p>

ptrblck: <p>Yes, your Ampere GPU needs CUDA&gt;=11.0 and should raise an error if you are trying to run it with CUDA10.0.</p>

Exlunear: <p>Thanks, I’ll try it.</p>
Why this simple one line code allocates 2GB of memory on the GPU?

Alon_Lekhtman: <p>I ran <strong>nvidia-smi</strong> before running any code, and it showed only xorg with 4MiB ( can’t attach print screen because I am new in this forum)</p>
<p>I wrote this code in a clean jupyter notebook, with new kernel:</p>
<blockquote>
<p>import torch<br>
a = torch.randn((1), device=torch.device(“cuda”), dtype=torch.float)</p>
</blockquote>
<p>After running <strong>nvidia-smi</strong> this is the result I got:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afe8a5c056bcfdf0ab03b8af88fa85b008b6906e.png" data-download-href="https://discuss.pytorch.org/uploads/default/afe8a5c056bcfdf0ab03b8af88fa85b008b6906e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afe8a5c056bcfdf0ab03b8af88fa85b008b6906e.png" alt="image" data-base62-sha1="p6a08yuT4GKJHnovZWAVSizXVPo" width="690" height="320" data-dominant-color="431935"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">801×372 37.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why does this code take 2GB of memory?</p>
<p>I suppose it is something pytorch does when initialising, but, is there a way to reduce this?</p>

Aayush_Garg: <p>I think there is another process that is taking the memory. Can you kill your jupyter kernel and check GPU usage?</p>

ptrblck: <p>The first CUDA operation will initialize the CUDA context, which will load the driver, all native PyTorch kernels, all kernels from linked libraries (cublas, cuDNN etc.) and will thus add this overhead.<br>
You could reduce it by enabling CUDA’s lazy loading functionality in CUDA 11.7. For now you could manually build it from source or wait until we publish the binaries as nightly builds soon (we are already building them, but are going through some more testing).</p>

Alon_Lekhtman: <p>Only the Xorg is left.</p>

Alon_Lekhtman: <p>Another person I asked to run the same code, showed only 500M of GPU memory for the process. Do you know what may be the difference? Why it is 4 times lower on his side?</p>

ptrblck: <p>The context size depends on the used library versions, CUDA/driver version, compute architecture, number of SMs on your GPU etc., so it won’t be constant.</p>
Can we allocate several torch.tensor continuously?

Ziyu_Huang: <p>Hi! I am trying to allocate several torch.tensor continuously, can we somehow do that? Thank you!!!</p>
<p>Well…their size are not equal…Allocate one big tensor and split it might be…not very practical…Otherwise maybe I can create a (1, m) and then split it and reshape each one??? Maybe…but dirty…</p>
<p>Thank you!!!</p>

ptrblck: <p>Allocating a large tensor and then slicing it, might work. However, what’s your use case and why do you depend on this memory allocation?</p>

Ziyu_Huang: <p>yes…I can …Well, I am developing CUDA program. Anyway, your way is OK…Thank you!!</p>
How many ConvLSTM cells are instantiated?

vk12: <p>Hello. I have defined the following ConvLSTM class, code borrowed from here (<a href="https://github.com/sladewinter/ConvLSTM" class="inline-onebox" rel="noopener nofollow ugc">GitHub - sladewinter/ConvLSTM</a>) and I’ve skipped some boilerplate initializations.</p>
<pre><code class="lang-auto">class ConvLSTM(nn.Module):

    def __init__(self, in_channels, out_channels, 
    kernel_size, padding, lstm_activation, frame_size):

        super(ConvLSTM, self).__init__()

        self.out_channels = out_channels

        # We will unroll this over time steps
        self.convLSTMcell = ConvLSTMCell(in_channels, out_channels, kernel_size,
            padding, lstm_activation,  frame_size)


    def forward(self, X):
        # Get the dimensions
        batch_size, _, seq_len, height, width = X.size()
        ...
        # Unroll over time steps
        for time_step in range(seq_len):

            H, C = self.convLSTMcell(X[:,:,time_step], H, C)    
        
            output[:,:,time_step] = H
     
        return output, H, C
</code></pre>
<p>If I view the summary of this model using torchinfo, it shows trainable parameters only on the first ConvLSTM Cell and ‘(recursive)’ in the rest. However, if I print the model parameters using</p>
<pre><code class="lang-auto">print(sum(p.numel() for p in t.parameters() if p.requires_grad))
</code></pre>
<p>it reports a different number of trainable parameters.</p>
<p>How many trainable ConvLSTM cells are instantiated by writing the layer in this form? 1 and the output is obtained recursively or seq_len cells with trainable parameters in each one?</p>

ptrblck: <p>I see only a single <code>ConvLSTMCell</code> being initialized. What dies <code>model.named_parameters()</code> return for your model?</p>

vk12: <p>Outputs:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(model)
ConvLSTM(
  (convLSTMcell): ConvLSTMCell(
    (conv): Conv2d(26, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  )
)

&gt;&gt;&gt; for name, param in t.named_parameters():
&gt;&gt;&gt;     print(name)

convLSTMcell.W_ci
convLSTMcell.W_co
convLSTMcell.W_cf
convLSTMcell.conv.weight
convLSTMcell.conv.bias
</code></pre>
<p>It seems that only 1 ConvLSTM cell is being initialized. It this a usual way to create ConvLSTM layers? I thought that I would have to initialize as many ConvLSTM cells as the number of inputs to the network and not only one and then reuse it.</p>
<p>Are you aware of any official implementation of a ConvLSTM layer I can look at?</p>
<p>Thank you for taking the time to answer!</p>
How do I use 3d inputs for a 2d model?

maryma: <p>I have 3d MRI images and I wanted to use classification for them. But they are not too much I do not get a good accuracy. I wanted to use pretrained models. Most of the pretrained models are 2d and I couldnt find any pretrained models for 3d inputs. What is your recommendation to me? Is there any way to change my 3d inputs to 2d inputs? I can iterate on the slice dimension but how can I make a dataloader? What are your ideas when I have 3d inputs but I want to use 2d models? Can you please explain in code?</p>

JuanFMontesinos: <p>Hmmm I don’t know if 2d pretrained models are gonna be useful for MRI images.<br>
That being said, you have some 3d models pretrained in datasets such us kinetics.</p>
<p>That being said I can offer you some alternatives:</p>
<ul>
<li>
<p><a href="https://arxiv.org/abs/1711.11248v3" class="inline-onebox" rel="noopener nofollow ugc">[1711.11248v3] A Closer Look at Spatiotemporal Convolutions for Action Recognition</a> Hybrid networks. You can apply some 3d convolutions at the beggining and then 2d convolutions. This way you can reduce the amount of parameters yet exploiding 3d info. You can adapt the 3D implementation to be initialized with 2d weights and finetune from there.</p>
</li>
<li>
<p>3D ResNet (pretrained) <a href="https://github.com/kenshohara/3D-ResNets-PyTorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - kenshohara/3D-ResNets-PyTorch: 3D ResNets for Action Recognition (CVPR 2018)</a></p>
</li>
<li>
<p>Using 2d models by extracting features from each slice or the mri. Just convert your batch which is<br>
B, T, H, W into BxT,H,W. Then extract the features (let’s say C feats). And you would get BxTxC, convert back to B,T,C. Aggregate the T feats to have a general descriptor.</p>
</li>
</ul>

maryma: <p>Thanks for your quick reply.</p>
<p>can you explain more about the third way? I couldnt understand it completely.</p>

JuanFMontesinos: <p>Pytorch’s layers require a dimension to be “the batch dim”.<br>
This means a 2 convolutional layer expects a tensor of size B,C,H,W.<br>
However you have an additional dimension (time), thus B,T,H,W. Ideally you would like to extract information for each t slice independently. What you can do is to fold the time and batch dimensions into a single one. BxT,H,W which would act as a fictional batch dim B’,H,W. Then you extract the features so you would obtain something like B’,C.<br>
As you know B’=BxT you can unfold the features so you recover the temporal dimension.</p>
<p>Then you just need to add some layers that combine the temporal information to perform classification.</p>
Scheduler error

ADONAI_TZEVAOT: <p>Got the following error while training my model</p>
<pre><code class="lang-auto">---&gt; 95         scheduler.step()
.
.
.
ValueError: Tried to step 1402 times. The specified number of total steps is 1400

</code></pre>
<p>But I’ve NOT specified anywhere in my code that the total steps should be 1400</p>
<p>please help</p>

Aayush_Garg: <p>You need to provide more information. Maybe share the snippet of your minimal code, the line where the error stack originates, and so on.</p>
<p>However, I remember this sort of error popping up while using <code>pytorch-lightning</code>. There this error occurs when you are using a <code>scheduler</code> in the <code>optimizer</code> without passing the <code>max_epochs</code> argument.</p>

ADONAI_TZEVAOT: <p>Here’s the code snipet:</p>
<pre><code class="lang-auto">  criterion = nn.BCEWithLogitsLoss()
  optimizer = torch.optim.Adam(model.parameters(),lr=0.001)
  scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.001,
                                                steps_per_epoch=int(len(train_dl)),
                                                epochs=num_epochs,
                                                anneal_strategy='linear')
</code></pre>

ptrblck: <p>You are defining the number of steps via <code>steps_per_epoch</code> and <code>epochs</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.OneCycleLR.html">docs</a>:</p>
<blockquote>
<ul>
<li><strong>epochs</strong> (<a href="https://docs.python.org/3/library/functions.html#int"><em>int</em></a>) – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None</li>
<li><strong>steps_per_epoch</strong> (<a href="https://docs.python.org/3/library/functions.html#int"><em>int</em></a>) – The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None</li>
</ul>
</blockquote>
<p>and as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(1, 1)

criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(),lr=0.001)
num_epochs = 10
steps_per_epoch = 2
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.001,
                                                steps_per_epoch=steps_per_epoch,
                                                epochs=num_epochs,
                                                anneal_strategy='linear')

for epoch in range(num_epochs):
    for _ in range(steps_per_epoch):
        optimizer.step()
        scheduler.step()
        
optimizer.step()
scheduler.step()
# ValueError: Tried to step 22 times. The specified number of total steps is 20
</code></pre>
Overhead using torch.profiler

lfsalasg: <p>Hi everybody, I have been working for a few months with pytorch and this is the first time I try to use torch.profiler since I’m trying to get information about CUDA time. However, I’m running a small test only with the cpu and I see that there is a very large overhead when running the models inside the profile instance.</p>
<p>I would like to understand if it is typical behavior or if I’m doing something wrong. Thanks in regards and I copy the code below:</p>
<pre><code class="lang-auto">start = time.time()
with profile(activities=[ProfilerActivity.CPU]) as prof:
    for _ in range (200):
        with record_function("forward"):
            y = model(a)
        with record_function("backward"):
            y = model(b)
print(f"True time {time.time() - start} seconds")
print(f"Time calculated by profiler:{prof.key_averages().self_cpu_time_total / 1e6} seconds")
</code></pre>
<p>The model CNN1 corresponds to:</p>
<pre><code class="lang-auto">class CNN1(Module):
    def __init__(self, nclasses:int) -&gt; None:
        super().__init__()
        self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
        self.cnn = Sequential(
            Conv2d(1, 32, (2,2), (1,1), (1,1)),
            ReLU(),
            MaxPool2d((2,2),(1,1),(1,1)),
            Conv2d(32, 64, (2,2), (1,1), (0,1)),
            ReLU(),
            MaxPool2d((2,2), (2,1),(0,1)),
            Conv2d(64, 128, (2,2), (2,1), (0,0)),
            ReLU(),
            MaxPool2d((2,2), (2,1), (0,0)),
            Conv2d(128, 512, (2,2), (2,2), (0,0)),
            ReLU(),
            Conv2d(512, 512, (2,2), (2,2), (0,0)), 
        )

        self.lineal = Sequential(
            Linear(512, 256),
            ReLU(),
            Linear(256, nclasses),
            LogSoftmax(dim=2)
        )
    
    def forward(self, x:Tensor):
        x.to(self.device) # Se envía el tensor a la GPU para las operaciones del modelo
        logging.debug(f"Input tensor shape is:{x.size()}")
        logits = self.cnn(x) # Forma del tensor es B, 512, 1, L 
        logits = torch.squeeze(logits, 2)
        logits = logits.permute(0, 2, 1) # Forma del tensor es B, L, 512
        logits = self.lineal(logits) # Forma del tensor es B, L, C
        logging.debug(f"Output tensor (log_logits) shape is:{logits.size()}")
        return logits

</code></pre>

ptrblck: <p>I would assume that some overhead would be expected as operations are being recorded, but how large is the overhead in your case? Profiling should of course still return the “real” runtime and not slow down the execution significantly as it would make the profile useless.</p>
Is there any problem with this model?

fokkersn: <p>from nets import mobilenet<br>
import torch.nn as nn<br>
from nets.layers import convolution</p>
<p>class MobileCenterNet(nn.Module):</p>
<pre><code>def __init__(self,device):
    super(MobileCenterNet, self).__init__()

    self.dimension = 128
    self.device = device
    self.backbone = mobilenet.mobilenetv2().to(self.device)

    self.p3 = convolution(k=1, inp_dim=320, out_dim=self.dimension)
    self.up_p3 = nn.Upsample(scale_factor=2,mode='nearest')
    self.p2 = convolution(k=11, inp_dim=96, out_dim=self.dimension)
    self.up_p2 = nn.Upsample(scale_factor=2,mode='nearest')
    self.p1 = convolution(k=11, inp_dim=32, out_dim=self.dimension)
    self.up_p1 = nn.Upsample(scale_factor=2,mode='nearest')
    self.p0 = convolution(k=1, inp_dim=24, out_dim=self.dimension)
    self.feature_layer = convolution(k=3,inp_dim=128,out_dim=self.dimension)
    self.heatmap_layer = nn.Sequential(
        nn.Conv2d(128, 64, 3, padding=1, stride=1, bias=False),
        nn.BatchNorm2d(64),
        nn.ReLU(inplace=True),
        nn.Conv2d(64, 1, 1, padding=0, stride=1)
    )

    self.wh_layer = nn.Sequential(
        nn.Conv2d(128, 64, 3, padding=1, stride=1, bias=False),
        nn.BatchNorm2d(64),
        nn.ReLU(inplace=True),
        nn.Conv2d(64, 2, 1, padding=0, stride=1),
    )

    self.reg_layer = nn.Sequential(
        nn.Conv2d(128, 64, 3, padding=1, stride=1, bias=False),
        nn.BatchNorm2d(64),
        nn.ReLU(inplace=True),
        nn.Conv2d(64, 2, 1, padding=0, stride=1),
    )

def forward(self, x):
    # feature extractor
    fm0, fm1, fm2, fm3 = self.backbone(x)

    fm3 = self.p3(fm3)
    up_fm3 = self.up_p3(fm3)
    reduce_dim_fm2 = self.p2(fm2)
    fm2 = 0.5 * up_fm3 + 0.5 * reduce_dim_fm2

    up_fm2 = self.up_p2(fm2)
    reduce_dim_fm1 = self.p1(fm1)
    fm1 = 0.5 * up_fm2 + 0.5 * reduce_dim_fm1

    up_fm1 = self.up_p1(fm1)
    reduce_dim_fm0 = self.p0(fm0)
    fm0 = 0.5 * up_fm1 + 0.5 * reduce_dim_fm0
    features = self.feature_layer(fm0)

    # detector
    hm = self.heatmap_layer(features)
    wh = self.wh_layer(features)
    reg = self.reg_layer(features)
    output = {'hm': hm, 'wh': wh, 'reg': reg}
    return output
</code></pre>
<p>def get_centernet(device):<br>
model = MobileCenterNet(device=device)<br>
return model</p>
<p>I’m trying to train a model with centerNet and mobilenet but I think there are some problem with my code.<br>
Could you find the problems,</p>

ptrblck: <p>Why do you think there is a problem? Are you seeing any errors and if so could you post them?</p>
Applying gradient checkpoint for Transformer

li-yi-dong: <p>I’m trying to apply gradient checkpoint to some sub-layer of <code>Transformer</code>, so that I can enable larger batch size and higher throughput for training.</p>
<p>But, without the knowledge of the activation size of each <code>OP</code> in <code>PyTorch</code>, I cannot decide to apply gradient checkpoint on which sub-layer.</p>
<p>Even though, I can insert  <code>torch.cuda.memory_summary</code> and timing at each line of forward path, that would be a boilerplate.</p>
<p>So, is there any cheatsheet for the activation size of each <code>OP</code> in <code>PyTorch</code>? Or, a guild line for applying gradient checkpoint?</p>
Using the torch::data::make_data_loader function in LibTorch

JamesDickens: <p>I’m having a bit of difficulty in LibTorch actually declaring and using torch::data::make_data_loader function call and then iterating over a dataset.</p>
<p>Supposing I have a header file sample_dataloader.h with code:</p>
<pre><code>#include &lt;torch/torch.h&gt;

class sample_dataloader : public torch::data::Dataset&lt;sample_dataloader&gt; {
     public: 
     torch::Tensor rand_val;
     sample_dataloader(){
         rand_val = torch::rand({10, 3, 600, 600}); // random batch of images
     }
     
     torch::optional&lt;size_t&gt; size() const override {
         return 10;
     }
     
     torch::data::Example&lt;&gt; get(size_t index) {
         return rand_val[index];
     }
}
</code></pre>
<p>Then in my main.cpp file I try to use it as in:</p>
<pre><code>#include &lt;torch/torch.h&gt;
#include "sample_dataloader.h"

int main() {
    auto train_set = sample_dataloader();
    int batch_size = 5;
    int num_workers = 2;
    auto dl = 
    torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt; 
     (std::move(train_set), DataLoaderOptions().batch_size(batch_size).workers(num_workers));
    for (auto&amp; batch : loader) {
       // do stuff 
    }
}
</code></pre>
<p>I get a compilation error that says “the range-based for statement requires a suitable begin function, but none was found”. When I try</p>
<pre><code>for (auto&amp; batch : *loader) {
       // do stuff 
}
</code></pre>
<p>instead, I get a memory error.</p>
<p>How am I supposed to use the torch::data::make_data_loader function properly?</p>

JamesDickens: <p>Solved this problem, correct usage is:<br>
“sample_dataloader.h”</p>
<pre><code class="lang-auto">#pragma once
#include &lt;torch/torch.h&gt;

class sample_dataloader : public torch::data::Dataset&lt;sample_dataloader&gt; {
public:
    torch::Tensor rand_val;
    sample_dataloader() {
        rand_val = torch::rand({ 10, 3, 600, 600 }); // random batch of images
    }

    torch::optional&lt;size_t&gt; size() const override {
        return 10;
    }

    torch::data::Example&lt;&gt; get(size_t index) {
        return { rand_val[index], torch::randint(5, {5}) };
    }
};
</code></pre>
<p>“sample_main.cpp”</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include "sample_dataloader.h"

int main() {
    auto train_set = sample_dataloader().map(torch::data::transforms::Stack&lt;&gt;());
    int batch_size = 5;
    int num_workers = 0;
    auto dl =
        torch::data::make_data_loader&lt;torch::data::samplers::SequentialSampler&gt;
        (std::move(train_set), torch::data::DataLoaderOptions().batch_size(batch_size).workers(num_workers));
    for (auto&amp; batch : *dl) {
        auto rand_im = batch.data;
        auto targets = batch.target;
    }
}
</code></pre>
TypeError: linear(): argument 'input' (position 1) must be Tensor, not int

desert_ranger: <p>Here’s my code -</p>
<pre><code class="lang-auto"># Here we import all libraries
import numpy as np
import gym
import matplotlib.pyplot as plt
import os
import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
from collections import deque 
env = gym.make("CliffWalking-v0")

#Hyperparameters
episodes = 5000
eps = 1.0
learning_rate = 0.1
discount_factor = 0.99
tot_rewards = []
decay_val = 0.001
mem_size = 50000

class NeuralNetwork(nn.Module):
    def __init__(self, state_size, action_size):
        super(NeuralNetwork, self).__init__()
        self.state_size = state_size
        self.action_size = action_size
#         self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(state_size, 30),
            nn.ReLU(),
            nn.Linear(30, 30),
            nn.ReLU(),
            nn.Linear(30, action_size)
        )
    def forward(self, x):
        x = self.linear_relu_stack(x)
        return x
		
model = NeuralNetwork(env.observation_space.n, env.action_space.n)
opt = torch.optim.Adam(params=model.parameters(), lr=learning_rate)
loss = nn.MSELoss()
replay_buffer = deque(maxlen=mem_size)

state = env.reset()
action = model(state)
</code></pre>
<p>While there are many SO posts on this topic, none of them solve the issue.</p>

thecho7: <p>Could you add <code>print(x)</code> on the first line of <code>def forward()</code>?</p>

desert_ranger: <p>Thank you for the reply. It prints x =  36.</p>
<p>I corrected my network to be the following , but I still get an error -</p>
<pre><code class="lang-auto">class NeuralNetwork(nn.Module):
    def __init__(self, state_size, action_size):
        super(NeuralNetwork, self).__init__()
        self.state_size = state_size
        self.action_size = action_size
#         self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(1, 30),
            nn.ReLU(),
            nn.Linear(30, 30),
            nn.ReLU(),
            nn.Linear(30, action_size)
        )
    def forward(self, x):
        print("x = ",x)
        x = self.linear_relu_stack(x)
        return x
</code></pre>

thecho7: <p>The input <code>x</code> should be a tensor but not an integer.<br>
e.g.</p>
<pre><code class="lang-auto">net = NeuralNetwork(state_size=10, action_size=5) # arbitrary number
x = torch.rand(1, 10) # Input should be a tensor
out = net(x)
</code></pre>
Combine AdaBoost and ResNet in pytorch

Mehrdad_Jannesar: <p>I want to combine Adaboost with ResNet. To get better hyperparameters. My goal is to remove the final layer of ResNet, which is a FC 512, and to bring it to Adaboost, and to use AdBoost instead of the decision tree of the FC network to repeat it until the convergence, and then get the final output of the network.<br>
Can you help me to do this?</p>

aatrey: <p>Did you end up completing this? I am curious how you did it and whether it worked.</p>

bryan123: <p>Hi <a class="mention" href="/u/mehrdad_jannesar">@Mehrdad_Jannesar</a>, may I know if you have completed Adaboost in CNN? Thank you</p>
Torch Operations Using Custom Functions

se7en.dreams: <p>I have two tensors:</p>
<pre><code class="lang-python">a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])
</code></pre>
<p>I’d like to obtain a matrix of shape <code>a x b</code>, where each value is obtained using a custom function. For the sake of the example, let’s say <code>f(a,b) = a * b + 2</code>.</p>
<p>So, my output would be:</p>
<pre><code class="lang-python">f(1,4) | f(1,5) | f(1,6)
f(2,4) | f(2,5) | f(2,6)
f(3,4) | f(3,5) | f(3,6)
</code></pre>
<p>How do I do this without using loops?</p>

smth: <p>please dont tag specific people.</p>

smth: <p>without loops, this is best done by first replicating tensors and then doing the operations:</p>
<pre><code class="lang-auto">a.view(-1, 1).repeat(1, 3) * b.view(1, -1).repeat(3, 1) + 2
</code></pre>

se7en.dreams: <p>Thanks for your response. I was able to do it using Numpy’s <code>frompyfunc</code>. I don’t suppose there is a Pytorch equivalent of that?</p>
<p>And thanks for the feedback, I won’t tag specific people from now on.</p>

smth: <p>there are but they are not going to be efficient.</p>
<p>There’s an experimental package called <code>torchdim</code> that does this really elegantly and efficiently. For it, however, you currently need the PyTorch nightly (not yet supported with a released PyTorch): <a href="https://github.com/facebookresearch/torchdim" class="inline-onebox">GitHub - facebookresearch/torchdim: Named tensors with first-class dimensions for PyTorch</a></p>
Lowest validation loss produces more false positives

SU801T: <p>I’m training a faster r-cnn object detector and have used the following to produce the loss for validation:</p>
<pre><code class="lang-auto">def evaluate_loss(model, data_loader, device):
    val_loss = 0
    with torch.no_grad():
      for images, targets in data_loader:
          images = list(image.to(device) for image in images)
          targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

          losses_dict, detections = eval_forward(model, images, targets)
          losses = sum(loss for loss in losses_dict.values())
          
          val_loss += losses
          
    validation_loss = val_loss/ len(data_loader)    
    return validation_loss
</code></pre>
<p>essentially, it sums up all types of losses output and divides this by the number of batches to give me a figure for the epoch.</p>
<p>However, I compared the output of a model that has the lowest validation loss, compared to another model that is output at a later epoch and found that although those models are less “sensitive”, they produce vastly less false positives.</p>
<p>As an FYI, I am using cross entropy loss and finding a few similar sized targets in a 512 x 512 pixel image (binary classification - object + background).</p>
<p>Is there any reason why this might be happening?</p>

eqy: <p>What’s the balance of positive/negative examples in your dataset? For example, it could be due to the model being more “accurate” overall if the number of false negatives is smaller despite more false positives.</p>

SU801T: <p>Well it’s object detection, looking for objects of 32 x 32 pixels in an image size of 512 x 512 pixels. So there is typically a lot of background class.</p>
<p>In regard to my validation outputs, for the model with the lowest validation loss:</p>
<pre><code class="lang-auto">true positive: 610
false negative: 25
false positive: 740
</code></pre>
<p>And for the model from a later epoch:</p>
<pre><code class="lang-auto">true positive: 595
false negative: 40
false positive: 383
</code></pre>
<p>So yes, there is sizeable discrepancy between the true positives and false negatives between the two models… and so it more accurate in terms of “sensitivity” measures. Although, it’s not very precise…</p>
<p>Also as an FYI, I do not apply a threshold on the probabilities of the outputs. These numbers are representative of every detection.</p>