akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Implement SVD with simple PyTorch functions for Core ML ops compatibility

rahulbhalley: <p>Hi everyone!</p>
<p><strong>Goal</strong></p>
<p>I want to implement <a href="https://pytorch.org/docs/stable/generated/torch.svd.html?highlight=svd#torch.svd" rel="noopener nofollow ugc"><code>torch.svd</code></a> function using simpler PyTorch functions like <code>torch.matmul</code> and others.</p>
<p><strong>Reason</strong></p>
<p>The reason being wanting to do is that I have a trained neural network (having SVD op) that I want to convert to Core ML model but Core ML’s <a href="https://apple.github.io/coremltools/source/coremltools.converters.mil.mil.ops.defs.html" rel="noopener nofollow ugc">MIL Ops</a> don’t have implementation of <code>svd()</code> function.</p>
<p><strong>One possible solution. But it’s not correct.</strong></p>
<p>One <a href="https://github.com/j2kun/svd/blob/main/svd.py" rel="noopener nofollow ugc">implementation</a> I found is in NumPy but doesn’t produces same dimensional <code>ndarray</code> outputs as produced by PyTorch’s <code>svd</code> implementation. Moreover, the signs of output <code>ndarray</code>'s scalar elements are also wrong.</p>
<p><strong>Can you please help me?</strong></p>
<p>I am stuck at this problem since January 2021 (like <em>almost 9 months</em> now). If anyone could please help me I’d be really really thankful to you!!</p>
<p>Regards<br>
Rahul Bhalley</p>

Linbin: <p>The link in your post implemented reduced SVD. You need to call<br>
<code>torch.linalg.</code> <code>svd</code> (<em>A</em> , <em>full_matrices=False</em>)<br>
to do the same.</p>
<p>Note that the returned UV is not unique. If you think it’s still not right, can you share some example?</p>

rahulbhalley: <p>Thanks for the reply <a class="mention" href="/u/linbin">@Linbin</a>!</p>
<p>I don’t know how to compute non-reduced SVD without the help of PyTorch. And there are no examples available in Python on GitHub to implement it either. Could you please give me a clue of how to do that? I need it to rewrite my neural network in Core ML format. I did try to look into SciPy’s and NumPy’s codebase but all in vain. I couldn’t understand how it could be done.</p>
<p>Edit: I’m not good with understanding the underlying C++ codebase.</p>
<p>Any help is highly appreciated!</p>

Linbin: <p><a class="mention" href="/u/xta0">@xta0</a> any thoughts on supporting PyTorch SVD in core ml?</p>

rahulbhalley: <p>I can create custom operator of SVD in Core ML but I first need to know how to compute SVD with<code>full_matrices=True</code> without using <code>torch.svd()</code> function but simpler ones.</p>
<p>CC <a class="mention" href="/u/xta0">@xta0</a>.</p>

xta0: <p>Haven’t looked into detail, but can you open an issue in the coremltools Github? See if they can try helping you implement that op in MIL.</p>
Using CrossEntropyLoss instead of BCELoss

WAN: <p>Hello, I have a dataset of text and labels and I want to do multiclass classification. I have 5 classes. I follow this toturial: <a href="https://www.analyticsvidhya.com/blog/2020/01/first-text-classification-in-pytorch/#wait_approval" class="inline-onebox" rel="noopener nofollow ugc">Text Classification Pytorch | Build Text Classification Model</a><br>
But I needed to change the loss function to be:CrossEntropyLoss</p>
<pre><code class="lang-auto">        loss = criterion(predictions, batch.Rating)        
</code></pre>
<p>this error keeps showing:</p>
<pre><code class="lang-auto">IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
</code></pre>
<p>Could you please help me with how can I call this loss function?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects a model output in the shape <code>[batch_size, nb_classes]</code> and a target in <code>[batch_size]</code> containing class indices in the range <code>[0, nb_classes-1]</code> for a multi-class classification.<br>
Based on the error message I guess your target has an additional dimension, which should be removed.</p>

WAN: <p>Thanks, it works fine now</p>
Fault-tolerant Dataset/DataLoader

Olivier-CR: <p>Hi,</p>
<p>I have a PyTorch seg training running on a SageMaker-managed EC2, and training errors after 1000 minibatches with the training Dataloader breaking because “RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0”</p>
<p>What could cause this? One bad record in the dataset? Is there a way to build fault-tolerant datasets/data loaders that don’t break if one record is bad?..</p>

ptrblck: <p>The error is most likely caused by an image with 4 channels (additional alpha channel), while the rest seem to use 3 channels (RGB only).<br>
You could iterate your <code>DataLoader</code> once and make sure it’s able to create the batches.<br>
How would a “fault-tolerant” dataset work? I.e. you could use a <code>try/except</code> block in the <code>Dataset.__getitem__</code>, but I wouldn’t know what you want to return in case the sample is broken.<br>
E.g. returning a random tensor instead or a static tensor would interfere with your training.</p>

Olivier-CR: <p>Indeed it was! a 4-channel PNG hidden among my 40k 3-channel PNGs… forcing Torchvision read_image to read to RGB fixed the issue</p>

Olivier-CR: <p>yes I guess the easiest option here is to make a try/except in the DataLoader to discard full batches that cause errors. Would you know how to do that? I cannot find resources for custom DataLoader class creation (<a href="https://pytorch.org/tutorials/recipes/recipes/custom_dataset_transforms_loader.html" rel="noopener nofollow ugc">this page</a> is named " Developing Custom PyTorch Dataloaders" but customizes only dataset and transforms, not the dataloader class itself)</p>
Unbalanced Dataset for YOLOv3 implementation

mth1996: <p>Hello Guys,</p>
<p>i am trying to implement the YOLOv3 architecture in Pytorch. You can find a quick recap for yolo here. <a href="https://github.com/aladdinpersson/Machine-Learning-Collection/tree/master/ML/Pytorch/object_detection/YOLOv3" rel="noopener nofollow ugc">Implementation of YOLOv3 in Pytorch</a></p>
<p>I took this implementation more or less and tried a refinement training with a custom dataset after pretraining the model. Now i came to a point where my Precisions tend to bias on the class with the highest amount of pictures in the training set. (class0: 2315 labels, class1: 867 labels, class2: 335 labels) As i am doing detection which is a multilabel problem, i can not simply use <code>WeightedRandomSampler</code>. So i intended to do a weighted loss implementation, to decrease the influence of the class0 as it has a great class influence.</p>
<p>Now the problem is, that in the Implementation they are using <code>BCEWithLogitsLoss()</code> as loss function where they write that the <code>weight</code> argument only weights the images per batch. What i need is a loss, with which i can penalty bigger datasets.</p>
<p>For Clearification a Code Snipped which shows what i wand to do.</p>
<pre><code class="lang-auto">class YOLOLoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss()
        self.bce = nn.BCEWithLogitsLoss()
        self.entropy = nn.CrossEntropyLoss()
        self.sigmoid = nn.Sigmoid()
        # Constants signifying how much to pay for each respective part of the loss
        self.lambda_class = 1
        self.lambda_noobj = 4
        self.lambda_obj = 2
        self.lambda_box = 8

    def forward(self, preds, target, anchors):
        """Copmute yolo loss

        Arguements:
            preds (tensor): tensor of shape (N, 3, S, S, 5+C)
            target (tensor): tensor of shape (N, 3, S, S, 6)
            anchors (tensor): tensor of shape (3, 2)

        Prediction format:
            (x_raw, y_raw, w_raw, h_raw, conf, [classes...])

        Target format:
            (x_offset, y_offset, w_cell, h_cell, conf, class)
        """
        device = preds.device
        # Normalize factor
        scale = preds.size(2)
        # target with -1 is ignored
        obj = target[..., 4] == 1      # (N, 3, S, S)
        noobj = target[..., 4] == 0    # (N, 3, S, S)
        
        # NO OBJECT LOSS
        # ==========================================
        noobj_loss = self.bce(
                        preds[..., 4:5][noobj],
                        target[..., 4:5][noobj],
                        )
</code></pre>
One of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 48, 3, 3]] is at version 2; expected version 1 instead

KeisukeShimokawa: <p>When I was training the GAN, the first iteration worked well, and the second training probably caused an error on the Discriminator side.</p>
<p>The details of the error are as follows. This dimension [3, 48, 3, 3] means the first layer of the Discriminator, but I didn’t use any in place operation.</p>
<pre><code class="lang-auto">one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3, 48, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>What is causing this error?</p>
<p>Full Discriminator code is below</p>
<p><a href="https://github.com/KeisukeShimokawa/papers-challenge/blob/master/src/gan/FQGAN/models/FQGAN_64.py" rel="nofollow noopener">https://github.com/KeisukeShimokawa/papers-challenge/blob/master/src/gan/FQGAN/models/FQGAN_64.py</a></p>

albanD: <p>Hi,</p>
<p>Is the training loop the one <a href="https://github.com/KeisukeShimokawa/papers-challenge/blob/70d2d2798124b66c437297ba0537e58d7fb2f428/src/gan/FQGAN/main.py#L122">here</a>?<br>
Given that it happens not at the first backward it can be:</p>
<ul>
<li>An optimizer.step() modified a weight that was already used by a forward and you call backward after:</li>
</ul>
<pre><code class="lang-auto">out = model(inp)
opt_model.step() # Modifies the model's weights inplace
out.sum().backward() # Will fail with this error
</code></pre>
<ul>
<li>You reuse some variables from one iteration to the next by mistake. To check this, the simplest thing is to wrap the content of the for loop into a single function to make sure everything goes out of scope properly:</li>
</ul>
<pre><code class="lang-auto">def one_step(sample):
    # Do one step with sample using everything in scope.
for sample in dataloader:
    one_step(sample)</code></pre>

JH_S: <p>you are my angle.<img src="https://discuss.pytorch.org/images/emoji/apple/fairy.png?v=10" title=":fairy:" class="emoji" alt=":fairy:"><img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=10" title=":+1:" class="emoji" alt=":+1:"> Have a nice day!</p>
Model partial derivatives into loss function

Marco_Repetto: <p>I want to force a monotonic relationship between a set of features and the model’s outcome, and I would like to do it by controlling the model’s partial derivatives into the loss function, like a regularization term.</p>
<p>I tried the solution proposed <a href="https://discuss.pytorch.org/t/how-to-include-gradient-of-nn-wrt-the-input-in-the-loss-function/103758">here</a>, but I got an error:</p>
<pre><code class="lang-auto">one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Is there a way to include the model’s partial derivatives into the loss?</p>

albanD: <p>Hi,</p>
<p>This is the same error as the one from the post you linked. Does the answer there help you?</p>

Marco_Repetto: <p>Hi Alban,</p>
<p>I tried the solution proposed (simply copy pasting) but is not working.</p>
Speed up this very minimal code? tensors to numpy to image

prismspecs: <p>I’m taking in latent vecs and generating images, pretty straight forward, but getting less than half the FPS as my Tensorflow model. Can someone help me speed this up?</p>
<pre><code class="lang-auto">def convert(model, inputs):
    z = inputs['z']
    jzf = [float(i) for i in z]
    jzft = torch.FloatTensor(jzf)
    jzftr = jzft.reshape([1, 512])
    latents = jzftr.cuda()
    truncation = inputs['truncation']

    img = G(latents, None, truncation)
    img = (img.permute(0, 2, 3, 1) * 127.5 + 128).clamp(0, 255).to(torch.uint8)
    return {'image': img[0].cpu().numpy()}
</code></pre>

albanD: <p>Hi,</p>
<p>What is the type of <code>z</code>?</p>
<p>Also creating the latent would be more efficient as: <code>latents = torch.tensor(jzf, dtype=torch.float, device="cuda").view(1, 512)</code>.</p>
Can not "import torch" with my python 3

2021: <p>Hi, the following is the error message.</p>
<pre><code class="lang-auto">:~$ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
python3: Relink `/lib/x86_64-linux-gnu/libsystemd.so.0' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
python3: Relink `/lib/x86_64-linux-gnu/libudev.so.1' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
Segmentation fault (core dumped)
</code></pre>
<p>Any comments are appreciated.</p>

albanD: <p>Hi,</p>
<p>Could you share some information about your system and how you installed pytorch?</p>
How to effectively load a large text dataset with PyTorch?

Emanuel_Huber: <p>I have hundreds of CSV files that each contain hundreds of megabytes of data. To create a class that inherits from PyTorch’s Dataset the <strong>getitem</strong> method must access a single sample at a time, where the <code>i</code> parameter of the function indicates the index of the sample. However, to perform lazy loading my class just saves the name of each file instead of saving all the data in memory.</p>
<p>So far so good, but I’m having doubts about how to access a single data in the <code>__getitem__</code> method, in my searches I found the following strategies:</p>
<ol>
<li>In <code>__init__</code> save the number of samples that each file has, then in <strong>getitem</strong> load the file with pandas and access the respective line. It is possible to know which file to access by the id received as a parameter, for example: having 2 files, where each has 100 samples, id 100 would represent the first sample of the second file, id 101 would represent the second sample of the second file and so on against. However, this approach has an obvious problem which is the number of IOs needed.</li>
<li>Perform the same process presented in 1, but store files in a cache as long as it fits in memory. This approach seems to be better than the first as it reduces the number of IOs as some files will be in memory. The problem with this approach is how the caching policy will be carried out, I thought of using something like <code>dask</code> to handle this, but <code>iloc</code> is quite inefficient according to the <a href="https://docs.dask.org/en/latest/dataframe-indexing.html" rel="noopener nofollow ugc">documentation</a>.</li>
</ol>
<p>Does anyone have a better idea or indication of some material?</p>

AbdulsalamBande: <p><a class="mention" href="/u/emanuel_huber">@Emanuel_Huber</a> , why dont you featurizer(embedding or one hot) your text data and store the data in binary format using numpy.memmap()  <a href="https://numpy.org/doc/stable/reference/generated/numpy.memmap.html" class="inline-onebox" rel="noopener nofollow ugc">numpy.memmap — NumPy v1.21 Manual</a> . Then you can use the normal <strong>getitem</strong> method.</p>

Emanuel_Huber: <p>That would be a good solution, but I found a more straightforward one that probably will be integrated on the official PyTorch code, the <a href="https://github.com/webdataset/webdataset" rel="noopener nofollow ugc">webdataset</a> package.</p>

Emanuel_Huber: <p>After messing with <code>webdataset</code> I could not use it for my case. When I convert my 20Gb dataset with webdataset ShardWriter or with <code>tarp</code> CLI that conversion generated a 200Gb file and I do not have this space available on my disk. I am not sure, but I think that TFRecord is a better and less disk-consuming approach.<br>
After this little journey, my ending was to load all the raw data in memory and tokenized it before passing it to the network. If you have enough RAM, I advise you to do the same.</p>
Multiple training jobs in a single GPU

itsm: <p>Hi guys,</p>
<p>What is the best practice of running multiple training jobs in a single GPU? Do you run them parallelly or sequentially?</p>
<p>Based on my benchmark, running two training jobs simultaneously takes longer than running a single training job. But the GPU is not fully utilized when jobs are running sequentially, which is a wasteful use of resources.</p>
<p>Thanks</p>
Questions about model saving and loading and resuming training

Youlong_Ding: <p>I first run 10 epochs for training. I save model/optimizer/scheduler state_dict() in epoch 2. And I run program again to load these states and resume training. But I found that the results are not the same as before.<br>
I think it is because the random seed. I manually set the random seed at the beginning of the program. When the program runs 2 epochs, the current random state is changed. So when I run program again, resuming the training at 2 epoch. The random seed I need to set is not the one before. How can I solve this?</p>
Different initialization on 2 GPUs

mpalaourg: <p>Hello,<br>
First, I didn’t find any appropriate category, so I’ll type my question here and I am sorry if it’s not the right place. My problem is the following:</p>
<p>I have two GPUs (TitanXp and 2080 Ti), where I get different results on my training, even after I used all the tips <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">here</a>. I pinpoint the issue in the initialization of the network (using the <code>xavier_uniform_</code> function) and to be exact after calling <code>Tensor.uniform_</code>. The strange part is that for the whole network (85 modules), a part of each module, first 50 -if I remember correctly- rows with 1024 values are identical and from there until the end of the tensor completely different. Do you have any clues, why that’s happen? Furthermore, do you have any idea/“trick” to get the same initialization on both machines?</p>
<p>Thanks in advance.<br>
George</p>

ptrblck: <p>Deterministic results are not guaranteed between different hardware releases.<br>
To get the same initialization, you could initialize the model on one device, save the <code>state_dict</code>, and load it on the other.</p>

mpalaourg: <p>Thanks for your answer, ptrblck! I thought of that, but I am not keen on the idea of loading the <code>state_dict</code> on the other machine, I would rather go with a more programming approach/solution. On the other hand, I want reproducible results and the model is kind of sensitive in its initialization, that’s why I asked for a “trick”.</p>
<p>Do you ever encounter something like that? My thought is that the behavior I report previously, is due to the different number of cores on each GPU, but I can’t get my head around it and think of a solution.</p>
<p>George</p>
An error raise when using python package 'constants'

Ford: <p>First, I import this package (constants) by the following code:</p>
<blockquote>
<p>import constants</p>
</blockquote>
<p>Then, an error which is  ’ AttributeError: module ‘constants’ has no attribute ‘TOP_K’ ', raises when I writte the following code:</p>
<blockquote>
<p>constants.TOP_K</p>
</blockquote>
<p>Can anyone fix this problem? Thanks.<br>
BTW, my python version is python3.6.</p>

ptrblck: <p>This issue doesn’t seem to be PyTorch-related, so I would recommend to post the question in the project repository.</p>
If your model is too large

slavavs: <p>My network model is quite large. But I want to know if it fits entirely in gpu memory. How can I do it?</p>

ptrblck: <p>Push the model to the device via <code>model.to('cuda')</code> and check the memory usage via <code>nvidia-smi</code>.<br>
If the model doesn’t fit, PyTorch will raise an error.<br>
Also, since the intermediate activations will use additional memory during training, you might run a dummy forward and backward pass and check the memory usage via:</p>
<pre><code class="lang-python">torch.cuda.memory_allocated()
torch.cuda.max_memory_allocated()
</code></pre>

slavavs: <p>I started nvidia-smi while working. How to understand the model fits into memory or not?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b095498193811b6834aacb0d75f06765d0decf0.png" data-download-href="https://discuss.pytorch.org/uploads/default/6b095498193811b6834aacb0d75f06765d0decf0" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b095498193811b6834aacb0d75f06765d0decf0.png" alt="1" data-base62-sha1="fgT4WsX72sfPUR2KdmPmlNMNFUA" width="690" height="344" data-dominant-color="161717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">962×480 16.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

kl_divergence: <p>The easiest is to put the entire model onto GPU and pass the data with batch size set to 1. If it doesn’t fit, then try considering lowering down your parameters by reducing the number of layers or removing any redundant components that might be taking RAM. From the screenshot, I see that the model fits, you need to lower your batch size to the minimum to find the threshold value.</p>

slavavs: <p>The model is placed at batch size = 50, 100, 200, 500 :). But the working time for the era does not change. With batch 50, the run time is 19 seconds; with batch 500, the run time is 18.7 seconds. It should be?</p>

slavavs: <p>As far as I understand, there is a memory inside the video card. My video card is 8GB. There are also processor cores. It seems to me that cores are responsible for parallel computing, the more there are, the faster the calculation. And the memory is just a buffer.</p>

slavavs: <p>For me, the most incomprehensible question is whether there will be an acceleration of work if I use not 1 GPU, but 4? Maybe someone has 4 GPUs, can I give my model to check the speed of work on 1 GPU and on 4?</p>

Arun_Kumar4: <p>my model size is 355mb i have predication time is 30sec so i want to reduce predication time please give any idea</p>
Fastest way to compute multiple outer products

Aesteban: <p>Given a 2d matrix of size (2000x1000) i need to compute the outer product of each row with itself. Finally all the outer products must be averaged. What is the fastest/most efficient way of doing so?<br>
This is by far the biggest bottle neck in my program. Ive come up with 2 solutions. The 2. of which, against all intuition, is way faster than the first. Any help to improve the speed of this computation will be greatly appriciated. The computation will be performed by the GPU if its of any relevance.</p>
<p>Slowest:</p>
<pre><code class="lang-auto">def avg_matrix_outer_products_v1(a):
    x_dim, y_dim = a.shape
    ourter_products = torch.matmul(a.view(x_dim, y_dim, 1), a.view(x_dim, 1, y_dim)).T
    return torch.mean(ourter_products , 2)
</code></pre>
<p>This method can run into memory issues but this is easily fixed by splitting “a” into submatracies with only 250-500 rows and calling the function multiple times.</p>
<p>Fastest:</p>
<pre><code class="lang-auto">def avg_matrix_outer_products_v2(a):
     x_dim = a.shape[0]
     ourter_products = torch.outer(a[0], a[0])
     for j in range(1, x_dim):
         ourter_products += torch.outer(a[j], a[j])

     return ourter_products / x_dim
</code></pre>

AlphaBetaGamma96: <p>If you have a matrix of size 2000 x 1000 and you won’t the outer product of the <code>i-th</code> row, you can use <code>torch.einsum</code>.</p>
<p>So, for example having a matrix that’s shape <code>[i,j]</code> doing <code>torch.einsum("ij-&gt;ijj", a)</code> will return the outer product for you.</p>

KFrank: <p>Hi Aesteban!</p>
<aside class="quote no-group" data-username="Aesteban" data-post="1" data-topic="134389" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e47774/48.png" class="avatar"> Aesteban:</div>
<blockquote>
<p>Given a 2d matrix of size (2000x1000) i need to compute the outer product of each row with itself. Finally all the outer products must be averaged.</p>
</blockquote>
</aside>
<p>The calculation you are asking for can be performed with a single<br>
matrix multiplication:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; def avg_matrix_outer_products_v2(a):
...      x_dim = a.shape[0]
...      ourter_products = torch.outer(a[0], a[0])
...      for j in range(1, x_dim):
...          ourter_products += torch.outer(a[j], a[j])
...      return ourter_products / x_dim
...
&gt;&gt;&gt; a = torch.randn (2000, 1000)
&gt;&gt;&gt; outA = avg_matrix_outer_products_v2(a)
&gt;&gt;&gt; outB = a.T @ a / a.shape[0]
&gt;&gt;&gt; outB.allclose (outA, atol = 1.e-7)
True
</code></pre>
<p>(And, yes, I gave up on your <code>_v1</code> version because it was annoying<br>
slow and memory inefficient.)</p>
<p>Best.</p>
<p>K. Frank</p>

Aesteban: <p>Thanks alot! Works really well! My program just gained a 10x speed up lol.</p>
Tutorial link on mask rcnn is broken, where can I find colab version

dato_nefaridze: <p>hello, tutorial about mask rcnn on pytorch tutorials is broken:<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.9.1+cu102 documentation</a>,<br>
when i click colab version it doesn’t work, it seems that it has problems. please help me to find colab version.</p>

ptrblck: <p>I also can’t access the notebook anymore and don’t know where it was moved to. Let’s check with the <code>torchvision</code> team.</p>

dato_nefaridze: <p>okay, i will wait to torchvision team (i needed that notebook for self study)</p>
Feedig images of different size to vision transformer

Mona_Jalal: <p>I have (not natural) images (~250 images) that I do some sort of preprocessing on, and after reducing their size (they are in range of gigapixels), I end up with images such as the following sizes:</p>
<p>For example, one of my features.pt is torch.Size([2231, 512]) and another is torch.Size([399, 512])</p>
<p>So, when I am feeding these images that are in train dataloader, do I need to normalize them to a specific size, and what size should that be? Or am I fine the way images are? The second item in torch.size is always 256.</p>
<p>Here’s a related post but I thought to create a separate question.</p><aside class="quote quote-modified" data-post="1" data-topic="134480">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/cuda-out-of-memory-when-using-vision-transformer/134480">CUDA out of memory when using vision transformer</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I am getting CUDA out of memory when using vision transformer. I have changed my batch size from 8 to 1 and still get the same error: 
attn_weights = torch.matmul(q, k.transpose(-2, -1)) / self.scale

RuntimeError: CUDA out of memory. Tried to allocate 1.15 GiB (GPU 0; 10.76 GiB total capacity; 8.33 GiB already allocated; 1.13 GiB free; 8.49 GiB reserved in total by PyTorch) 
This is the self attention class: 
class SelfAttention(nn.Module):
    def __init__(self, in_dim, heads=8, dropout_rate=0…
  </blockquote>
</aside>
weights not passed to optimiser are trained when nn.ModuleList is used

user7: <p>when I run a regular model, and pass only one layer to the optimiser, it behaves as expected and only one layer is trained. but when training a model that contains <code>nn.ModuleList</code>, even parameters not passed to the optimiser are trained.</p>
<pre><code class="lang-auto">
    class testModel(nn.Module):
      def __init__(self):
        super(testModel, self).__init__()
        self.layers = nn.ModuleList([nn.Linear(3,3)]*2)
      def forward(self, x):
        x = self.layers[0](x)
        x = F.relu(x)
        x = self.layers[1](x)  
        return x
    
    t = testModel()
    x = torch.ones((1,3)) * -1
    target = torch.ones((1,3)) * 200
    
    w1_b4_train= t.layers[0].weight.data.numpy().copy()
    w2_b4_train= t.layers[1].weight.data.numpy().copy()
    
    opt = optim.Adam(t.layers[1].parameters(), lr = 0.01)
    loss = nn.MSELoss()
    
    for i in range(10):
      res = t(x)
      opt.zero_grad()
      l = loss(res, target)
      l.backward()
      opt.step()
    
    w1_after_train = t.layers[0].weight.data.numpy().copy()
    w2_after_train = t.layers[1].weight.data.numpy().copy()
    
    print(np.array_equal(w1_b4_train,w1_after_train))
    print(np.array_equal(w2_b4_train,w2_after_train))
</code></pre>
<p>this code returns:</p>
<pre><code class="lang-auto">    False
    False
</code></pre>
<p>why is that? is there a way to fix this?</p>

ptrblck: <p>You are creating references in:</p>
<pre><code class="lang-python">[nn.Linear(3, 3)]*2
</code></pre>
<p>such that you are in fact initializing a single linear layer and are just repeating it.<br>
Use <code>[nn.Linear(3, 3) for _ in range(2)]</code> to create new objects.<br>
Here is a small example:</p>
<pre><code class="lang-python"># use references
layers = nn.ModuleList([nn.Linear(3,3)]*2)
# parameters are equal
print(layers[0].weight)
print(layers[1].weight)

# inplace manipulations changes both parameters
with torch.no_grad():
    layers[0].weight.fill_(0.)
print(layers[0].weight)
print(layers[1].weight)

# create new modules
layers = nn.ModuleList([nn.Linear(3,3) for _ in range(2)])
# different
print(layers[0].weight)
print(layers[1].weight)

with torch.no_grad():
    layers[0].weight.fill_(0.)
print(layers[0].weight)
print(layers[1].weight)
</code></pre>
Csv file different delims

PhysicsIsFun: <p>Greetings!</p>
<p>I need help cleaning a csv file. It has 6 columns of floats. The problem: If the next float in a row has a minus sign, the delimiter between the two floats is 3 spaces. If the next number is positive (no sign), the delimiter is 4 spaces. I have no idea how to read this file properly into a Python object (array, data frame tensor?). Or how to rewrite the file first with consistent delimiters as the read functions already ask for delimiters…</p>
<p>Any help?</p>
<p>PiF</p>
<p>EDIT: I just solved this problem using the Linux shell program vim. To replace a single string with another string is a single command there. Wonder whether Python would be as easy as well.</p>

ptrblck: <p>Good to hear you’ve solved the issue. I think <code>pandas</code> is a great tool to clean and read data frames, so it could be useful in the future.</p>
libcublassLt missing from link instruction(s) during cmake --build

paul2021: <p>Not sure what category this belongs in (build??) and I see other references to the subject library but not in this regard.  During build, some links are failing with a number of the following type of message:</p>
<p>/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmulAlgoInit@libcublasLt.so.11’</p>
<p>It seems that this library, introduced after CUDA 10.0, is not included in the link, e.g.,</p>
<p>/usr/bin/c++  -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-deprecated-copy -Wno-uninitialized -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG  -rdynamic test_cpp_c10d/CMakeFiles/ProcessGroupGlooAsyncTest.dir/ProcessGroupGlooAsyncTest.cpp.o  -o bin/ProcessGroupGlooAsyncTest -L/opt/conda/lib -Wl,-rpath,/opt/pytorch/build/lib:/opt/conda/lib:/usr/local/cuda/lib64  lib/libtorch_cpu.so  lib/libc10d_cuda_test.so  lib/libgtest_main.a  -lpthread  lib/libtorch_cuda.so  lib/libc10_cuda.so  /usr/local/cuda/lib64/libcudart.so  /opt/conda/lib/libnvToolsExt.so  lib/libprotobuf.a  lib/libc10.so  -lmkl_intel_lp64  -lmkl_gnu_thread  -lmkl_core  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.so  -lm  /usr/lib/x86_64-linux-gnu/libdl.so  lib/libdnnl.a  -ldl  -Wl,–no-as-needed,"/opt/pytorch/build/lib/libtorch_cpu.so" -Wl,–as-needed  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/lib/x86_64-linux-gnu/libcudnn.so  lib/libgtest.a  -pthread</p>
<p>adding the library results in success</p>
<p>If there’s a proper way to do this, I’m all ears, otherwise, FYI.</p>
Making Transfer Learning work just right

jimmiemunyi: <aside class="onebox allowlistedgeneric" data-onebox-src="https://jimmiemunyi.github.io/blog/tutorial/2021/10/17/Making-Transfer-Learning-Work-Pyorch.html">
  <header class="source">
      <img src="https://jimmiemunyi.github.io/blog/images/favicon.ico" class="site-icon" width="48" height="48">

      <a href="https://jimmiemunyi.github.io/blog/tutorial/2021/10/17/Making-Transfer-Learning-Work-Pyorch.html" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 17 October 2021">Jimmie Munyi’s Research Blog – 17 Oct 21</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://jimmiemunyi.github.io/blog/tutorial/2021/10/17/Making-Transfer-Learning-Work-Pyorch.html" target="_blank" rel="noopener nofollow ugc">Making Transfer Learning work in PyTorch</a></h3>

  <p>Tips to effectively use Transfer Learning in the Vision Domain.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Hey good people. Here is a post about tips and tweaks you can employ to make transfer Learning work just right.</p>
<p>Most blog posts I have read about this topic just suggest changing the last linear layer and freezing all other parameters but there is more tweaks you can try</p>
<p>E.g<br>
Discriminative Learning Rates, Don’t Freeze Batch Norm layers, Unfreezing the model after a few epochs, using a custom and better head (classifier)</p>
Calling loss.backward() multiple times vs. aggregating losses

ZimoNitrome: <p>I have some code that uses multiple loss functions and therefore calls loss.backward(retain_graph=True) multiple times.</p>
<p>I heard that doing only a single backward pass can be more efficient (time-wise). Should I in that case sum all the losses or average them?</p>
<p>And is there any difference between calling backward on each loss separately aggregating them?</p>

soulitzer: <p>Calling loss.backward multiple times on different losses is equivalent to summing the individual losses first and backwarding a single time on the summed loss. Averaging them will mean your gradients will be dampened by a factor of n. It is the same if you don’t care about the gradients wrt to any of the losses individually, and yeah it will be more efficient depending on how much of the graph the losses share.</p>
<p>A better way is passing in multiple losses to <code>torch.autograd.backward</code> at once (see <a href="https://pytorch.org/docs/stable/generated/torch.autograd.backward.html#torch.autograd.backward" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.backward — PyTorch 1.9.1 documentation</a>) . It also does the same thing but avoids performing the unnecessary sum.</p>
Concatenation the output of Bert and transformer

Omama: <p>I want to feed two models with different features, then how I can concatenate the output of BERT dense layer with the output of the transformer, then pass the concatenated output to other fully connected layer, then to the softmax layer?</p>
<p>In other words, how to train ensemble pytorch models in two different data loaders</p>
<p>def train(epoch):<br>
tr_loss = 0<br>
n_correct = 0<br>
nb_tr_steps = 0<br>
nb_tr_examples = 0<br>
model.train()<br>
for _,data in enumerate(training_loader3, 0):<br>
ids = data[‘ids’].to(device, dtype = torch.long)<br>
mask = data[‘mask’].to(device, dtype = torch.long)<br>
targets = data[‘targets’].to(device, dtype = torch.long)<br>
outputs = model(ids, mask)<br>
loss = loss_function(outputs, targets)<br>
tr_loss += loss.item()<br>
big_val, big_idx = torch.max(outputs.data, dim=1)<br>
n_correct += calcuate_accu(big_idx, targets)</p>
<pre><code>    nb_tr_steps += 1
    nb_tr_examples+=targets.size(0)
    
    if _%100==0:
        loss_step = tr_loss/nb_tr_steps
        accu_step = (n_correct*100)/nb_tr_examples 
        print(f"Training Loss per 100 steps: {loss_step}")
        print(f"Training Accuracy per 100 steps: {accu_step}")

    optimizer.zero_grad()
    loss.backward()
    # # When using GPU
    optimizer.step()

print(f'The Total Accuracy for Epoch {epoch}: {(n_correct*100)/nb_tr_examples}')
epoch_loss = tr_loss/nb_tr_steps
epoch_accu = (n_correct*100)/nb_tr_examples
print(f"Training Loss Epoch: {epoch_loss}")
print(f"Training Accuracy Epoch: {epoch_accu}")

return 
</code></pre>
<h1>
<a name="training-the-bert-model-1" class="anchor" href="#training-the-bert-model-1"></a>Training the BERT model.</h1>
<p>for epoch in range(EPOCHS):<br>
train(epoch)</p>
How to perform distribute model based on pytorch using send and recv?

Maxwell_Albert: <p>Here is a problem about how to implement model-parallel on pytorch.<br>
Since I have read tuturial that it uses very simple instructions, here are codes</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.model_a = Model_A().to('cuda:0')
        self.model_b = Model_B().to('cuda:1')
    def forward(self, x):
        x = self.model_a(x.to('cuda:0'))
        x = self.model_b(x.to('cuda:1'))
        return x
</code></pre>
<p>However, I do not have many GPUs. I want to use <code>torch.distribute.send</code> and <code>torch.distribute.recv</code> in forward and backward to build model-parallel. But how can I do it? Is there any tutoral?</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.conv1 = nn.Conv2d(1,32,3,padding = 1)
        self.pool1 = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(32,64,3,padding = 1)
        self.pool2 = nn.MaxPool2d(2,2)
        self.conv3 = nn.Conv2d(64,128,3,padding = 1)
        self.pool3 = nn.MaxPool2d(2,2)
        
        self.fc1 = nn.Linear(128*3*3,625)
        self.fc2 = nn.Linear(625,10)
          
    def forward(self,x,rank):
        if rank == 0:
            x = self.pool1(F.relu(self.conv1(x)))
            x = self.pool2(F.relu(self.conv2(x)))
            x = self.pool3(F.relu(self.conv3(x)))
            dist.send(x,1)
        elif rank == 1:
            dist.recv(x,0)
            x = x.view(-1,128*3*3)
            x = F.relu(self.fc1(x))
            x = self.fc2(x)
       
        return x
</code></pre>
<p>I could use <code>rank</code> and <code>send</code> and <code>save</code> in forward. But how can I change backward?</p>
How to extract features from pre-trained FasterRCNN pytorch?

Maariiee: <p>Hi,<br>
I need some help.<br>
I would like to use the pre-trained model Faster from pytorch package :</p>
<p>import torchvision<br>
import torch<br>
from torchvision.models.detection import fasterrcnn_resnet50_fpn<br>
model = fasterrcnn_resnet50_fpn(pretrained=True)</p>
<p>Faster RCNN has 2 outputs : (label, bbox) for each Region that it has selected. I would like to extract, for each region, the vector from the last fully connected layer. The vector where I can find all the probabilities of belonging to the different possible categories. So the vector just before the softmax …</p>
<p>For example, with Coco dataset (91 categories), I would like to have one vector [1,91] for each region.</p>
<p>I have read a lot of solution for VGG, Resnet … but nothing about pre trained faster model.<br>
Can someone help me ?<br>
Thanks</p>
False positives hard hats

fpalsson: <p>I’m new to this, so please bare with me. If this is the wrong topic please let me know.</p>
<p>I’ve tried training hard hat detection using both of these datasets:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://universe.roboflow.com/universe-datasets/hard-hat-universe-0dy7t">
  <header class="source">

      <a href="https://universe.roboflow.com/universe-datasets/hard-hat-universe-0dy7t" target="_blank" rel="noopener nofollow ugc">Roboflow</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:500/332;"><img src="https://storage.googleapis.com/roboflow-platform-sources/dqDSqgYXUkTAVngTJseP/yrm8YXKcEK4p9iqXRcvy/original.jpg" class="thumbnail" width="500" height="332"></div>

<h3><a href="https://universe.roboflow.com/universe-datasets/hard-hat-universe-0dy7t" target="_blank" rel="noopener nofollow ugc">Hard Hat Universe Object Detection Dataset</a></h3>

  <p>Download 7035 free images labeled with bounding boxes for object detection.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>https//universe.roboflow.com/joseph-nelson/hard-hat-workers/3</p>
<p>The problem I’m seeing is that I get A LOT OF detections of hard hats/helmets for normal heads with no helmet. There are a lot more incorrect classifications than correct. I suspect that is due to the fact that all annotations for helmets include the complete head as well.</p>
<p>I guess there are hundres of ways to screw up the training. I’ve been using this template to train, and just replaced the dataset:</p>
<p><a href="http://colab.research.google.com/drive/1gDZ2xcTOgR39tGGs-EZ6i3RTs16wmzZQ" rel="noopener nofollow ugc">colab.research.google.com/drive/1gDZ2xcTOgR39tGGs-EZ6i3RTs16wmzZQ</a>  (link removed as I was not allowed to post multiple links)</p>
<p>Can anyone give any guidance?</p>
Scr_mask size in Transformers for Language Model

sleepybear: <p>The <code>src_mask</code> has been used to prevent Transformers from cheating! Then in language model task, the size of <code>src_mask</code> (or in source code of pytorch <code>attn_mask</code>) should be same as the length of the <code>seq_len x seq_len</code> (Is this utterance true?). However I find the toy example of Pytorch (<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">example link</a>) which generates this mask as size of <code>batch_size x batch_size</code>. Which one is true? If the latter is true, can you explain me why we have to generate mask by size <code>batch_size x batch_size</code>?</p>
Fcn_resnet101 for segemenation

Neelam_Jain: <p>Hello,</p>
<p>I am new in using models. I have used fcn_resnet101  for images. I am trying to generate the mask for person in the image. However, the edges of mask are not proper.  I have tried with multiple values of confidence. But there is always some issue with the edges<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0ee03bb1e03c81fb8cc651d3ed45f47dc2dbb55.jpeg" alt="test2-enhancement9" data-base62-sha1="tOhfYHnbDLSSLTSYMMMTnW95kQ5" width="341" height="342"></p>
<pre><code>tensor_int = read_image(str(Path(image)))

batch_int = torch.stack([tensor_int])
batch = convert_image_dtype(batch_int, dtype=torch.float)
#model = fcn_resnet50(pretrained=True, progress=False)
model = fcn_resnet101(pretrained=True, progress=False)
</code></pre>
<h1><a name="model-wide_resnet101_2pretrainedtrue-progressfalse-1" class="anchor" href="#model-wide_resnet101_2pretrainedtrue-progressfalse-1"></a>model = wide_resnet101_2(pretrained=True, progress=False)</h1>
<pre><code>model = model.eval()
normalized_batch = F.normalize(batch, mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))
output = model(normalized_batch)['out']

sem_classes = [
</code></pre>
<p>‘<strong>background</strong>’, ‘aeroplane’, ‘bicycle’, ‘bird’, ‘boat’, ‘bottle’, ‘bus’,<br>
‘car’, ‘cat’, ‘chair’, ‘cow’, ‘diningtable’, ‘dog’, ‘horse’, ‘motorbike’,<br>
‘person’, ‘pottedplant’, ‘sheep’, ‘sofa’, ‘train’, ‘tvmonitor’<br>
]<br>
sem_class_to_idx = {cls: idx for (idx, cls) in enumerate(sem_classes)}<br>
normalized_masks = torch.nn.functional.softmax(output, dim=1)<br>
print(normalized_masks.size())</p>
<h1><a name="printnormalized_masks015shape-2" class="anchor" href="#printnormalized_masks015shape-2"></a>print(normalized_masks[0][15].shape)</h1>
<pre><code>person_masks = [
    normalized_masks[img_idx, sem_class_to_idx[cls]]
    for img_idx in range(batch.shape[0])
    #for cls in ['__background__']
    for cls in ['person']
]
</code></pre>
<p>Is there anything which can be done for better edges?</p>
Why are huge batch sizes used for pretraining and small ones for finetuning?

BramVanroy: <p>In most, if not all papers on language models, I find that they often use very large batch sizes for pretraining on a language modeling task. But when they then finetune their model to show its performance on downstream tasks, the batch sizes are suddenly very small.</p>
<p>For instance, the <a href="https://arxiv.org/pdf/1907.11692.pdf" rel="noopener nofollow ugc">RoBERTa paper</a> shows that its batch size during pretraining was 8k sentences (Table 9 in the appendix), however for finetuning the batches are considerably smaller (Table 10, appendix): 16 (RACE), 48 (SQuAD), 16, 32 (GLUE).</p>
<p>This has puzzled me since forever and I have never discovered the rationale behind this. Is it a matter of scale? Something like: while pretraining you have so much different data, that you just want as much in one go as you can - it does not matter as much that the loss is smoothed out (averaged) over such huge batches. But when finetuning over a smaller dataset you do not want to average the loss over too much of the dataset at once because you then lose peculiarities of samples quickly.</p>
<p>Or is there another reason? All ideas are welcome.</p>
The impact of padded images in Vision transformers and efficient nets

Oussama_Bouldjedri: <p>I am working with images with different input sizes, as far as I understood Vision transformers and efficient nets, work mainly on specific input sizes 224, 384,…<br>
I was wondering what would be the effect of padded images on vision transformers (all the types of this research direction including the ones based only on attention concept and the variation that uses both convolutions and attention concept ) and on efficient nets?</p>