akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Special shape dataloader

username: Kurtis_Petitboeuf
cooked: <p>Hello. I am trying to prepare some data for training and testing.</p>
<p>I am running in a problem as most of the tutorials i’ve seen are built around image datasets but i’ve got some custom data that i want to use for the Neural Network now…</p>
<p><strong>Explanations</strong></p>
<p>Basically, let’s say i have 5 CSV files (data1,2,3,4,5) with data and 5 CSV files (state1,2,3,4,5) with states.</p>
<p>The network should use the corresponding data file to guess the value that is written in the state file.</p>
<p>The data files contain 2 columns, X and Y (timeseries) of 100 points each.</p>
<p>The state files only contain 1 column and 1 data, (kept or not kept (0 or 1))</p>
<p><strong>The issue i have now</strong></p>
<p>I can succesfully create a dataloader and input tensors that i create after the CSV files.</p>
<p>I’ll get 3 tensors, 1 for DataXvalues, 1 for DataYvalues, 1 for State</p>
<p>I create a dataLoader like this</p>
<p>myDataLoader = torch.utils.data.DataLoader((myXTensor,myYTensor,myStateTensor))</p>
<p><strong>And it seems to work good, but…</strong></p>
<p>But i only have the stuff i need for 1 train or test pass now, and i don’t see how im supposed to fit the 4 other sets i should add next! (Data2,3,4,5 and State2,3,4,5)</p>
<p>When i lookup my DataLoader the data in it is somewhat like this</p>
<p>[0] - Data for X<br>
[1] - Data for Y<br>
[2] - State</p>
<p>Should i just add a lot of tensors and i could itterate doing jumps of 3 with other sets of data (is that a setting in the DataLoader?) (I would do one pass with Input 0,1 Output 2 - Next pass with Input 3,4 Output 5 - The next one with 6,7,8…)?</p>
<p>If there’s more info required i’ll give all the details i can!</p>

username: Kurtis_Petitboeuf
cooked: <p>Update on this -</p>
<p>What i am planning to do is a List for Timeseries Data and a List of States</p>
<p><strong>So i would itterate over the lists to secure the proper values for each pass</strong> (1 full Timeseries X + Y and the Kept/Not Kept state)</p>
<p>Does this seem like a solution that would work?</p>
<p>Also, i would be treating the XY values of the Timeseries like this</p>
<p>All the X (times) first and then underneat all the Y (Values), <strong>Does this make sense?</strong></p>
Loading pytorch model

username: sammy007
cooked: <p>i am trying to load a model in pytorch<br>
but getting following error</p>
<pre><code class="lang-auto">import torch
from models import micro_models
model = micro_models.NetworkCIFAR
model.load_state_dict(torch.load("model.pth"))
</code></pre>
<p>last line is showing following error<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
TypeError: load_state_dict() missing 1 required positional argument: ‘state_dict’</p>

username: Alexey_Demyanchuk
cooked: <p>Hi. Make sure, that <code>torch.load("model.pth")</code> actually return dict object and not something else (perhaps, <code>None</code>)</p>

username: sammy007
cooked: <pre><code class="lang-auto">dt = torch.load("/home/ml/Desktop/workspace/nsga-net-master-trial/search-GA-BiObj-micro-20201201-132957/arch_1/model.pt")
&gt;&gt;&gt; type(dt)
&lt;class 'collections.OrderedDict'&gt;
</code></pre>
<p>it returns OrderesDict</p>

username: Alexey_Demyanchuk
cooked: <p>Ok. I tried to provide <code>load_state_dict()</code> with incompatible dict object and I get different error (“Error(s) in loading state_dict”)<br>
It is a snippet:</p>
<pre><code class="lang-auto">model = torchvision.models.resnet18()
od = OrderedDict() 
torch.save(od, 'tmp.pt')
model.load_state_dict(od)
model.load_state_dict(torch.load('tmp.pt'))
</code></pre>
<p>The only way I can reproduce the error “TypeError: load_state_dict() missing 1 required positional argument: ‘state_dict’” is actually to call this function without any arguments like this <code>model.load_state_dict()</code></p>
<p>What is your pytorch version, by the way?</p>

username: Alexey_Demyanchuk
cooked: <aside class="quote no-group" data-username="sammy007" data-post="1" data-topic="104715">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sammy007/48/31779_2.png" class="avatar"> sammy007:</div>
<blockquote>
<p>File “”, line 1, in<br>
TypeError: load_state_dict() missing 1 required positional argument: ‘state_dict’</p>
</blockquote>
</aside>
<p>Also, I noticed that error came from line 1 of some file “”, complaining no argument is provided to <code>load_state_dict()</code>. While I can’t look into that file it is hard to tell any further <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: sammy007
cooked: <p>i resolved it. i had not created instance of class model. we need to create instance of class model and then call function</p>
<pre><code class="lang-auto">model = micro_models.NetowrkCIFAR(24, 5, 11, False, genotype)
model.load_state_dict(torch.load("model.pth"))
</code></pre>
<p>those are the some of the parameters we need to pass to model class to create instance<br>
that line was missing before<br>
Thank you for your help</p>
Use not all GPUs in each server

username: sakh251
cooked: <p>Hi,<br>
I have two servers with 3 GPUs each. I can run my code when I use all GPUs on servers (6 GPUs). I want to make benchmark by using 2 GPUs on each (4 GPUs) and 1 GPUs on each server (2 GPUs).</p>
<pre><code class="lang-auto">ngpus_per_node = 1 # or can be 2 or 3
args.world_size = ngpus_per_node * args.world_size # 2  (for 2 machine) is sent to for world_size
</code></pre>
<p>when I use all GPUs on each machine it works fine, but by less than it the code stuck in following line without any error:</p>
<pre><code class="lang-auto">model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[gpu])
</code></pre>

username: PistonY
cooked: <p>You could use <code>os</code> to set which device python could use.</p>
<pre><code class="lang-python">import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'
</code></pre>

username: Wangaroo
cooked: <p>Or you can set environment variables before running the code.</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0,1 python train.py
</code></pre>
<p>Set <code>CUDA_VISIBLE_DEVICES</code> to the gpu index you want to use.</p>
Cuda out of memory error. Why do my tensors use too much memory?

username: soojin99
cooked: <p>Hello.<br>
I think I’ve done wrong. It uses too much memory.</p>
<p>Before training I made sure any memory isn’t allocated,<br>
but it dies saying<br>
RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 7.93 GiB total capacity; 6.30 GiB already allocated; 25.75 MiB free; 6.78 GiB reserved in total by PyTorch)</p>
<p>I do not know why 6.3GB is alreay allocated.</p>
<p>Is there something I am doing wrong?<br>
This is my training function.</p>
<p>‘’’<br>
def train(input, target):<br>
# initializing grad and hidden state<br>
hidden = model.init_hidden()<br>
optimizer.zero_grad()<br>
loss = 0</p>
<pre><code>for i in range(input.size()[0]):
    output, hidden = model(input.data[i].unsqueeze(0), hidden)
    l = criterion(output, target.data[i])
    loss += l
    output.detach()

# backpropagation
loss.backward()

# optimizing
optimizer.step()

# learning rate scheduling
scheduler.step()

return loss
</code></pre>
<p>‘’’</p>
<p>this is my gru unit</p>
<p>‘’’<br>
import torch.nn as nn<br>
import torch</p>
<p>class GRU(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size, output_size, batch_size, device, num_layers):<br>
super(GRU, self).<strong>init</strong>()<br>
self.hidden_size = hidden_size<br>
self.num_layers = num_layers<br>
self.batch_size = batch_size<br>
self.gru = nn.GRU(input_size, hidden_size, num_layers, batch_first=False, dropout=0.2)<br>
self.elu = nn.ELU()<br>
self.fc = nn.Linear(hidden_size, output_size)<br>
self.device = device<br>
self.to(device)</p>
<pre><code>def forward(self, x, h):
    out, h = self.gru(x, h)
    out = self.elu(h[0])
    out = self.fc(out)
    return out, h


def init_hidden(self):
    hidden = torch.zeros(self.num_layers, self.batch_size, self.hidden_size).to(self.device)
    return hidden
</code></pre>
<p>‘’’</p>

username: ptrblck
cooked: <p>The losses will be accumulated to <code>loss</code> in the first loop and with them the complete computation graphs.<br>
It’s thus expected that your memory usage would grow and your GPU might run out of memory.<br>
You could have a look at <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">this post</a> and see if another approach would be suitable for your use case.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Multi-class Semantic segmentation format

username: Ciaran1981
cooked: <p>Hello All,</p>
<p>I am trying to adapt my binary semantic segmentation code to multi-class semantic-seg.</p>
<p>I am having trouble figuring out how to convert my training masks to an appropriate format/shape.</p>
<p>My training labels are a single band image (grayscale) where unique integer (eg 1,2,3,4) values represent a label. How should this be transformed to the correct format?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>For a multi-class segmentation you would most likely use <code>nn.CrossEntropyLoss</code>, which expects the target masks to have the shape <code>[batch_size, height, width]</code> and contain the class indices in the range <code>[0, nb_classes-1]</code>.<br>
If your current targets are grayscale images with discrete gray values for each class, you could map these values to class indices e.g. by using a lookup table.</p>
Different results based on cudatoolkit version

username: snowe
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I just noticed that when I train a network in an anaconda environment with cudatoolkit=10.2.89, I get different results to when I train the exact same network in a different environment with cudatoolkit=10.1.243 (everything else is the same). Is this behaviour to be expected?</p>
<p>I seed everything and I can reproduce the results within each environment, they are just different from each other…</p>
<p>Any help is very much appreciated!</p>
<p>All the best<br>
snowe</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am afraid this is expected. Results (especially on CUDA) are reproducible only for a fix hardware/library version and if the deterministic flag is set to true on pytorch.</p>
<p>If you change anything there, the floating point arithmetic order can lead to different floating point results. These differences then get amplified by the model depth and gradient descend, leading to a final result that is completely different (even though, if your model is stable, the final loss value should be similar).</p>

username: snowe
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, thank you for your response!</p>
How to calculate the norm over a 2d matrix?

username: John_Price
cooked: <p>I have a distance matrix D with 2 dimensions from which I want to calculate the norm (||D||). Is it possible to use torch.norm(D) still having the two dimensions (same shape like before calculation) after the calculation? I saw that it is possible to calculate either the norm over the rows or over the colomns.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Which norm are you looking for here?<br>
Depending on what you’re looking for, you’ll need to set both dim and p to the right values.</p>

username: John_Price
cooked: <p>hi, first of all thanks for yout help. Can you pls tell me what p is?</p>

username: albanD
cooked: <p>You can check the doc for details on each argument: <a href="https://pytorch.org/docs/stable/generated/torch.norm.html?highlight=norm#torch.norm">https://pytorch.org/docs/stable/generated/torch.norm.html?highlight=norm#torch.norm</a></p>
<p>But as you will see there. This function can do many types of norms so you first need to know which norm you want <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: John_Price
cooked: <p>so p is float in my case. and I want to do the .norm() operation over the float entries of a 2d matrix (E, C) in which every entry is represening a distance. The shape (E, C) of the resulting matrix should not change.</p>

username: albanD
cooked: <p>I am not sure to follow what you mean. The norm of a matrix is a single number. Which norm are you looking for here? Euclidean, frobenius, … ?</p>

username: John_Price
cooked: <p>okay thank you! I was quite unsure about that. the matrix holds euclidean distances in each entry. so you mean that I can not have a matrix as a result. the result should be a scalar value?</p>

username: albanD
cooked: <p>If these are already euclidean distances, then you don’t need the norm function?<br>
Which value of p do you use for the distance computation?</p>

username: John_Price
cooked: <p>I’ve forgotten to mention, that I took th.cdist() for the distance computation between cluster centers and data points (image patches) which is Dx-ci^dct in the following formula: -1/2h^2 * ||Dx - ci^dct||² . So you mean that I do not need to calculate the norm here in the formula with th.norm(x) where x = Dx - ci^dct because I already have the eucledian distances?</p>

username: albanD
cooked: <p>cdist already computes the p-norm distance.<br>
Distance and norms are very similar in simple spaces.</p>

username: John_Price
cooked: <p>okay, thank you very much.</p>
Mask R-CNN optimizer and learning rate scheduler

username: JamesDickens
cooked: <p>In the Mask R-CNN paper the optimizer is described as follows training on MS COCO 2014/2015 dataset for instance segmentation (I believe this is the dataset, correct me if this is wrong)</p>
<blockquote>
<p>We train on 8 GPUs (so effective minibatch<br>
size is 16) for 160k iterations, with a learning rate of<br>
0.02 which is decreased by 10 at the 120k iteration. We<br>
use a weight decay of 0.0001 and momentum of 0.9. With<br>
ResNeXt [45], we train with 1 image per GPU and the same<br>
number of iterations, with a starting learning rate of 0.01.</p>
</blockquote>
<p>I’m trying to write an optimizer and learning rate scheduler in Pytorch for a similar application, to match this description.</p>
<p>For the optimizer I have:</p>
<pre><code class="lang-auto">def get_Mask_RCNN_Optimizer(model, learning_rate=0.02):
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=0.0001)
    return optimizer
</code></pre>
<p>For the learning rate scheduler I have:</p>
<pre><code class="lang-auto">def get_MASK_RCNN_LR_Scheduler(optimizer, step_size):
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=step_size, gammma=0.1, verbose=True)
    return scheduler
</code></pre>
<p>When the authors say “decreased by 10” do they mean divide by 10? Or do they literally mean subtract by 10, in which case we have a negative learning rate, which seems odd/wrong. Any insights appreciated.</p>
Indexing for group lasso

username: GeoffNN
cooked: <p>I’m trying to implement a Group Lasso penalty on a batched input of shape <code>(batch_size, *)</code>, given groups made of lists of coordinates.</p>
<p>As a simple example, if I have a batch of 3 grey-scale images of shape (3, 4, 2), I would define my groups as</p>
<pre><code class="lang-auto">    [(2, 0), (2, 1), (3, 0), (3, 1)],
    [(0, 2), (0, 3), (1, 2), (1, 3)],
    [(2, 2), (2, 3), (3, 2), (3, 3)]]
</code></pre>
<p>Then, I’d like to get the batch-wise group lasso penalty of shape <code>(batch_size,)</code>. Each coordinate of this corresponds to one datapoint. For datapoint 0, this would be<br>
<code>torch.norm( [x[0, 0, 0], x[0, 0, 1], x[0, 1, 0], x[0, 1, 1] + x[0, group2] ...</code>.</p>
<p>I can’t figure out how to index x to get this.</p>
<p>As a second part, I’d like the groups to be defined starting from the last dimensions.<br>
Supposing here that <code>x</code> has shape <code>(batch_size, n_channels, M, N)</code>, I want to be able to define my groups only using the last 2 dimensions, and sum the values over the channels.</p>
How could I merge list or synchronize amound processes

username: coincheung
cooked: <p>Hi,</p>
<p>I am using distributed mode to train my model, the launch command is like this:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=4 train.py
</code></pre>
<p>In theory, this would start 4 processes.<br>
In my program, each processes would generate a list of strings:</p>
<pre><code class="lang-auto">process 1: a = ['a', 'b', 'c']
process 2: a = ['1', '2', '3']
...
</code></pre>
<p>I need to merge the lists in to one whole list and share them among the processes, so that after this operation, each process has a list whose content is:</p>
<pre><code class="lang-auto">a = ['a', 'b', 'c', '1', '2', '3']
</code></pre>
<p>How could I do this please ?</p>
<p>By the way, I noticed that there is a function named <code>torch.cuda.synchronize()</code>, will this function ensure that all the processes are synchronized at this line, or is this only ensure the synchronization of the backend operation without considering the python frontend operations ?</p>
How can I backward different grads to different inputs in one operation?

username: Chaofei
cooked: <p>In my work, I need to back-propagate different gradient values to different inputs in one operation (e.g. add, matmul, conv).<br>
For example, for y = torch.matmul(A, X), I can get grad_y_a and grad_y_x, and I want to backprop grad_y_a to A, and grad_y_x to X.<br>
For now, I can get around it by repeating the operation:</p>
<pre><code class="lang-auto">y_a = torch.matmul(A,X.detach())
y_b = torch.matmul(A.detach(), X)
</code></pre>
<p>But this approach wasted  a lot of computation, is there a better way to do this?<br>
Many thanks!</p>
Preparing dataset using Torchtext for Hierarchical Attention Network

username: shahriarshayesteh
cooked: <p>Hi all,</p>
<p>I am going to use a dataset from Toxic Comment Classification Challenge that contains comments as the data and level of toxicity as the labels. In order to do that, I have a plan to use  HAN architecture but my problem arises when I am going to make the dataset ready for this architecture using torchtext data loader. I grabbed the code for this from another forum on this website and when I run it on the data it takes forever to run TabularDataset.splits. Is there any advice you can give me to help me to prepare the data for my project?</p>
<p>Thanks,</p>
<pre><code>from torchtext.data import Field,NestedField,get_tokenizer

from spacy.tokenizer import Tokenizer

def tokenizer1(text):

 nlp = spacy.load('en_core_web_sm')

 doc = nlp(text)

 return [sent.text for sent in doc.sents]

 nlp = spacy.load('en_core_web_sm')

 tokenizer = Tokenizer(nlp.vocab)

TEXT = Field(sequential=True,tokenize=get_tokenizer("basic_english"), lower=False)

LABEL = Field(sequential=False, use_vocab=False)

nesting_field = NestedField(nesting_field = TEXT,include_lengths = True, tokenize=tokenizer1)


from torchtext.data import TabularDataset

r1 = "data dir"

tv_datafields = [('id',None),('comment_text', nesting_field),('severe_toxic', LABEL)]

data11 = TabularDataset.splits(

           path=r1, # the root directory where the data lies

           train='train1.csv',

           format='csv',

           skip_header = True,# ADDING THIS PART AND ZERO INDEX

           fields=tv_datafields)[0]</code></pre>
How to append a tenor to another torch:: Tensor

username: risignsun
cooked: <p>Hi, Experts,</p>
<p>I am trying to append a tensor to another tensor in pytorch it is:<br>
data.append(datum.numpy())<br>
labels.append(label)</p>
<p>In libtorch, I use<br>
data.append=datum;<br>
labels.append=label1;</p>
<p>And it shows errors . Any comments or suggestions ? Thanks.</p>

username: ptrblck
cooked: <p><code>append</code> is not a method of the tensor, but a Python <code>list</code> in your example.<br>
In C++ you can create a <code>std::vector&lt;Tensor&gt;&amp; tensors</code> and use <code>torch::stack(tensors)</code> instead.</p>

username: risignsun
cooked: <p>Thanks, that is very helpful</p>
Position invariant perceptual loss

username: ZimoNitrome
cooked: <p>Is there any way to get positional-invariant perceptual loss?<br>
i.e. these image pairs would receive a low loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee44f9bd1747f920774d5efcfece56a21f40671c.png" alt="cf746e57d06a_bE9Y400tb0" data-base62-sha1="xZPpSq4NFpWuvexFUF6ZfCqVr9G" width="84" height="84"> compared to <img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/967426b845545920f812c376071e5997ab5f64eb.png" alt="087b12a73b83_B44L536Iso" data-base62-sha1="lsYtm1rcZHnMWrrMnOzR7O1Vi5B" width="84" height="84"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee44f9bd1747f920774d5efcfece56a21f40671c.png" alt="cf746e57d06a_bE9Y400tb0" data-base62-sha1="xZPpSq4NFpWuvexFUF6ZfCqVr9G" width="84" height="84"> compared to <img src="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8db1ed999c6e1126519a6099d395dbad3ddf7d2.png" alt="eb1c9f04e639_1MMupMjNHt" data-base62-sha1="uWoB1p9gtQFdU0ylTzEHt1UJks2" width="84" height="84"></p>
<p>Because they contain the same objects.</p>
<p>But this image would receive a high loss:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee44f9bd1747f920774d5efcfece56a21f40671c.png" alt="cf746e57d06a_bE9Y400tb0" data-base62-sha1="xZPpSq4NFpWuvexFUF6ZfCqVr9G" width="84" height="84"> compared to <img src="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1ab084130290ec4f793121816dda8c992119efc1.png" alt="503fd7c3498a_OThVRmItUJ" data-base62-sha1="3O6B44WWgJGTSCTb7zT525210pX" width="84" height="84"></p>
<p>Since the objects are not the same.</p>
<p>Experimentation of the above images show that none of the layers in VGG19 has this property:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/0/8017af9189e344a541d6e56b0c2cf3b9383177e5.png" alt="Code_SqgKSTLCtv" data-base62-sha1="ih9RuLaneGD94OtMrMNO0ipFbnv" width="530" height="370"></p>
Will PyTorch free intermediate tensors from GPU after function returns?

username: wasabi
cooked: <p>Hi guys, I want to put some structure on the weight matrix so I have something like this in the forward function.</p>
<pre><code class="lang-auto">class myNN(nn.Module):
    def __init__(self):
        super(myNN, self).__init__()
        self.weight = nn.Linear(a, b)
    def transform(self):
        transformed_weight = do_something(self.weight)
        return transformed_weight
    def forward(self, x):
        transformed_weight = self.transform()
        x = do_something2(transformed_weight, x)
        return x
</code></pre>
<p>do_something() and do_something2() create some intermediate tensors (also transformed_weight in forward() is also intermediate), will they be freed after forward() returns? I keep getting CUDA out of memory error after several iterations. Thanks.</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="wasabi" data-post="1" data-topic="104663">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wasabi/48/13151_2.png" class="avatar"> wasabi:</div>
<blockquote>
<p>do_something(self.weight)</p>
</blockquote>
</aside>
<p>It is weird to do_something() on modules, it is only correct to transform module parameters (but it is ok if you only use a module as a container). Otherwise, maybe you should use stuff from nn.functional instead.</p>
<p>Intermediate tensors are only released before backward() if they’re not needed for gradient computations; generally, multipliers are needed, but summands are not (thus inplace summations usually work).</p>
How to speed up the process

username: Johnson_Mark
cooked: <p>Hello all, I have a feature vector size of <code>128x5</code> and a corresponding index label <code>128x1</code>. For example,</p>
<pre><code class="lang-auto">f =[f1, f2, f3, f4, f5]
labels =[2, 3, 2, 1, 2]
</code></pre>
<p>that means <code>fi</code> size of 1x5, and corresponding to label is <code>j</code> (i.e. f1, f3, and f5 have a label is 2, f2 has label 3</p>
<p>In my task, I want to compute the average feature of a label. For example for label 2 we have</p>
<pre><code class="lang-auto">f(1) = f4 
f(2) = (f1+f3 + f5)/3
f(3) = f2
</code></pre>
<p>Currently, my implementation loop all labels and compute the average, but the speed is very slow (when the size of the label increases to 5 Million). How should I speed up it? Thanks.</p>
<pre><code class="lang-auto">    num_freq_label = torch.zeros(len(labels))
    for ind in range (len(labels)):
         ind_label = (labels == ind).nonzero()
         f[ind:ind+1, ...] +=  torch.sum(f[ind_label], dim=0)
         num_freq_label[ind] +=ind_label.size(0)
    f = torch.div(f, num_freq_label.view(-1,1))
</code></pre>
How to use a .mat file with all my data in pytorch

username: Haya_Busa
cooked: <p>First of all, sorry if the question was answered before but I couldn’t find anything specific to my case (noob here).</p>
<p>I have a .mat file where I have a matrix 100x100x10000, representing 10000 different images of 100x100.<br>
I wanted to use this .mat file in pytorch and create a dataset of it so then I could use it in DataLoader.</p>
<p>I wanted to test this dataset in a GAN architecture that was provided to me but I got stuck right on step one. I’m really giving my first steps here… So, sorry for that.<br>
Any help would be much appreciated!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I guess you can use tools like <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html">https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html</a> to load the mat file into a numpy array and then move that to pytorch (copyless) with <code>torch.from_numpy(your_array)</code>.</p>
Why we need DECLARE/DEFINE/REGISTER_DISPATCH mechanism？

username: ckmufeng
cooked: <p>As we can use native_functions.yaml to define dispatch， why we need DECLARE/DEFINE/REGISTER_DISPATCH mechanism at the same time？</p>

username: albanD
cooked: <p>Hi,</p>
<p>These two have different purposes.<br>
The native_functions.yaml allows you to select the device and in general works at the level of the Dispatcher.<br>
The dispatch macro is a trick closer to the old TH macros that allow to write code that is generic for different dtype as well!</p>
Using data_parallel in libtorch with nested modules

username: melspec
cooked: <p>I have been following the mnist_parallel example (<a href="https://github.com/pytorch/pytorch/issues/19540" rel="noopener nofollow ugc">19540</a>)for using the data_parallel call.  My module actually has nested modules, similar to how torchvision implemented GoogLeNet.  The inception layers are actually a separate module.  When I do this, the call to data_parallel crashes.  If I move the inception (<a href="https://github.com/pytorch/vision/blob/master/torchvision/csrc/models/googlenet.cpp" rel="noopener nofollow ugc">can be seen here</a>) layer out into the main module, then the sequential call fails with something similar to</p>
<blockquote>
<p>weight of size [16, 96, 1, 1], expected input[128, 16, 15, 15] to have 96 channels, but got 16 channels instead</p>
</blockquote>
<p>The question is, how do I properly implement nested modules in libtorch when I am going to be making the call to data_parallel?  I apologize for not posting the code for my net, its a bit big and ugly, but it started life very close to the torchvision GoogLeNet implementation.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="melspec" data-post="1" data-topic="103979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/67e7ee/48.png" class="avatar"> melspec:</div>
<blockquote>
<p>I apologize for not posting the code for my net, its a bit big and ugly, but it started life very close to the torchvision GoogLeNet implementation.</p>
</blockquote>
</aside>
<p>Could you try to write a short proxy code, which could reproduce this issue?<br>
I’m not sure where the error is coming from when similar models seem to work.</p>

username: melspec
cooked: <p>Apologies for the delay.  I got it sorted out.  The rules seem to be:</p>
<ol>
<li>You cannot you Sequential unless you plan to purposefully shard your model.  It cannot be used in a model with a generic data_parallel call (where you replace your forward call with data_parallel).  If used with that call you will get errors indicating that the weights are not on the correct GPU, like the one above.</li>
<li>You may use nested modules, however they also need to be wrapped in Cloneable, just like the main module.</li>
</ol>
Can gradient back-propagate through circular padding?

username: 111319
cooked: <p>When using zero-padding mode in Conv, the paddings are leaf nodes, so the gradient will not back-propagate through the padding to the previous layer.</p>
<p>What about circular or reflect mode? In these two modes, the padding values are the sliced results of the previous layer, so that the gradient can back-propagate to previous layers.</p>
<p>I am not familiar with Cuda, could anyone tell me what the implementation is in Pytorch?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes the gradients will flow back to the input corresponding to everywhere it is used (including padding).</p>
Rejecting Blank Frames

username: abhiray92
cooked: <p>Hi Everyone,</p>
<p>I am performing video classification using CRNN model. I have extracted the frames from the videos and subtracted their background. Which has created alot of blank frames for leading and trailing number of frames. I want to reject the blank frames. Can anyone suggest a suitable method to reject the blank frames?</p>
<p>Thanks</p>
Incomplete training dataset of NN

username: srl123
cooked: <p>Hi all,<br>
I have a (huge) theoretical question connected with a regression problem of neural network:  what approach should I follow if my training dataset is incompleted and or my measurements are affected by strong biases?</p>
<p>let’s assume I want my net to predict the Y variable given the X,W, Z variables. Unfortunately I know that Y depends also on another variable, K, of whom I have no records. Of course the training dataset is essential to produce a “optimal” net, but I am wondering if there is any approach that considers, I don’t know, the inclusion of a stochastic variable that may take into account this lack of data.</p>
<p>Connected with the previous: what if my training dataset is not exact? how can I consider the uncertainty related to my measurements in training my net?</p>
<p>thank you!</p>

username: qmeeus
cooked: <p>Assuming that K is related to X, W, Z (no conditional independence: <code>P(K, D) = P(K | D) * P(D)</code>), then it should not be a problem:</p>
<pre><code class="lang-auto">P(Y, K | D) = P(Y | K, D) * P(K, D) = P(Y | K, D) * P(K | D) * P(D)
</code></pre>
<p>where the first and second terms in the right-most side of the equation are estimated by your model and K is a hidden variable.</p>
<p>If the K variable cannot be predicted from the data (i.e. <code>P(K, D) = P(K) * P(D)</code> or equivalently <code>P(K | D) = P(K)</code>) then your model will be missing a variable. It does not mean that it will not manage to predict anything good but rather that it does not have all the information needed to predict your target variable correctly.</p>

username: srl123
cooked: <p>correct, thank you for your answer so you’r saying train with what you have. And what about the uncertainty related to the input measurements?</p>

username: qmeeus
cooked: <p>There are methods to account for uncertainty in the training data. If you can quantify it or estimate, then you can modelize the error but this has more to do with statistics  and with statistical analysis in general than with neural networks (I mean, the answer with DNNs will not be different than for any predictive model).<br>
Also, you should know whether it’s your inputs, your outputs or both that are biased. Unfortunately, if your outputs are biased, then your model will be biased. If your inputs are biased, then your model might learn to cope with that (to some extend of course: the expression “garbage in, garbage out” still holds)</p>

username: srl123
cooked: <p>Can you provide the name of some of these statistical analyses you’r talking about? thanks</p>

username: qmeeus
cooked: <p>Here is one example that can give you a good starting point: <a href="https://en.wikipedia.org/wiki/Errors-in-variables_models" rel="noopener nofollow ugc">https://en.wikipedia.org/wiki/Errors-in-variables_models</a></p>
Input for torch.nn.gumbel_softmax

username: namespace_pt
cooked: <p>Say I have a tensor named <code>attn_weights</code> of size [1,a], entries of which indicate the attention weights between the given query and |a| keys. I want to select the largest one using <code>torch.nn.functional.gumbel_softmax</code>.</p>
<p>I find <a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=gumbel%20softmax#torch.nn.functional.gumbel_softmax" rel="noopener nofollow ugc">docs about this function</a> describe the parameter as <em><strong>logits</strong></em> <em>- […, num_features] unnormalized log probabilities</em>. I wonder <strong>whether should I take <code>log</code> of <code>attn_weights</code> before passing it into <code>gumbel_softmax</code>?</strong> And I find Wiki defines <code>logit=lg(p/1-p)</code>, which is different from barely logrithm. I wonder which one should I pass to the function?</p>
<p>Further, I wonder <strong>how to choose <code>tau</code></strong> in <code>gumbel_softmax</code>, any guidelines?</p>

username: ptrblck
cooked: <p>Based on the example code snippet in the docs it seems that “unnormalized log probabilities” would refer to raw logits, so no <code>log</code> or <code>softmax</code> should be applied, if I understand it correctly.</p>

username: namespace_pt
cooked: <p>ok I got it. Thank you very much. I’ll check out the origin paper.</p>
Video classification for varying length of videos

username: abhiray92
cooked: <p>Hi,<br>
I have gone through almost all the articles for performing the said activity, even used the <code>collate_fn </code> and custom collate functions, but could not make it work. I was wondering if someone can help, it would be a life saver.</p>
<p>My directory structure is something like -</p>
<pre><code class="lang-auto">root / 
    |_ABCD/
           |_  ABCD_1.jpg
           |_  ABCD_2.jpg
    |_EFGH/
           |_EFGH_1.jpg
           |_EFGH_2.jpg
           |_EFGH_3.jpg
</code></pre>
<pre><code class="lang-auto">Labels are mentioned in the following way - 
ABCD |  Basketball
EFGH |  Football
</code></pre>
<p>My code for loading data is -</p>
<pre><code class="lang-auto"># for CRNN
class Dataset_CRNN(data.Dataset):
    "Characterizes a dataset for PyTorch"
    def __init__(self, data_path, folders, labels, frames, transform=None):
        "Initialization"
        self.data_path = data_path
        self.labels = labels
        self.folders = folders
        self.transform = transform
        self.frames = frames

    def __len__(self):
        "Denotes the total number of samples"
        return len(self.folders)

    def read_images(self, path, selected_folder, use_transform):
        X = []
        for i in self.frames:
            image = Image.open(os.path.join(os.path.join(path, selected_folder, '{}_{}.jpg'.format(selected_folder,i))))
            if use_transform is not None:
                image = use_transform(image)

            X.append(image)
        X = torch.stack(X, dim=0)

        return X

    def __getitem__(self, index):
        "Generates one sample of data"
        # Select sample
        folder = self.folders[index]

        # Load data
        X = self.read_images(self.data_path, folder, self.transform)     # (input) spatial images
        y = torch.LongTensor([self.labels[index]])                  # (labels) LongTensor are for int64 instead of FloatTensor

        # print(X.shape)
        return X, y
</code></pre>

username: ptrblck
cooked: <p>What are you trying to achieve, what is not working, and where are you stuck at the moment? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: abhiray92
cooked: <p>I have solved this, but I’m stuck with a different issue now. I’ll post it in a different thread.<br>
Thanks alot!</p>
How do we pack the data as input for enumerate?

username: Skybender153
cooked: <p>I have a cifar-10 input data of size x_train : (50000,3072), y_train : (50000,). I wanted to use the x_train, y_train and pack them into the trainloader function which uses a batch size of 100, so that when I call as follows:</p>
<p>for batch_idx, (inputs, targets) in enumerate(trainloader):<br>
print(inputs.shape)<br>
print(targets.shape)</p>
<p>Output:<br>
(100,32,32,3)<br>
(100,)</p>
<p>I wasted a lot of time in doing so, Can some one help me with this?</p>

username: pchandrasekaran
cooked: <p>You could write a basic custom dataset and use that with a dataloader.</p>
<pre><code class="lang-auto">class DS(torch.utils.data.Dataset):
    def __init__(this, X=None, y=None, mode="train"):
        this.mode = mode
        this.X = X #Maybe do reshaping here
        if mode == "train":
            this.y = y

    def __len__(this):
        return this.X.shape[0]

    def __getitem__(this, idx):
        if this.mode == "train":
            return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx]) #or torch.FloatTensor(this.y[idx]) depending on use case
        else:
            return torch.FloatTensor(this.X[idx])

tr_data_setup = DS(X_train, y_train.reshape(-1,1)) #
trainloader = torch.data.utils.DataLoader(tr_data_setup, batch_size=100, ......)
</code></pre>
<p>You could also expand this to perform augmentations on the image if necessary.</p>

username: Skybender153
cooked: <p>Thank you that solved it.</p>