akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How I make a gradient flow via one-hot vector?

username: I_H_Yoo
cooked: <p>Hi.</p>
<p>I’m implementing Label Smoothing for Transformer but the gradient of my model didn’t flow. What is a problem of my model?</p>
<pre><code class="lang-python">def cal_loss(pred, gt, ignore_idx, smoothing=None):
    '''
    Calculate cross-entropy loss between very before layer of softmax and ground truth(gt)
    param:
        pred: predicted value (before applying softmax)
        gt: grount truth
        ignore_idx: index that will be ignored when calculating its probability
        smoothing: if float is passed, then apply label smoothing
    
    shape:
        pred: [Batch_size, (Seq_len-1), Tgt_vocab_size]
        gt: [Batch_size, (Seq_len-1)]
    '''
    if smoothing != None:
        confidence = 1 - smoothing
        target_vocab_size = pred.size(2)

        # generate one-hot vector through the probability that model generates
        one_hot = torch.zeros_like(pred, requires_grad=True).scatter(2, gt.unsqueeze(2), 1) # [Batch_size, (Seq_len-1), Tgt_vocab_size]
        one_hot = one_hot * confidence + (1 - one_hot) * smoothing / (target_vocab_size - 2) # ['SOS', 'PAD']
        log_prob = nn.functional.softmax(one_hot, dim=2)
        
        non_pad_mask = gt.ne(ignore_idx) # where gt != ignore_index
        # NLL-Loss: $\sum_x log(p_{theta}(y \rvert x)) q(w)
        loss = - (one_hot * log_prob).sum(dim=2) # [Batch_size, (Seq_len-1)]
        loss = loss.masked_select(non_pad_mask).mean() # gather all values where mask == True and return 1-D array
    else:
        loss = nn.functional.cross_entropy(pred, gt, ignore_idx=ignore_idx, reduction='sum')
    return loss
</code></pre>
Custom dataloader with lists and tensors

username: k2lin
cooked: <p>Hi all,</p>
<p>I am working on a project involving images and point clouds.<br>
However, the point cloud tensor each has different lengths and other inputs have the same lengths.<br>
I know it is possible to have a custom collate function, but is it possible to use the default one for the inputs with the same length and a custom one for variable length?<br>
Thanks.</p>
Error training in double precision

username: hesana
cooked: <p>Hello,</p>
<p>I have a 3D-CNN which I would like to train using double precision float on my CPU.<br>
After creating the model I turn it into a double precision one using model.double() but the forward pass fails with the error:</p>
<pre><code class="lang-auto">  File "/opt/anaconda3/envs/ann/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 480, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #2 'weight' in call to _thnn_conv3d_forward
</code></pre>
<p>However, if I check the type of the parameters and, in particular, of the layer that gives the error with:</p>
<pre><code class="lang-python"> for param in self.compressionConv.state_dict():
     print(self.compressionConv.state_dict()[param].dtype)
</code></pre>
<p>I get a long list of torch.float64 which, I assume, is what I want.</p>
<p>Is there anything else I need to do to make the model use double precision.</p>
<p>Thanks!<br>
Helios</p>

username: albanD
cooked: <p>Hi,</p>
<p>Did you properly convert the input to double as well by doing <code>input = input.double()</code>? (this is <strong>not</strong> an inplace operation on Tensors, you have to use the result).</p>

username: hesana
cooked: <p>Hi,</p>
<p>great, that fixed it!<br>
I didn’t do it because I was using instead</p>
<pre><code class="lang-python">torch.set_default_dtype(torch.float64)
</code></pre>
<p>and I understood that the effect of that was to make torch always use double precision tensors. If not, what is that command doing?</p>
<p>Thanks a lot!<br>
Helios</p>

username: albanD
cooked: <p>It does unless you specify ortherwise <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Doing <code>torch.tensor([1, 2])</code> will still give you a long type for example.</p>
Tensorflow datasets

username: seliad
cooked: <p>I have a tensorflow dataset (as numpy generator) with all the preprocessing required.</p>
<p>What is the best way to convert it into torch.utils.data.Dataset and later torch.utils.data.Dataloader?</p>

username: albanD
cooked: <p>Hi,</p>
<p>When you mean a numpy generator, you mean that it already contains a sampler? Or can you extract all the data from it beforehand?<br>
To build a dataset, you need to be able to access a particular element, you cannot do it with only a generator I’m afraid.</p>
<p>Note that you can still bypass the Dataset/Dataloader completely if you don’t need them and just add a couple <code>torch.from_numpy()</code> in your training loop beore passing the data to your model!</p>
Training with missing labels

username: Yoni_Keren
cooked: <p>Hi!</p>
<p>I’m training a net with a multi-task problem, let’s say that the input is a facial image, and the 2 losses are cross entropy on gender and cross entropy age (age having 101 different classes) so that total_loss=loss_age+loss_gender</p>
<p>The problem is that some images have a missing label, eg. an image when only an age is given, or just the gender.<br>
I know that I should set the loss to 0 (or any constant) for a missing label, but I am not sure if I should manually write a new loss/criterion or if there’s an elegant way to write that in pytorch? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> How would you go about it?</p>

username: aplassard
cooked: <p>First, you should treat age as a regression problem unless there is a good reason you need to do classification.</p>
<p>Second, you can simply index into the output of your models and select the rows where you have an output, then calculate the loss only on those rows.</p>

username: Yoni_Keren
cooked: <p>Classification works better for age estimation  (take a look at any age estimation paper or the recent papers from cvpr).</p>
<p>Also, my original suggestion of setting the loss to 0 or a constant is way way cleaner than your suggestion. My question was is there a more elegant way.</p>

username: justusschock
cooked: <p>Regarding the underlying mathematics, it is no difference between setting the loss to zero and not calculating/backpropagating it at all, since a loss value of zero means, that there are no mistakes and thus no gradient update is necessary (which is the same result if you don’t calculate a loss for this part at all).</p>
<p>On the implementation side you could use negative values to indicate missing variables and thus simply do something like</p>
<pre><code class="lang-auto">pred[target&lt;0] = target[target&lt;0]
</code></pre>
<p>and later on just calculate the loss as you did before. This would result in a loss-value of 0 for the considered prediction/data sample as the prediction and the target would be same-valued.</p>

username: Yoni_Keren
cooked: <p>Nice, thanks justusschok <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: addisonklinke
cooked: <p><a class="mention" href="/u/justusschock">@justusschock</a> Good point with zero loss not contributing to backprop. In newer versions of PyTorch, <code>nn.CrossEntropyLoss</code> has an <code>ignore_index</code> parameter which applies your logic internally to save a bit of code. For instance, no matter what the <code>pred</code> tensor is, the loss will always be zero if <code>target</code> contains only zeroes</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

criterion = nn.CrossEntropyLoss(ignore_index=0)
pred = torch.randn(3, 5)
target = torch.zeros(3, dtype=torch.long)
criterion(pred, target)
</code></pre>
Mutiplication of lists of matrices

username: KENG
cooked: <p>I have two lists of matrices. The first one has shape [(N, n1), (N, n2), (N, n3)]. The second of has shape [(n1, B), (n2, B), (n3, B)]. I would like to multiply the two lists to obtain [(N, B), (N,B), (N, B)]. Is it possbile to do this in Pytorch without looping through the lists?</p>
<p>Thank you very much!</p>

username: LeviViana
cooked: <p>You can pad these matrices with zeros and concatenate them to form one big tensor. An then, you’ll be able to perform a normal matrix multiply, and slice the result to get the corresponding sub-matrices results. However, I’m not sure it is the best way to do this, it will depend on the values of n1, n2, etc. Maybe looping is better.</p>
<p>If you really need performance, and if you have big matrices, you’ll actually need to implement your own CUDA kernel, and this I think is the best you can do.</p>

username: KENG
cooked: <p>Thanks for your reply! In my cases, n1, n2…may differ by a lot. So I think zero padding may not be efficient. Since I also want the autogradient in Pytorch, to implement CUDA kernels may not be easy.  I guess for loops may be the only choice.</p>
Rearranging Alexnet Pretrained Network

username: terro
cooked: <p>I am a beginner and I am trying to create a new model using the layers of the pretrained network AlexNet. I would like to remove the avgpool layer, Add normalization layers in between the convolutional layers and add addtional fully connected layer to “classifier” part.</p>
<pre><code class="lang-auto">class MyAlexNet(nn.Module):
    def __init__(self, my_pretrained_model):
        super(MyAlexNet, self).__init__()
                 self.conv = my_pretrained_model.features
                 self.my_new_layers = nn.Sequential(nn.Linear(1000, 100),
                                        nn.ReLU(),
                                        nn.Linear(100, 6))

    def forward(self, x):
        x = self.pretrained(x)
        x = self.my_new_layers(x)
        return x


pretrained = models.alexnet(pretrained=use_pretrained)        
model_ft= MyAlexNet(pretrained)
</code></pre>
<p>I suppose I have to use a structure as above but I do not how and if it is possible to add the before cited layers</p>
Object detection, evaluate funct from engine.py

username: azer
cooked: <p>Hi,</p>
<p>I have converted my dataset to coco format to produce a mask rcnn model. I use the one epoch and evaluate functions from engine.py. My results are ok but I would like to know if there is a way to get the AP in a list to further produce a plot AP per epoch without going all the way to coco api.</p>
<p>Any ideas would be appreciated.</p>
Model computation time measurement when using Torchscript in C++ (reposted)

username: Jungmo_Ahn
cooked: <p>Hello</p>
<p>In order to measure the computation time of the deep learning model when using GPU, we have to keep in mind about GPU operations asynchronicity. Thus, we should use torch.cuda.Event or torch.autograd.profiler in Python.</p>
<p>I found torch::autograd::profiler in libtorch but there is no API documentation to know how to use functions in the library. This looks like C++ version of torch.autograd.profiler which to measure model computation time. However, there are many examples to measure computation time in Python but it is hard to find any example to measure computation time when using GPU in C++.</p>
<p>(This might be because the majority of users except me are familiar to use C++ and C++ library. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20">)</p>
<p><strong>Could someone provide a simple example to measure computation time of Torchscript model considering both CPU time and GPU time in C++?</strong></p>
<p>FYI, in order to correctly measure computation time in Python3, please refer this link:</p><aside class="quote" data-post="1" data-topic="26964">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daulbaev/48/1824_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964">How to measure time in PyTorch</a> 
  </div>
  <blockquote>
    I have seen lots of ways to measure time in PyTorch. But what is the most proper way to do it now (both for cpu and cuda)? 
Should I clear the memory cache if I use timeit? 
And is it possible to get accurate results if I’m computing on a cluster? And is it a way to make this results reproducible? 
And what is better: timeit or profiler?
  </blockquote>
</aside>

<p>Many thanks!</p>
<ul>
<li>I copied this question <a href="https://discuss.pytorch.org/t/model-computation-time-measurement-when-using-torchscript-in-c/93218">here</a></li>
</ul>

username: Jungmo_Ahn
cooked: <pre><code class="lang-auto">std::chrono::steady_clock::time_point begin = std::chrono::steady_clock::now();
model.forward(inputs).toTensor();
cudaDeviceSynchronize(); // declared in cuda.h
std::chrono::steady_clock::time_point begin = std::chrono::steady_clock::now();
</code></pre>
<p>This is my solution to measure elapsed time for model computation when using GPU.<br>
If there is a better way, plz suggest me.</p>
<p>Thanks!</p>
c10::Error: Couldn't find an operator for aten::dropout(Tensor input, float p, bool train)

username: Jonson
cooked: <p>I got an error with <code>c10::Error: Couldn't find an operator for aten::dropout(Tensor input, float p, bool train)</code> when I tried to load a torchscript model by libtorch c++ api. The linking libtorch library is compiled with <code>script/build_pytorch_android.sh</code> script.</p>
<pre><code class="lang-auto">target platform: android arm64-v8a
pytorch tag: v1.6.0
</code></pre>
<p>And if I linked the program with x86 prebuilt libtorch library, and run it in x86 platform, all worked well.</p>
Non-numerical input - how to implement and how does it work?

username: Tomash
cooked: <p>Hi,<br>
I am working on some regression problem and I have a question about implementation of algorithm where as an input we don’t have a single number, because we know it only with some standard deviation (or more with the distribution similar to gaussian).<br>
For example as a label I don’t have: 1 but the maximum is 1 and minimums are in 0.5 and 1.5.<br>
Is it possible (and if yes- how) to pass an input like a distribution (for example gaussian) or a number with uncertainty. As an output we can have a single number.<br>
Anyone?</p>

username: googlebot
cooked: <p>Sure, just train on random samples, e.g. x=torch.distributions.Normal(m,s).rsample()</p>

username: Tomash
cooked: <p>But then I need to train every element of data few times (as much as number of samples). I mean for every label e.g. 1 I will make distribution built of 100 elements, so also I need to pass labeled data 100 times for every step of iteration, so for example if I work on pictures datasets - every single picture will be processed 100 times with random label. Am I right?</p>

username: googlebot
cooked: <p>As you are already doing stochastic optimization, using one sample per element per iteration is not too different from optimizing usual static inputs - every new element (random dataset row) feeds a different value to the same input “slot” anyway. It is true that you may need more epochs due to sampling.</p>
<p>There is an approach to generate multiple samples, also called “particles”, and process them together, like an expanded batch, but it is mainly used with probabilistic/variational modeling, where densities are used explicitly.</p>
Self contained simple example of Meta Learning

username: kl_divergence
cooked: <p>I am looking for a simple self-contained implementation of MAML with current Pytorch version. Almost all the implementations I’ve come across rely on older versions, library specific (for ex. TorchMeta). For simplicity, I want to obtain query and support set from same distribution. I want to mimick learning of fast weights and how gradients of it are transferred to original model.</p>
Tensor to numpy

username: mohammed_guermal
cooked: <p>hi,</p>
<p>I have a problem when going from tensor to numpy for instance when i use this :<br>
‘’‘transforms.ToTensor()’‘’<br>
then i conver my tensors to arrays to plot them i get an image where i have the same image repeated<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3b0634741dfa5cee96f76bed97833880f5db993.jpeg" alt="cap.PNG" data-base62-sha1="ucGEDlia4jlCSVb6x5wmvT1TFIf" width="486" height="205"></p>
<p>however when i use “” ((torch.from_numpy(image_1))“”" I get the correct answer.</p>
<p>My problem now is that my Neural network output a tensor and when i convert it to numpy to plot it i get the error above</p>
<p>anyone have an idea how I can solve this ???</p>
<p>to transform tensor to numpy i use .cpu().numpy()</p>

username: ptrblck
cooked: <p>Most likely you are reshaping/permuting in a wrong way.<br>
Usually these kind of interleaved outputs are created, if you wanted to swap some dimensions and have used <code>view/reshape</code> instead of <code>permute</code>.<br>
Could this be the case?</p>

username: mohammed_guermal
cooked: <p>thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>  I tried permute and it worked better thanks A lot</p>
Loss suddenly bumps and becomes const

username: nitsan57
cooked: <p>Hi,<br>
I am training a cnn that outputs an image.<br>
My loss goes down well, and then it jumps and keep ± constant.<br>
Did someone saw this phenomenom?<br>
It happens on the first or second epoch.<br>
I tried to use Adam , SGD, and also changed the activaitions with no luck.<br>
The only thing that helps is to learn less conv filters.<br>
Someone has an idea?<br>
Thanks,<br>
Nitsan</p>

username: ptrblck
cooked: <p>I’ve seen similar behavior in the past, where a “bad update step” produced large gradients and pushed the model parameters out of the “nice” loss landscape.<br>
I think lowering the learning rate helped, but I would also assume that e.g. Adam should reduce this issue.<br>
Anyway, you could check the gradients for the update steps and compare their magnitude, in case you want to debug your model.</p>

username: nitsan57
cooked: <p>Hi,<br>
Thanks for the replay<br>
I just want to update that using AMSGrad  seems to resolve the issue.<br>
Thanks again.</p>
Create a dataset subset with independent indexing

username: kpatil
cooked: <p>I want to create a subset of a dataset which has its own indexing, i.e. following <code>range(len(subset))</code>.</p>
<p>When I create a subset  with provided utility and iterate over the subset then the indexing followed is that of the original dataset.</p>
<pre><code class="lang-auto">n= len(dataset)
indices = np.random.choice(n, int(n*0.5))
subset= torch.utils.data.Subset(dataset, indices)

loader = DataLoader(subset, batch_size=100, shuffle=True, num_workers=0)

for batchid, (inputs, labels, index) in enumerate( iter(loader)):
    assert(np.max(np.array(index)) &lt; len(subset)) # this assert is invoked

</code></pre>

username: ptrblck
cooked: <p>Your created <code>indices</code> will contain random indices in the range <code>[0, len(dataset)]</code> and will not be clipped by the length of the subset.<br>
I assume that the returned <code>index</code> value is the <code>index</code> passed to <code>__getitem__</code>, so the behavior is expected.</p>
<p>If you want to pass indices in the range <code>[0, len(subset)]</code>, you could directly pass these indices to <code>Subset</code> using <code>torch.arange(int(n*0.5))</code>.</p>
Loss function for reconstruction and its last layer

username: aaks
cooked: <p>I am trying to generate a colored image using the traditional encoder-decoder model. Since the pixels are normalized in the range [0,1], it makes sense to have a sigmoid in the last layer of the decoder. Now, for the same, if I have the normalized ground truth image, what should be my loss function? nn.MSE is one option, but I am not keen on using it (something to do with my problem statement). Is there another loss function criterion that I can use?</p>

username: ptrblck
cooked: <p>You could try to use e.g. <code>nn.BCELoss</code>, if it would fit your use case. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Extract image to smaller images in dataloader

username: r3a2t10
cooked: <p>How can I extract a image to smaller images in dataloader instead creating a new dataset that croped the image manually</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        img = Image.open(self.files[index % len(self.files)])
        img_lr = self.lr_transform(img)
        img_hr = self.hr_transform(img)

        return {"lr": img_lr, "hr": img_hr}
</code></pre>
<p>I want to extract the img(2048x2048) crop into four smaller images (1024x1024 each)</p>
<pre><code class="lang-auto"> __ __          __ __
|     |  =&gt;    |__|__| 
|__ __|        |__|__|
</code></pre>

username: ptrblck
cooked: <p>If you don’t want to apply the cropping inside the <code>Dataset.__getitem__</code>, you could use it in the <code>DataLoader</code> loop:</p>
<pre><code class="lang-python">for data, target in loader:
    # apply the transformation here
</code></pre>
<p>To create these four patches, you could either use <code>unfold</code> or <code>torchvision.transforms.functional.five_crop</code> and remove the center crop:</p>
<pre><code class="lang-python">data = torch.randn(1, 1, 2048, 2048)

patches = data.unfold(2, 1024, 1024).unfold(3, 1024, 1024)
print(patches.shape)
&gt; torch.Size([1, 1, 2, 2, 1024, 1024])
# reshape if needed

imgs = TF.five_crop(data, size=1024)
# remove center crop, imgs is a list, so you might want to `torch.stack` them
imgs = imgs[:4]
</code></pre>
Parameter dim in weight_norm with GRU

username: geistgesicht
cooked: <p>Hi,</p>
<p>I am wondering how to set the ‘dim’ parameter in ‘weight_norm’ when used with a GRU. My code:</p>
<blockquote>
<p>rnn1 = weight_norm(weight_norm(nn.GRU(input_size=3, hidden_size=4), name=‘weight_hh_l0’), name=‘weight_ih_l0’),</p>
</blockquote>
Is it necessary to load optimizer?

username: XiXiRuPan
cooked: <p>When I torch.load(pretrained_model). Is it necessary to load optimizer under I set a proper learning rate? Thanks for your reply.</p>

username: ptrblck
cooked: <p>If you would like to continue the training and stored the <code>state_dict</code>s of the model and optimizer, both should be reloaded to continue the training properly.</p>
Calculating IoU on bboxes coordinates

username: nobodykid
cooked: <p>Hi, I’m looking for a way to calculate intersection over union given the predicted and ground truth bbox.</p>
<p>I found that you can do it using <code>ops.boxes.box_iou</code> from torchvision (<a href="https://discuss.pytorch.org/t/box-intersection-over-union-returns-zero/62187" class="inline-onebox">Box intersection-over-union returns zero</a>), but it seems I can’t find the docs referencing to that function.</p>
<p>There’s also recommendation on <a href="https://discuss.pytorch.org/t/iou-pytorch-implementation/21473" class="inline-onebox">IOU pytorch implementation</a>, but they’re using classes when my current case is class-agnostic.</p>

username: Xuesong_Wang
cooked: <p>I found the same problem as well. Actually, you can use <code>ops.box_iou(output, target)</code> to calculate the IoU directly. <a href="https://pytorch.org/docs/stable/_modules/torchvision/ops/boxes.html#nms" rel="nofollow noopener">Source code page.</a></p>
PyTorch version compatibility with Libtorch iOS

username: mark_jimenez
cooked: <p><a class="mention" href="/u/xta0">@xta0</a> Just wanted to ask if there is any guidance for compatibility between torchscript generated from a specific version of PyTorch with the corresponding LibTorch version on iOS.</p>
<p>Should the PyTorch and LibTorch iOS version always match? (ex. PyTorch 1.5 generated torchscript should only be used on LibTorch 1.5)</p>
<p>Or should LibTorch work on any torchscript generated from a base version of PyTorch (say torchscript generated from PyTorch 1.5 should work for LibTorch 1.5 or greater unless otherwise specified)?</p>
<p>Thanks for any help!</p>
Wha is the difference between SWA torch.optim.utils_swa and torchcontrib.SWA?

username: sunggu_kyung
cooked: <p>I’m interested in SWA so I’m trying to use it, but I don’t know what to use. I don’t know the difference between the following blogs and docs swa.</p>
<p>which one is the lastest version of SWA torch.optim.utils_swa or torchcontrib.SWA and what is the difference?</p>
<p>blog: <a href="https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/" rel="nofollow noopener">https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/</a><br>
docs : <a href="https://pytorch.org/docs/stable/optim.html" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html</a></p>

username: ptrblck
cooked: <p>SWA was added into PyTorch <code>1.6</code> in <a href="https://github.com/pytorch/pytorch/pull/35032">this PR</a>, so it looks as if the contrib implementation was merged into the core. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>EDIT: <a href="https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging/">Here</a> is also the blog post.</p>
Kfold cross validation and random transformation

username: terro
cooked: <p>Good Morning,<br>
I should train my network performing 5 fold cross validation and train each fold for 25 epochs. I have to perform data augmentation with random rotation and random translation. Starting from a dataset of 500 images I create 10 new images for each image of the original dataset by randomly translating and rotating. I have followed the tutorial here <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a>. In each epoch the augmented dataset will be different. I would like to have the same augmented dataset for each epoch and for each fold but I do not know how I can do.</p>
<p>Thank you for your help</p>

username: ptrblck
cooked: <p>If you have to make sure the same randomly augmented images are used and each fold contains exactly the same samples, I would recommend to create this particular dataset in a separate script and store each fold as PyTorch tensors using <code>torch.save</code>.<br>
In your training script you could simply load the folds and train your model for the specified number of epochs.</p>
<p>Alternatively, it <em>should</em> be possible to properly seed the code and create the same folds in your training script, but you would have to be very careful and would have to verify each step to avoid accidentally calling into the random number generator (which would thus change e.g. the augmentation).</p>

username: terro
cooked: <p>I cannot create the particular dataset since I have not sufficient memory on the disk. The dataset is augmented on the fly. In order to seed the code is it sufficient to call for example np.random.seed(value) at the beginning  of each epoch?</p>

username: ptrblck
cooked: <p>You don’t need to create all augmented samples at once and could still use the lazy data loading approach to create the folds.</p>
<p>However, if you want to use the seeding approach, refer to the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> and note that slight changes, such as <a href="https://discuss.pytorch.org/t/does-a-dataloader-change-random-state-even-when-shuffle-argument-is-false/92569/4">moving or executing the validation loop</a> might call into the random number generator and could thus change the transformation and creation of the folds.</p>
How to deal with different variable length and real numbers

username: ygl
cooked: <p>Hi, so I have many long documents. The label is gender. For each sentence of document, I map it into an embedding space. So I have a [num_of_sentences, 768] vector for each doc. They could be [1, 768], could be [3000, 768]. The average is [1000, 768]</p>
<p>My question is if I want to feed this vector into an attention network using LSTM, etc, how do I pad it since the format is (seq_len, batch, input_size) while the length is extremely different and they are real numbers?</p>
How to pad a batch of documents?

username: Mainul
cooked: <p>Hello PyTorch experts:</p>
<p>Sentences and documents both can be variable length.</p>
<p>Lets say, we have following 2 docs:</p>
<pre><code class="lang-auto">doc1=[torch.tensor([1,2,3,4]),torch.tensor([4,5,6]),torch.tensor([7,5])]
doc2=[torch.tensor([1,2,3]),torch.tensor([4,5])]
</code></pre>
<p>(Here, each tensor is a sentence and each number in the tensor is an index of embedding matrix)</p>
<pre><code class="lang-auto">doc1=pad_sequence(doc1,batch_first=True)
doc2=pad_sequence(doc2,batch_first=True)

batch=pad_sequence([doc1,doc2],batch_first=True)
</code></pre>
<p>This will throw an error because the length of the longest sentence in the first document is greater than the length of the longest sentence in the second sentence.</p>
<p>So, we need to pad each sentence to the longest sentence of the batch.</p>
<p>One solution would be, find the length of the longest sentence in each batch and pad each sentence to that fixed length. But the pad_sequence function of PyTorch does not support that.</p>
<p>Am I missing something? Is there any other PyTorch way to do this?</p>

username: vdw
cooked: <p><code>pad_sequence</code> takes as input a list of tensors. However, you give it a list of list of tensors.</p>
<p><code>pad_sequence</code> can only pad all sequences within the same list of tensors (e.g., <code>doc1</code> and <code>doc2</code>) but not across multiple lists. You probably need to do this manually as you described: find the longest sentence among all documents and then pad all sentences accordingly.</p>

username: Mainul
cooked: <p>Would not it be nice to have padding function like this: pad list of list of tensors?<br>
or at least have the ability to pad to a maximum length. Because transforming a document into a tensor is a common thing in NLP.</p>

username: vdw
cooked: <p>I’m not sure if this is a practical enough use case. In general, you aim to minimize padding, which essentially boils down to padding only to the length of the longest sequence in a batch. And there are existing solutions like the <code>BucketIterator</code> that creates batch which sequence of almost always equal length. At least for RNNs, that’s the de-facto way to do it. For CNNs and Transformers, things might be a bit different.</p>
<p>If you pad all sequences with respect to the longest one across your whole dataset/corpus – that is, beyond the scope of a batch – you create many batches where all sequences have padding. That’s particularly bad if you have some few or just one very long sentence. Sure, with <code>pack_padded_sequences</code> you get around this but you still loss training speed when your sequences are unnecessarily long (again, at least for RNNs).</p>
How to convert at::tensor to std::vector<float>

username: jinchen62
cooked: <p>Hi,</p>
<p>I was not able to convert at:tensor <strong>t</strong> to std::vector <strong>v</strong>. What I used:</p>
<blockquote>
<p>std::vector&lt;float&gt; v(t.data&lt;float&gt;(), t.data&lt;float&gt;() + t.numel());</p>
</blockquote>
<p>Is there any way to do that?</p>
<p>Thanks,</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/jinchen62">@jinchen62</a><br>
Make your tensor contiguous. Then try this:<br>
std::vector v(t.data_ptr(), t.data_ptr() + t.numel());</p>

username: jinchen62
cooked: <p>Thanks for reply! I added</p>
<blockquote>
<p>t = t.contiguous();</p>
</blockquote>
<p>but still doesn’t work.</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/jinchen62">@jinchen62</a> Did you fill the <code>float</code> as the template type? like this?<br>
std::vector v(t.data_ptr(), t.data_ptr() + t.numel())<br>
Can you paste the error message here? I did not try on my computer, but this should work.</p>

username: jinchen62
cooked: <p><a class="mention" href="/u/glaringlee">@glaringlee</a> I think you need to add “\” before “&lt;” to print the type. I ran</p>
<blockquote>
<p>t = t.contiguous();<br>
std::vector&lt;float&gt; v(t.data_ptr&lt;float&gt;(), t.data_ptr&lt;float&gt;() + t.numel());</p>
</blockquote>
<p>Is it what you are saying?<br>
Actually there is no error, it just get stuck at this point, even I’ve waited for a long time.</p>

username: glaringlee
cooked: <p><a class="mention" href="/u/jinchen62">@jinchen62</a> oh, crab…sry…I did not realize the type is not get printed out.</p>
<pre><code class="lang-auto">torch::Tensor ten = torch::rand({12, 12}, torch::TensorOptions(torch::kCPU).dtype(at::kFloat)); 
std::vector&lt;float&gt; v(ten.data_ptr&lt;float&gt;(), ten.data_ptr&lt;float&gt;() + ten.numel());
for (auto a : v) std::cout &lt;&lt; a &lt;&lt; std::endl;
</code></pre>
<p>This works for me. You have to ensure the tensor type and vector type are the same.</p>

username: jinchen62
cooked: <p><a class="mention" href="/u/glaringlee">@glaringlee</a> I’ve figured it out. The problem is my tensors are in cuda device, transfer to cpu fixs the problem.<br>
Thanks a lot!</p>
Varying iteration time in when using pytorch distributed

username: alet
cooked: <p>Hi,</p>
<p>I’m making some modifications to <a href="https://github.com/facebookresearch/moco" rel="nofollow noopener">MoCo</a>, which runs pytorch multiprocessing. Running the default code leads to very consistent iteration times (between 0.1s and 0.13s). After my modification (essentially adding some optimizable conditional normalization layers as input to the ResNet) the runtime has become stochastic, a bit more than half the iterations take ~0.5-0.6s (I was expecting this increase), but some take 1.5-2s.</p>
<p>The dist-backend is nccl and I’m using 7 GPUs, although the same issue appears when using the default 8 GPUs.</p>
<p>I’m wondering whether this stochasticity implies some bug in my implementation and, if so, what’s the best way of debugging distributed models.</p>
<p>Thanks!</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/alet">@alet</a>, can you share the implementation of modified model? Especially the “optimizable conditional normalization layers”. Would I be correct if I assume the program is using DDP both before and after the modification?</p>
Load my own test data (images)

username: Mdhvince
cooked: <p>Hi,</p>
<p>I just want to know, how can I load my own test data (image.jpg) in pytorch in order to test my CNN.</p>
<p>I’ve tried <code>test_data = datasets.ImageFolder(data_dir + '/test_cnn', transform=test_transforms)</code></p>
<p>but during the test, I have too many indices because my test data is automatically labeled by ImageFolder. A solution is to drop the classes but I don’t know How</p>
<p>Thanks for helping</p>

username: ptrblck
cooked: <p>An easy solution would be to just drop the labels and use your test data in the <code>DataLoader</code> loop:</p>
<pre><code class="lang-python">for data, _ in loader:
    data = data.to(device)
    ...
</code></pre>
<p>This approach is a bit hacky, so the cleaner solution would be to write your own test dataset:</p>
<pre><code class="lang-python">class TestDataset(Dataset):
    def __init__(self, path, transform=None):
        self.image_paths = glob.glob(path + '*.jpg')
        self.transform = transform

    def __getitem__(self, index):
        x = Image.open(self.image_paths[index])
        if self.transform is not None:
            x = self.transform(x)

        return x

    def __len__(self):
        return len(self.image_paths)
</code></pre>

username: Mdhvince
cooked: <p>Thanks for your answer. But How can you drop labels?  (this is my problem)</p>

username: Mdhvince
cooked: <p>Sorry I’m a bit beginner in DL area</p>

username: ptrblck
cooked: <p>No need to excuse for that. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"><br>
I’ve just used an underscore in the data loading loop, to drop the targets:</p>
<pre><code class="lang-python">for data, _ in loader:
    data = data.to(device)
    ...
</code></pre>
<p>In this loop only the data is used and no targets are available.<br>
You could also just get the targets and not use them in any way.</p>

username: Mdhvince
cooked: <p>Thank you so much, it solve my problem</p>
Prunning weights (not optimizing some part of weights)

username: mahdi_morafah
cooked: <p>Hello,</p>
<p>I want to not optimize (do back propagation) some part of weights in a neural network. For example, if I have a two fully connected layers (as my neural network). Let’s say nn.Linear(784, 84) and nn.Linear(84,10). So my weights would be w1 = [84, 784], w2 = [10, 84]. I want to only optimize 1 and 2 row of w2 so other rows in w2 should not be optimized. I have checked Pytorch modules and functions, but still don’t know how to do this in Pytorch. I really appreciate your help. Thanks in advance</p>

username: albanD
cooked: <p>Hi,</p>
<p>Maybe the pruning module with a custom mask is going to be the simplest thing you can use? <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.prune.custom_from_mask.html#torch.nn.utils.prune.custom_from_mask">doc here</a>.</p>

username: mahdi_morafah
cooked: <p>Hi,</p>
<p>Thanks for your reply. Actually, I want to prune different part of the weights at different iterations (and put back the weights that I pruned in the previous iteration). Is there a way that I set the require grad = False for some rows of weights in an iteration? (I think that would be easier for my purpose than the prune mask)</p>
<p>Thanks,</p>

username: albanD
cooked: <p>Hi,</p>
<p>In that case you might have to do it manually. You cannot set autograd-related properties to only a subset of a Tensor I’m afraid.<br>
But you can for example, save the part of the weight you don’t want changed and restore it after doing the optimizer’s step.</p>

username: mahdi_morafah
cooked: <p>This does not work. I want the neural network to be optimized on the pruned version, so the pruned weights do not participate neither in the feed-forward nor the back-propagation. Using the prune with custom mask, every time i should define a neural network from scratch which I wanted to avoid if possible. Hence, I was thinking of the somehow set the specific rows of the weights to zero and freeze them during training (in this case they don’t participate neither in the feed-forward nor the back-propagation). But, I don’t know how this can be implemented in pytorch.</p>
<p>Thanks for your help</p>

username: albanD
cooked: <blockquote>
<p>But, I don’t know how this can be implemented in pytorch.</p>
</blockquote>
<p>I think it depends a lot on the behavior you want.<br>
But just setting part of the Tensor to 0 before the forward and again after the optimizer’s step would work just fine if it’s what you want. Or you could also set some gradients to 0 before doing the optimizer step (be careful as weight decay/momentum might make parameters with 0 grad still change).<br>
But that depends how your pruning change with the iterations.</p>
<p>I think you can do just that:</p>
<pre><code class="lang-auto">for sample in dataset:
    model.prune_some_weights()
    out = model(sample)
    loss = crit(out)
    opt.zero_grad()
    loss.backward()
    opt.step()
    model.prune_some_weights()
</code></pre>
How to add appropriate noise to a neural network with constant weights so that back propagation training works?

username: pinocchio
cooked: <p>I have a neural network in a synthetic experiment I am doing where scale matters and I do not wish to remove it &amp; where my initial network is <strong>initialized with a prior that is non-zero and equal everywhere</strong>.</p>
<p>How do I add noise appropriately so that it trains well with the gradient descent rule?</p>
<p>I was thinking of adding the noise from xavier to my constant weight NN. So I create a new NN with xavier init or standard pytorch init and then to all weights add the constant value I need.</p>
<p>Would that work?</p>
<hr>
<p>cross-posted:</p>
<ul>
<li>Quora: <a href="https://www.quora.com/unanswered/How-do-I-add-appropriate-noise-to-a-neural-network-with-constant-weights-so-that-back-propagation-training-works" rel="nofollow noopener">https://www.quora.com/unanswered/How-do-I-add-appropriate-noise-to-a-neural-network-with-constant-weights-so-that-back-propagation-training-works</a>
</li>
<li>SO: <a href="https://stats.stackexchange.com/questions/483809/how-to-add-appropriate-noise-to-a-neural-network-with-constant-weights-so-that-b" rel="nofollow noopener">https://stats.stackexchange.com/questions/483809/how-to-add-appropriate-noise-to-a-neural-network-with-constant-weights-so-that-b</a>
</li>
<li>pytorch: <a href="https://discuss.pytorch.org/t/how-to-add-appropriate-noise-to-a-neural-network-with-constant-weights-so-that-back-propagation-training-works/93411" class="inline-onebox">How to add appropriate noise to a neural network with constant weights so that back propagation training works?</a>
</li>
</ul>
Efficiently get a tensor's nonzero elements

username: gabinkbl
cooked: <p>I would like to get all nonzero elements along the last dim of  a tensor and place them in a different tensor. I assume I know the upper bound of the number of nonzero elements along each row. For example:<br>
Input = torch.tensor([[0,2,0,0,0],<br>
[1,2,0,0,3],<br>
[0,0,0,0,0],<br>
[0,0,0,0,2]])</p>
<p>Output = torch.tensor([[0, 2, 0],<br>
[1, 2, 3],<br>
[0, 0, 0],<br>
[0, 0, 2]])<br>
The order of the elements along the last dim is unimportant.</p>
<p>Currently, I do it using sort():<br>
Output = Input.sort(descending=True, dim=-1)[0][:,:3]</p>
<p>I guess there is a more time-efficient way here. Is there?</p>
Data Parallelization / nn.DataParallel / multi-processing for general tensor operations on dataset

username: Dipayan_Das
cooked: <p>Just like pytorch provides option like nn.DataParallel() to efficiently make use of multiple-GPUs, is there any such option for general purpose tensor operations that might not require mutli-GPUs but rather multi-processing ?</p>
<p>Like if i have a function like:</p>
<pre><code class="lang-auto">&gt; def generate_dataset(dataloader, val=False):
&gt;     f = open('new_dataset.txt', 'w')
&gt;         for x,y in tqdm(dataloader, total=len(dataloader)):
&gt;             pred = []
&gt;             for x_d in x:
&gt;                  pred.append(general_tensor_operations(x_d))        #THIS IS'NT NECESSARILY A nn.Module()
&gt;             data = format_to_string_data([pred, y.item()])
&gt;             f.write(data)
&gt;     f.close()
</code></pre>
<p>Note that,</p>
<ol>
<li>I am making use of a torch dataloader object</li>
<li>the tensor operation on x is NOT performed by a nn.Module() model</li>
<li>the tensor operation doesn’t work on batch data, rather on each slice of the mini-batch</li>
<li>and i write the obtained (pred,y) to a new txt file</li>
</ol>
<p>Now the processing bottle neck occurs at the general_tensor_operations(). So if I am to parallelize such a function (generate_dataset()), how do i do it ? Note that, if i am using multiple processes, then they should use non-overlapping subsets of the original dataloader (otherwise there will be duplicates in the new_dataset.txt). A code snippet shall help a ton.</p>
<p>Thank you in advance!</p>

username: mrshenli
cooked: <p>If this is on the same machine, will <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener"><code>torch.multiprocessing.queues.SimpleQueue</code></a> work for you?</p>
<p>If you need across-machine communication, you can use <a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener"><code>torch.distributed.rpc</code></a>.</p>
Merging tensors as 'direct sum'

username: DavidWRomero
cooked: <p>Hi guys,</p>
<p>I am having troubles with a simple operation. I would like to combine two tensors of forms ( torch.Size([28, 1, 28, 1], torch.Size([1, 28, 1, 28] ) into a single tensor of size  torch.Size([ 28, 28, 28, 28])  as a direct sum, i.e., without broadcasting operations.</p>
<p>(Note that this is not the case with, for example, sum, as it broacasts sums and thus modifies the values of the original tensors).</p>
<p>Thanks a lot! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>David</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure to understand what you mean by “as a direct sum, i.e., without broadcasting operations.”?<br>
Could you give an example with small Tensors showing which value you want in the output?</p>
Advantages of using an nn.Module

username: Valerio_Biscione
cooked: <p>Hello, I am trying to conceptually understand what are the advantages in terms of speed in using an nn.Module.</p>
<p>What is the difference between subclassing an nn.Module compared to calling the functions such as nn.Conv2d, nn.Linear etc directly on the tensors? Assuming that all tensor are sent to a cuda device, are these equivalent in terms of speed, and is the nn.Module just a convenience class then?</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>There is no benefit in terms of speed. It is mostly for convenience:</p>
<ul>
<li>Easy to move many parameters to the right device at once</li>
<li>Easy to hold and work with these parameters</li>
<li>Easy to interface with the optimizer</li>
<li>etc</li>
</ul>