akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can't get finetuning without freezing weights

antoon: <p>Hi. I’m trying to use FineTuning with AlexNet. Actually, if I use AlexNet pretrained setting requires_grad=False and adding new layers, it works quite well after ten epochs. The problem arises when I don’t set requires_grad=False because I want AlexNet layers to train too. I get very low accuracy test, I tried with various learning rate but can’t get past a validation accuracy of 0.11, even after 20 epochs. Is this normal? Do I have to try with more epochs,like 100, to get a better result? Or am I missing something?</p>
Element wise multiplication of the last dimension

111303: <p>Hello everyone,</p>
<p>I have the following two tensors:</p>
<p><code>E</code> with size <code>(3, 2, 4)</code>, lets visualize it as<br>
<code>e11, e12</code><br>
<code>e21, e22</code><br>
<code>e31, e32</code><br>
where every <code>e</code> is a vector with dimension 4</p>
<p><code>G</code> with size <code>(3, 2)</code>, lets visualize it as<br>
<code>g11, g12</code><br>
<code>g21, g22</code><br>
<code>g31, g32</code><br>
where every <code>g</code> is a scalar</p>
<p>As a result I want to multiply every vector e of E with the corresponding scalar g from G as following:<br>
<code>g11*e11, g12*e12</code><br>
<code>g21*e21, g22*e22</code><br>
<code>g31*e31, g32*e32</code><br>
where * is element wise multiplication, or in other words every vector e from the tensor <code>E</code> (let’s say <code>e11</code>), I want it multiplied with the corresponding scalar (or <code>g11</code> for <code>e11</code>), etc… The size of the result should be <code>(2, 2, 4)</code>, same as the <code>E</code> tensor.</p>
<p>Thank you.</p>

111303: <p>I found out that first unsqueezing the G tensor, repeating it 4 times along the 3-th dimension, and element-wise multiplying it with E does the job, but there may be a more elegant solution. Here is the code:</p>
<p><code>G_tmp = G.unsqueeze(2).expand(-1, -1, 4)</code><br>
<code>res = G_tmp * E</code></p>
<p>Feel free to correct me, or propose a more elegant solution <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

googlebot: <p>You don’t need expand() there, tensor broadcasting takes care of it.</p>
How to do not equal to

Umair_Javaid: <p><code>a.eq(b)</code> is used to do element wise comparison, where it returns True if the values are equal. How to do not equal to? where True is returned in places where elements are not equal</p>

russellizadi: <p>The <code>~</code> operator negate the bool tensor:</p>
<pre><code class="lang-auto">import torch
a = torch.tensor([1, 2])
b = torch.tensor([1, 0])
print(a.eq(b))
print(~a.eq(b))
</code></pre>
<pre><code class="lang-auto">tensor([ True, False])
tensor([False,  True])
</code></pre>
What is dropout in convolutional layers and how does that different from max-pooling-dropout?

smani: <p>When dropout is applied to fully connected layers some nodes will be randomly set to 0.</p>
<p>It is unclear to me how dropout work with convolutional layers. If dropout is applied before the convolutions, are some nodes of the input set to zero?</p>
<p>If that so how does this differ from <a href="https://arxiv.org/abs/1512.01400" rel="nofollow noopener">max-pooling dropout</a>? Even in max-pooling-dropout some elements in the input are randomly dropped (Set to zero). So the idea of max-pooling dropout is same as dropout in convolutional layers?</p>
Doubt in .view() function in Neural Style Transfer

thomas-jac: <p>I was going through the <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html#introduction" rel="nofollow noopener">neural style transfer</a> program using pytorch and have a doubt in the usage of view in the <strong>init</strong> method of the Normalization class. Just before this class we defined</p>
<pre><code class="lang-python">cnn_normalization_mean = torch.tensor([0.485, 0.456, 0.406]).to(device)
cnn_normalization_std = torch.tensor([0.229, 0.224, 0.225]).to(device)
</code></pre>
<p>And inside the __init__method were the lines</p>
<pre><code class="lang-python">self.mean = torch.tensor(mean).view(-1, 1, 1)
self.std = torch.tensor(std).view(-1, 1, 1)
</code></pre>
<p>Why did we reshape them as wouldn’t broadcasting take care of it?<br>
My current understanding is that each of the three numbers in the tensors defined above are supposed to normalize a single dimension (one each for height, width and classes), is this okay?</p>
<p>It might be a bit silly, but I’m having a hard time getting an intuition in 3D. Any help will be much appreciated, thanks!</p>
<p>Edit:<br>
The complete <strong>init</strong> method is</p>
<pre><code class="lang-python">def __init__(self, mean, std):
        super(Normalization, self).__init__()
        # .view the mean and std to make them [C x 1 x 1] so that they can
        # directly work with image Tensor of shape [B x C x H x W].
        # B is batch size. C is number of channels. H is height and W is width.
        self.mean = torch.tensor(mean).view(-1, 1, 1)
        self.std = torch.tensor(std).view(-1, 1, 1)
</code></pre>

chetan06: <p>Can you provide the complete code within init because saying something without the code seems a bit difficult</p>

thomas-jac: <p>Done! Let me know if anything else is needed.</p>

chetan06: <p>For each image they are taking mean and std per channel so that it can be used for normalization. Normalizing the dimensions don’t sound so appropriate actually we normalize the data within each channel or you can say that we normalize the feature map and we do so per channel that’s why its getting a shape like that. Hope you got me.</p>

thomas-jac: <p>Ah, that makes sense but I’m not able to figure out how reshaping it in that way allows it to normalize the data within each channel. Could you please illustrate with a small example?</p>
<p>I’m sorry, I’m pretty new to DL and PyTorch and having a bit of difficulty with certain concepts <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

chetan06: <p>The answer to your question lies in the code where they have used this mean. Actually there are many ways to write the code. It may be so that this type of reshaped array is being used in there case. You proceed your reading and you will get your answer.</p>
<p>You don’t have to be sorry. Anyone can be stuck and always feel free to share your doubt with the community.</p>
How to implement a scikit-learn estimator in PyTorch

Leockl: <p>I had developed an estimator in Scikit-learn but because of performance issues (both speed and memory usage) I am thinking of implementing the estimator to run using GPU.</p>
<p>To do this, one way I can think of is to implement the estimator in PyTorch (so I can use GPU processing) and then use Google Colab to leverage on their cloud GPUs and memory capacity.</p>
<p>What would be the best way to implement an estimator which is already scikit-learn compatible in PyTorch?</p>
<p>Any pointers or hints pointing to the right direction would really be appreciated. Many thanks in advance.</p>

ptrblck: <p>I’m not sure how your estimator works, but you could probably just port the code to PyTorch.<br>
A lot of numpy operations are implemented in PyTorch, so that you could change the namespace and run the PyTorch operation.</p>
<p>If you encounter some methods which are missing, let us know as there might be some workarounds.</p>

Leockl: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. The estimator I have developed is a classifier. You can see it here if you’re interested: <a href="https://github.com/leockl/helstrom-quantum-centroid-classifier" rel="nofollow noopener">https://github.com/leockl/helstrom-quantum-centroid-classifier</a></p>
<p>Based on what you said about a large number of numpy operations are implemented in PyTorch, I don’t think I will have issues with the code as it is mostly all written using numpy arrays.</p>
<p>However, I am still not very clear on how to inherit from scikit-learn’s BaseEstimator and ClassifierMixin (which allows for methods like .fit(), .predit() etc.) into PyTorch. Any clues, directions or even better examples on this would really be appreciated. Many thanks in advance.</p>

ptrblck: <p>I’m not sure what the best way would be, but you might want to take a look at <a href="https://github.com/skorch-dev/skorch/tree/master/skorch">skorch</a> which implements a sklearn-compatible API for PyTorch models, so I assume they have solved the issues. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Leockl: <p>Many thanks for your input <a class="mention" href="/u/ptrblck">@ptrblck</a>. I had a look at skorch and if I am not mistaken, skorch implements sklearn-compatible APIs for “PyTorch models”, and not the other way round, ie. implement “sklearn models” for PyTorch, which is what I am after.</p>
<p>I guess I will have to google around more to see what I can find. Some folks over at reddit have suggested a few ideas which might work too.</p>
Unable to print a tensor

ziqipang: <p>Hi! I have been trying to draw some samples from a weight tensor and do some following stuff. This part is in a very large project, and <strong>it sometimes breaks down after being operated millions of iterations</strong>.</p>
<p>Therefore, I tried to print out the sample_op and weight tensor to debug. However, <strong>I got another bug for the sentence “print(sample_op)”, which looks like I even cannot print out the tensor of sample_op.</strong> I wish to get some help, s. Thanks for your attention and help!</p>
<p>The code is like this:</p>
<pre><code class="lang-auto"># sample_weight and op_weight are all tensors
sample_op = torch.multinomial(sample_weight, 2, replacement=False)
try:
    probs_slice = F.softmax(torch.stack([
        op_weight.data[i, idx] for idx in sample_op]), 
        dim=0)
except RuntimeError:
    print(sample_op)
    print(op_weight)
    exit(0)
</code></pre>
<p>The error log is like this:</p>
<pre><code class="lang-auto">File "/cache/user-job-dir/nas-branch/models/model_search.py", line 109, in binarize
    print(sample_op)
  File "/home/work/anaconda/lib/python3.6/site-packages/torch/tensor.py", line 114, in __repr__
    return torch._tensor_str._str(self)
  File "/home/work/anaconda/lib/python3.6/site-packages/torch/_tensor_str.py", line 311, in _str
    tensor_str = _tensor_str(self, indent)
  File "/home/work/anaconda/lib/python3.6/site-packages/torch/_tensor_str.py", line 209, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/home/work/anaconda/lib/python3.6/site-packages/torch/_tensor_str.py", line 83, in __init__
    value_str = '{}'.format(value)
  File "/home/work/anaconda/lib/python3.6/site-packages/torch/tensor.py", line 361, in __format__
    return self.item().__format__(format_spec)
RuntimeError: CUDA error: device-side assert triggered
</code></pre>

tom: <p>The device assert is the “hard error” option for CUDA kernels: It invalidates the CUDA context, which for practical purposes means you need to restart PyTorch. The problem here is that there isn’t any good method to report errors due to the asynchronous nature of CUDA. To find out which kernel is the one erroring, either insert torch.cuda.synchronize() between lines or run with the environment variable CUDA_LANCH_BLOCKING=1 set.<br>
My guess would be torch.multinomial. For example, it is allergic to all-zero probability tensors (and because it returns integers cannot use NaN).</p>
<p>Best regards</p>
<p>Thomas</p>

ziqipang: <p>Thanks, I will take a look!</p>
Question on Backward Direction of Bidirectional GRU/RNN

kekgle: <p>Hi! I was wondering how to recover the last output element of the backward direction of a RNN.</p>
<p>Imagine my input sentence is:<br>
“The quick brown fox jumped over the lazy dogs”</p>
<p>For the backward direction, I would like to recover the last hidden state for the word <em>The</em> (last one in the reversed sequence).</p>
<p>After I separate the two directions (<strong>assume batch_first=True</strong>) using:</p>
<pre><code class="lang-auto">out = out.view(batch_size, seq_length, 2, hidden_size)
</code></pre>
<p>and recover the backward direction using:</p>
<pre><code class="lang-auto">backward = out[:,:,1,:]
</code></pre>
<p>if I want to recover the output for the word <em>The</em>, this should be:</p>
<pre><code class="lang-auto">y = backward[:,0,:]
</code></pre>
<p>or do I need to take the last index as for the forward direction?</p>
<pre><code class="lang-auto">y = backward[:,-1,:]
</code></pre>
RuntimeError: Function 'SolveBackward' returned nan values in its 0th output

barrykui: <p>The loss came to NaN, and I debug the autograd by using <code>torch.autograd.set_detect_anomaly(True)</code> and got the error below.</p>
<pre><code class="lang-auto">....
  File "main.py", line 374, in compute_plane
    ca_cb, LU = torch.solve(B, A + eps)

Traceback (most recent call last):
  File "main.py", line 879, in &lt;module&gt;
    main()
  File "main.py", line 420, in main
    train(fk, train_loader, val_loader, model, optimizer, lr_scheduler, last_iter+1, tb_logger, criterion=loss_fun)
  File "main.py", line 544, in train
    loss.backward()
  File "/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'SolveBackward' returned nan values in its 0th output.
</code></pre>
<p>Does anyone have a solution for this problem? pls help, Thanks.</p>

ptrblck: <p>Could you post the inputs to <code>torch.solve</code>, which create the NaNs?</p>
<p><a href="https://github.com/pytorch/pytorch/issues/37712#issuecomment-623004539">GitHub issue</a> for reference.</p>

barrykui: <p>Finally, I found the problem which should be the mathematic error in the previous code. Thank you all the same.</p>
Sensitivity of output y with respect to input x?

Abueidda: <p>Hi All,</p>
<p>I have developed a regression CNN model using Pytorch, and I am interested in finding the sensitivities of the output y with respect to the input x. <strong>Please note that I am interested in finding the sensitivity of the output with respect to the input rather than finding the sensitivity of the loss function with respect to the input.</strong> Is it possible to do that? If yes, any guidance would be much appreciated.</p>
<p>Thanks<br>
Diab</p>

akernel: <p>Hi Diab,</p>
<p>Could you clarify what you mean by sensitivity?  I’d love to help but am not sure exactly what you’re looking for.</p>
<p>Thanks!<br>
Adam</p>

Diego: <p>Hi, so if by <strong>sensitivity</strong> you mean, how sensible is the output to changes in the input (e.g the gradients). Then something like this could work:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; inp = torch.randn(5)
&gt;&gt;&gt; 

&gt;&gt;&gt; inp.requires_grad_()
tensor([-0.1457,  0.1341,  0.6884, -1.9446, -0.2557], requires_grad=True)
&gt;&gt;&gt; x = inp * 7 
&gt;&gt;&gt; x = x/10
&gt;&gt;&gt; out = torch.exp(x)
&gt;&gt;&gt; out.sum().backward()
&gt;&gt;&gt; inp.grad
tensor([0.6321, 0.7689, 1.1334, 0.1794, 0.5853])
</code></pre>
<p>Also, one clarification, you normally don’t compute the <strong>sensitivity</strong> of the loss function with respect to the input, but rather, to the model’s parameters. Hope, this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Abueidda: <p>Hi Deigo,</p>
<p>Thanks for the reply. What does .sum() do here?<br>
Thanks</p>

Abueidda: <p>I mean d_output/d_input. Sorry for not being clear.</p>
Continue train after eval has different accuracy than only train

Y_Y: <p>Hi all,<br>
Say I have a model training for 10 epoch. If I want to eval the model after every epoch, here are the two experiments.<br>
Exp1.</p>
<pre><code class="lang-auto">for epoch in range(10):
    model.train()
    calculate_train_accuracy()
</code></pre>
<p>Exp2.</p>
<pre><code class="lang-auto">for epoch in range(10):
    model.train()
    with torch.no_grad():
        for i, data in enumerate(dataset_test.dataloader):
            model.data = data 
        model.eval()
    calculate_train_accuracy()
</code></pre>
<p>Shouldn’t the two experiments have the same accuracy in last epoch if I set same random seed? But I got different accuracy values. Any of you have a idea about this?<br>
Thank you all.</p>
<hr>
<p>Solution: I figured out the difference was related to dataloader. Even if you fix the random seed outside epoch for loop, if you set shuffle=True in dataloader, the random seed will be messed up between test and training sets.</p>
Training in parallel

blade: <p>I’m training a VAE similar to the implementation in PyTorch’s Github. The main function looks like:</p>
<pre><code class="lang-auto">if __name__ == "__main__":
    for epoch in range(1, args.epochs + 1):
        train(epoch)
</code></pre>
<p>Assuming that I have another input parameter for training function, <code>pi</code>, I would like to write a code that trains multiple models with different parameter <code>pi</code>.</p>
<pre><code class="lang-auto">if __name__ == "__main__":
    for i in range(10):
        pi = get_param(seed=i)
        for epoch in range(1, args.epochs + 1):
            train(epoch, pi)
</code></pre>
<p>My question is how can I run this in parallel on GPU, such that each core trains a single model.</p>

ptrblck: <p>If your GPU is already fully utilized, you won’t be able to train models in parallel and the calls will be added to the queue.<br>
On the other hand, if you have multiple devices, you could run the training routines on each device and they will be run in parallel.</p>
Dataloader not returning paths variable from hdf5 dataset

SU801T: <p>Hi,</p>
<p>I have an issue returning the paths of files from a hdf5 dataset using the dataloader.</p>
<p>I get the following error:<br>
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found object</p>
<p>It works fine, for dataum and label, which are tensors, however, it does not work for “path”.</p>
<p>An example path stored in the hdf5 dataset:<br>
[‘mults/train/0/5678.ndpi/40x/40x-236247-16635-80640-8704.png’]</p>
<p>Below is the code on the features_dataset class:</p>
<pre><code class="lang-auto">
import h5py
from torch.utils import data
from torchvision import transforms as T

class Features_Dataset(data.Dataset):
    def __init__(self, archive, phase):
        self.archive = h5py.File(archive, 'r')
        self.labels = self.archive[str(phase) + '_labels']
        self.data = self.archive[str(phase) + '_all_arrays']
        self.img_paths = self.archive[str(phase) + '_img_paths']

    def __getitem__(self, index):
        datum = self.data[index]
        label = int(self.labels[index])
        path = self.img_paths[index]
        return datum, label, path

    def __len__(self):
        return len(self.data)

    def close(self):
        self.archive.close()

if __name__ == '__main__':
    train_dataset = Features_Dataset(archive= "featuresdata/train.hdf5", phase= 'train')
    trainloader = data.DataLoader(train_dataset, num_workers=1, batch_size=1)
    print(len(trainloader))
    for i, (data, label, path) in enumerate(trainloader):
        print(path)


</code></pre>
<p>Here is an small script of what I did to store the particular paths in the hdf5 dataset:</p>
<pre><code class="lang-auto">if sys.version_info &gt;= (3, 0):
        string_type = h5py.special_dtype(vlen=str)
else:
        string_type = h5py.special_dtype(vlen=unicode)  # noqa


labels_shape = (dataset_length, args.batch_size)
path = args.HDF5_dataset + f'{phase}.hdf5'

with h5py.File(path, mode='a') as hdf5_file:
     array_paths = hdf5_file.create_dataset(
                    f'{phase}_img_paths', labels_shape, maxshape=(None, args.batch_size), dtype=string_type)


</code></pre>
<p>I’m not really sure what to do as paths are returned fine with alternative dataloaders when reading from. a directory of images…</p>
How to load image in default range [0,255] using torchvision?

sreeni5493: <p>torchvision transforms ToTensor converts the image to [0,1]. Rather than multiplying by 255 afterwards. Can I avoid this normalization in the below code snippet in some way?</p>
<pre><code>transform = transforms.Compose([transforms.ToTensor()])</code></pre>

jytug: <p>You can do this easily without using <code>ToTensor</code>. All you need is to define your own transform, like this:</p>
<pre><code class="lang-auto">class ToTensorWithoutScaling(object):
    """H x W x C -&gt; C x H x W"""
    def __call__(self, picture):
        return torch.ByteTensor(np.array(picture)).permute(2, 0, 1)

transform = transforms.Compose([transforms.ToTensorWithoutScaling()])
</code></pre>
<p><em>Edit</em>: Intermediate conversion to <code>np.array</code> is needed before calling <code>ByteTensor</code> constructor</p>
RuntimeError: sparse_.is_sparse() INTERNAL ASSERT FAILED at /pytorch/torch/csrc/autograd/generated/Functions.cpp:541, please report a bug to PyTorch

amrhssnn: <p>Hi everyone!<br>
I’m trying to run a model which consists of multiple graph convolutional networks and a recurrent neural network.<br>
I have converted the inputs of my graph convoulutional networks (i.e., feature matrix and adjacecny matrix) to torch.sparse.FloatTensor. I only use torch.sparse.mm() in the GCN and the loss function is BCEWithLogitsLoss().<br>
In the forward pass everything works fine but as soon as loss.backward() is executed the below error is raised:<br>
RuntimeError: sparse_.is_sparse() INTERNAL ASSERT FAILED at /pytorch/torch/csrc/autograd/generated/Functions.cpp:541, please report a bug to PyTorch.<br>
Can someone help me with that error? thanks.</p>

ptrblck: <p>Could you post a code snippet with random inputs to reproduce this issue, please?</p>
Parallel computations on a whole batch

Wolf_Nuyts: <p>Hi,</p>
<p>I have a problem where i need to calculate a simularity matrix before i can calculate my loss function. This simularity matrix takes a lot of time and is only dependant on the batch data itself. That is why i would like to calculate it in parrallel before the batch has started but i don’t know how i can do that. Is it possible with the torch.Dataloader? I know you can preprocess the data there and transform each sample on his own but can you aswell return a computation on the whole batch itself? Or is there another way to do this?</p>

ptrblck: <p>Depending on the speed of this operation, you might calculate the similarity matrix inside the <code>DataLoader</code> loop, while the GPU is busy with the model training.<br>
However, I’m not sure what method you are using and how slow/fast it is on the CPU and GPU.</p>
<p>Could you post the applied method and the relative time it’s using in a training iteration?</p>
Append items from batch to dataset

SU801T: <p>Hi,</p>
<p>I am extracting features and storing them in a dataset. Currently, I use a batch size of 1 where I append each batch into a hdf5 dataset. However, I’m stuck on how to extract each item from a batch, when the batch size is greater than 1. I basically want to store each item from a batch as its own row/instance in the dataset.</p>
<p>Any idea on what I can do?</p>
<p>Cheers,</p>
<pre><code class="lang-auto">for i, (inputs40x, inputs20x, paths40x, paths20x, labels) in enumerate(dataloaders_dict):

                print(f'Batch ID: {batch_idx}')

                inputs40x = inputs40x.to(device)
                inputs20x = inputs20x.to(device)

                labels = labels.to(device)
                paths = paths40x

                # delete the last fc layer.
                modules = list(resnet50.children())[:-1]
                resnet = nn.Sequential(*modules)

                x40 = resnet(inputs40x)
                x20 = resnet(inputs20x)
                x_all = torch.cat([x40, x20], dim=1)

                # torch.Size([1, 2048, 1, 1]) batch, feats, 1l, 1l
                array_40[batch_idx, ...] = x40.cpu()
                array_20[batch_idx, ...] = x20.cpu()
                array_all[batch_idx, ...] = x_all.cpu()
                array_labels[batch_idx, ...] = labels[:].cpu()
                array_paths[batch_idx, ...] = paths
                array_batch_idx[:,...] = batch_idx

                batch_idx +=1
</code></pre>
Conditioned Dataloader

Ameen_Ali: <p>Hello All, <br><br>
Let’s say I have a dataloader which returns video samples where each video sample is of the following shape : [number_of_frames X 2048] (number of frames can vary from the following set : [60 , 80 , 100 , 120]. <br><br>
I want in each batch returned from the dataloader to have samples with the same number of frames.<br>
for example if the batchsize=16 . I want batches of the following shape :  [16 X 80 X 2048] , [16 X 60 X 2048]…<br>
Anyidea how to do it?<br>
Thanks</p>

ptrblck: <p>Should these batches be returned sequentially or are you expecting to get a batch for each sequence length in a single step?</p>
<p>In the latter case, you could wrap each <code>Dataset</code> for a specific sequence length in one <code>MainDataset</code>, which could return a frame for each sequence length. The <code>DataLoader</code> would then create the batches.</p>

Ameen_Ali: <p>I am expecting to get a batch for each sequence length<br>
for example one batch could be of shape : [16 , 80 , 2048] another batch would be of also [16 , 80 , 2048] or [16,120,2048] I thought about sorting the samples by lengths but I want to shuffle them!</p>

ptrblck: <p>You could probably write a custom sampler, which could yield the data indices based on the condition of the data length, but I’m not sure how complicated that would be.<br>
Maybe the easier approach is to create separate <code>DataLoaders</code>, one for each length, use their <code>iterators</code>, and call randomly <code>next</code> on them?</p>
<p>Something like this might work:</p>
<pre><code class="lang-python">dataset1 = TensorDataset(torch.randn(10, 1), torch.randn(10, 1))
dataset2 = TensorDataset(torch.randn(10, 2), torch.randn(10, 2))
dataset3 = TensorDataset(torch.randn(10, 3), torch.randn(10, 3))

loader1 = DataLoader(dataset1, batch_size=2)
loader2 = DataLoader(dataset2, batch_size=2)
loader3 = DataLoader(dataset3, batch_size=2)

loader_iter1 = iter(loader1)
loader_iter2 = iter(loader2)
loader_iter3 = iter(loader3)

iters = [loader_iter1, loader_iter2, loader_iter3]

np_epochs = 50
for epoch in range(nb_epochs):
    curr_iter = random.choice(iters)
    try:
        data, target = next(curr_iter)
        print(data)
    except StopIteration:
        break
</code></pre>
Creating a new variable in the middle of the loss function to be added to the graph

Abdelpakey: <p>Hi all,</p>
<p>I’m calculating the scatter-matrix for the output of the network, the output shape like this<br>
batch_size=100<br>
features=1000<br>
tensor(100, 1000)</p>
<p>In the middle of calculation, I create a new tensor (that requires grad) to be added later to another tensor (requires grad).</p>
<p>The code works fine when I create the new tensor like this</p>
<pre><code>S_w= torch.zeros((d,d),requires_grad=True).cuda()
</code></pre>
<p>but I want to create it in an elegant and safe way to be part of the graph not a leaf variable that can be created on CPU to have an option to move it or not to the CPU.</p>
<p>Here is my code snippet:<br>
…</p>
<pre><code>S_w= torch.zeros((d,d),requires_grad=True).cuda()   // Want to creat this in proper way

S_w +=lamb* torch.eye(*S_w.size(),out=torch.empty_like(S_w)) //Want to creat  the other tensor and add it in a proper way

for cl in range (C):
    Sc= torch.matmul(normalized_feat_for_each_class_tensor[y==cl].T,
                      normalized_feat_for_each_class_tensor[y==cl])

    Nc=torch.tensor(normalized_feat_for_each_class_tensor[y==cl].shape[0],dtype=torch.float32)
    S_w +=Sc/(Nc-1)/C   
</code></pre>
<p>…</p>
Torch tensor memory size discrepancy

pytorchnewbie: <p>Hi! So I’m using  " watch -n 0.1 nvidia-smi " to try to debug a memory leak im having. I set a pdb.set_trace() at the very beginning of my main file. I then did:</p>
<p>test = torch.zeros((4,400,400))<br>
test = test.to(1)</p>
<p>I then print:</p>
<p>torch.cuda.memory_allocated(device=1)<br>
and get:<br>
2560000</p>
<p>All good so far. However, suddenly my nvidia-gpu watch displays gpu <span class="hashtag">#2</span> (ignore the fact that its not 1, because theres a mismatch between torch gpu <span class="hashtag">#s</span> and nvidia), except it now shows around 550MiB of memory being used!!!</p>
<p>I then do<br>
test = torch.zeros((4,400,400))<br>
test2 = test2.to(1)</p>
<p>So again, I print:<br>
torch.cuda.memory_allocated(device=1)<br>
and get:<br>
2880000</p>
<p>But this time, my nvidia-watch thing doesn’t change at all!<br>
Clearly, the torch.cuda.memory_allocated is correct, but I’m not sure whats going on with my nvidia watch… This is a problem because due to this memory leak i cant run my code</p>
<p>Thanks!</p>

googlebot: <p>The difference is mainly caused by cuda runtime, that eats hundreds of megabytes once (lazily) initialized; that’s mainly because these is a lot of cuda kernels in torch library. Also, torch memory allocator allocates memory in bigger blocks, so you should also consider cached memory when tracking usage (try torch.cuda.memory_summary()).</p>
Size of LSTM output and linear input

waw_waw: <p>Hi, I have a question when using LSTM,<br>
the input size of LSTM is [4,256,100], so this is a 4 seconds sequence, each one has length 100, and batch size is 256. and the output of the LSTM is [4,256,20].<br>
I added a Linear layer after the LSTM, the size of linear is (4<em>256</em>20,output_size).<br>
But during testing, the input will be [4,1,100], which is only one batch, and there will be an error for the input size of the Linear layer. Are there any ways to fix this?</p>
<p>I know in CNN, the batch size for training and testing can be different, during training, I set the batch size to be 32 or 64, and during testing, I can set the batch size to be 1 and it still works</p>
Make_grid with unequal grid?

chilango: <p>I can use <code>make_grid</code> fine to make such a grid:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2e7e40cf018af1a2e54bfc21012c9ebc8ad7215.png" alt="tiles_1bg_200pad" data-base62-sha1="yEQn2Bg67G9ozq5pWvUIogY3dGd" width="483" height="369"></p>
<p>But how can I make a grid like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6774cdc3c20b00dff658126a57e8f71e017db7a1.png" data-download-href="https://discuss.pytorch.org/uploads/default/6774cdc3c20b00dff658126a57e8f71e017db7a1" title="example"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6774cdc3c20b00dff658126a57e8f71e017db7a1.png" alt="example" data-base62-sha1="eLdqGmzaeeGaadFYpknOWwJzh4J" width="496" height="375" data-dominant-color="878787"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">example</span><span class="informations">858×648 429 Bytes</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Ideally I would like to have the images in each grid padded randomly with different values.</p>
<p>Thank you.</p>

ptrblck: <p>You could probably use <a href="https://matplotlib.org/3.1.1/tutorials/intermediate/gridspec.html"><code>matplotlib.gridspec</code></a> for it.<br>
I don’t think this will pad your inputs, so you might need to add the padding in PyTorch or numpy.</p>

chilango: <p>Thanks that’s a nice suggestion.</p>
Multiply pytorch 4 dimension array with scalar value

cbd: <p>I have array A with dimension [ 1, 3, 256, 256]. I want to multiply A with vector B having value B=[1 2 3]. It means all the values of vector A channel 1[ A(1,1,256,256)] should multiply with constant value 1, channel 2[ A(1,2,256,256)] should multiply with constant value 2 and channel 3 with constant value 3.</p>
<p>How to transform B such that it gives desired result by direct multiplication A*B?</p>

beaupreda: <p>Hello,</p>
<p>This code seems to be doing what you want (let me know if it is not!)</p>
<pre><code class="lang-auto">    a = torch.ones(size=(1, 3, 8, 8), dtype=torch.float32) * 2.0
    print(a)
    b = torch.tensor(data=[1, 2, 3], dtype=torch.float32)
    print(b)

    b = b.unsqueeze(dim=0)
    b = b.unsqueeze(dim=2)
    b = b.unsqueeze(dim=3)
    # b passes from a (1 x 3 x 1 x 1) tensor to a (1 x 3 x 8 x 8) tensor
    b = b.expand_as(a)

    # element wise multiplication
    print(a * b)
</code></pre>

praveen_kandula: <p>This works. But I think  “b = b.expand_as(a)” in your code is not necessary as broadcasting will help.</p>

beaupreda: <p>Yes, you are right! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

cbd: <p>It works. Same thing i wants. Thanks</p>
Why is my eval accuracy so low?

drliu: <p>I’m running a super simple feed forward neural network on a dataset consisting of 128 features over 50 classes. These features are sentence embeddings I’ve extracted from a BERT model where I’m already reaching 0.73 accuracy.</p>
<p>During training, the loss goes down consistently and the accuracy on the training set is usually at least 0.40 depending on what I set the hyperparameters at.</p>
<p>But the accuracy on the dev set is <strong>always</strong> extremely low. In most cases it’s usually &lt;0.02, which is worse than random guessing.</p>
<p>Things I’ve tried:</p>
<ul>
<li>Learning rates of <code>[0.1, 0.01, 0.001, 0.0005, 0.0001, 0.00001]</code>
</li>
<li>Adding extra layers in the neural net</li>
<li>Using sigmoid and tanh instead of ReLU</li>
<li>Removing dropout layer</li>
<li>Adding batch norm layer</li>
<li>SGD instead of Adam</li>
</ul>
<p>Code:</p>
<pre><code class="lang-auto">class NeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden1 = nn.Linear(128, 80)
        self.output = nn.Linear(80, 50)

        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(p=0.5)

    
    def forward(self, x):
        x = self.hidden1(x)
        x = self.relu(x)
        x = self.dropout(x)

        x = self.output(x)
        return x

nnet = NeuralNet()

loss_function = nn.CrossEntropyLoss()

optimizer = optim.Adam(nnet.parameters(), lr=0.001)

nnet.double().to(device)

# Create the DataLoader for our training set.

train_data = TensorDataset(train_features, train_labels)

train_sampler = RandomSampler(train_data)

train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=32)

validation_data = TensorDataset(test_features, test_labels)

validation_sampler = SequentialSampler(validation_data)

validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=32)


epochs = 100

for epoch in range(1, epochs+1):
    nnet.train()
    epoch_loss = 0
    epoch_acc = 0
    for batch in train_dataloader:
        features = batch[0].to(device)
        labels = batch[1].to(device)
        optimizer.zero_grad()

        outputs = nnet(features)
        vals, inds = torch.max(outputs, dim=1)
        
        loss = loss_function(outputs, labels)
        epoch_loss += loss

        acc = torch.eq(inds, labels).sum().item() / labels.shape[0]
        epoch_acc += acc

        
        loss.backward()
        optimizer.step()
    avg_epoch_loss = epoch_loss / len(train_dataloader)
    avg_epoch_acc = epoch_acc / len(train_dataloader)
    print(f"Epoch: [{epoch}/{epochs}], loss: {avg_epoch_loss}, acc: {avg_epoch_acc}")


nnet.eval()
total_acc = 0
for batch in validation_dataloader:
    features = batch[0].to(device)
    labels = batch[1].to(device)

    with torch.no_grad():
        outputs = nnet(features)
        vals, inds = torch.max(outputs, dim=1)
        loss = loss_function(outputs, labels)
        print(f"Loss: {loss}")
    acc = torch.eq(inds, labels).sum().item() / labels.shape[0]
    total_acc += acc
avg_acc = total_acc / len(validation_dataloader)

print(f'Acc: {avg_acc}')
</code></pre>
Understanding how dropout works

syomantak: <p>Hi,<br>
I am wondering how Dropout is actually implemented in Pytorch. I found the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#dropout" rel="nofollow noopener">source</a> but I can’t understand much. My main question is how can PyTorch apply dropout after giving it the output of linear layer without any dropout?<br>
Basically, I mean the code is usually something like this</p>
<pre><code class="lang-auto">x1 = F.relu(self.fc1(x))
x2 = self.dropout1(x1))
</code></pre>
<p>So dropout1 function receives tensor <code>x1</code>, and has no idea what the input to <code>fc1</code> layer was, nor does it know the weights. Mathematically, it is clear that dropout function needs to know tensor <code>x</code> and weights of layer <code>fc1</code> to randomly remove weights and get an output.<br>
How, then, is it able to compute <code>x2</code> properly without taking <code>x</code> or <code>fc1</code> weights as function arguments?</p>

russellizadi: <p>Dropout operates independent of the previous or the next layer and it is noting but sampling elements of the input with some probability and neglecting the rest, i.e. replacing with zero. For sampling elements of a tensor, you can sample from binomial distribution as many times as the size of the input.</p>
<p>Here’s a simple implementation:</p>
<pre><code class="lang-auto">import torch

dis = torch.distributions.binomial.Binomial(probs=.7)
w = torch.nn.Linear(3, 2)
input = torch.randn(8, 3)
h = w(input)
output = h*dis.expand(h.shape).sample()
print(output)
</code></pre>

syomantak: <p>I feel so stupid here. I thought Dropout function applies it to tensor <code>x</code> in my example. If I understood correctly, it is actually applied to <code>x1</code> then?</p>

russellizadi: <p>That is true. In fact, you can replace the previous layer with any other  operation. Think of the dropout as a colander with random holes <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Train Pytorch model on Android device

abediee: <p>Hi!<br>
Is it possible to train a NN model on Android device? (and not using a pre-trained model)</p>

tom: <p>In principle this is possible if you compile a full version of libtorch and use that. It won’t be lightning fast, though.</p>
<p>Best regards</p>
<p>Thomas</p>

abediee: <p>Thank you so much!</p>
<p>regards</p>
Creating Shakespeare-like content with LSTM

felixn-w: <p>Dear Pytorch Community,</p>
<p>I am new to Pytorch and neural networks in general. After reading Andrej Karpathy blog entry on <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow noopener">recurrent neural networks</a> I tried to rebuild his Shakespeare example with Pytorch and the use of some of the official Pytorch tutorials.<br>
However the tests I do so far do no generate any usefull output.<br>
I would be happy for any improvement recommendations of my [coding on github] (<a href="https://github.com/felixn-w/shakespeare" rel="nofollow noopener">https://github.com/felixn-w/shakespeare</a>)<br>
The basic idea is to read all words of a data set with Shakespeare texts, assign an integer to each letter. The model then consists of a LSTM layer and a linear layer meant to predict a respective next letter.</p>
<p>Your assistance is highly appreciated</p>

Angry_potato: <p>Was your attempt successful.</p>
PyTorch RNN LSTM

QuirkyDataScientist1: <p>I have some basic questions about Pytorch that I have not see answered and for me its very much so needed in order for me to get started. First and foremost I want to use the University of Maryland Global Terrorism Database (GTD) part of the START website which has over 190,000 observations, and approximately 135 dimensions (features). I plan on using one hot encoding for the variables, deleting some features out, and using feature engineering for others. That being said, I have some questions:</p>
<p>With the dataset - do I need to delete out the header columns? Or can I leave those in there?</p>
<p>Once I delete out what I don’t want - lets say I am left with 130 features, including latitude, logitude values, as well as total attacks for numbers, - my understanding is that whatever I input it will output feature wise correct? So I will get 130 predicted values for the new features - what does this mean with regards to my input?</p>
<p>How should I set it up?</p>
How to bring different tensors to the same size?

Kla: <p>Hi,</p>
<p>I’m trying to prepare the input_data for my project but my tensors are of different sizes, for example: [1, 1256], [1, 6497], [1, 58672], etc. I want to have them all of the size [1, 58672]. Can you please tell me how can I do it?</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Depending how you would like to increase this dimension, you could use</p>
<ul>
<li>
<code>F.pad</code> and pad the dimension to the desired shape</li>
<li>create another tensor in the “missing” shape and use <code>torch.cat((x, other), dim=1)</code> to concatenate them</li>
<li>concatenate the tensor to itself and pad the rest</li>
</ul>
<p>Let me know, if that would work or if you would like to apply another approach. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Kla: <p>Hi ptrblck, thank you for your answer. I’m actually trying to create a Dataset for my audio project but unfortunately I’m still a beginner and I have to learn a lot. I have tried this site: <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a> but it’s still not working. Can you please suggest me a website where I can learn something more, for example F.pad and torch.cat?</p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.pad">docs</a> are generally a good starting point.<br>
This code should work for the padding approach:</p>
<pre><code class="lang-python">target_len = 58672
x = torch.randn(1, 1256)
x = F.pad(x, (target_len - x.size(1), 0))
print(x.shape)
&gt;  torch.Size([1, 58672])
</code></pre>

Kla: <p>Thank you very much, it really helped me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to find torch.sigmoid source code

moluchase: <p>I know how to implement the sigmoid function, but I don’t know how to find the implementation of <code>torch.sigmoid</code> in pytorch source code.<br>
I coun’t find the relevant implementation function in the torch directory<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/pytorch/tree/master/torch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/pytorch/tree/master/torch" target="_blank" rel="nofollow noopener">pytorch/pytorch</a></h3>

<p>Tensors and Dynamic neural networks in Python with strong GPU acceleration - pytorch/pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

tom: <p>I wrote <a href="https://lernapparat.de/selective-excursion-into-pytorch-internals/" rel="nofollow noopener">a tutorial on diving into PyTorch code</a> a while ago, but it still is largely accurate.<br>
In particular <code>rgrep sigmoid aten/</code> should be helpful.</p>
<p>Best regards</p>
<p>Thomas</p>
How to warp an image according to 3D location to LDR

fishaning: <p>I have a RGB picture 256 /320/ 3 and can got the pixel world location. According to the world location, I can calulate the spherical coordinates(256/320/3). Now I should warp the RGB image’s pixel to a LDR image (with missing values) which is 160/320/3. How can I implement this warp by Pytorch except for the loop method?<br>
maybe this question can be write as:<br>
out[ height[m,n] , width[m,n] , : ] = RGB[ m , n , : ]</p>
.shape return no value. Bug or not?

Umair_Javaid: <p>The following code returns no value. Is this a bug?</p>
<pre><code class="lang-auto">lab_ = torch.tensor((2))
print("lab_.size: ",lab_.shape)
</code></pre>
<p><code>lab_.size:  torch.Size([])</code></p>

InnovArul: <p>It is not a bug.<br>
In python, (2) is considered as single element scalar, where as (2,) is considered as a tuple with one element (pl. note the comma inside brackets).</p>
<p>To get your expected output,</p>
<pre><code class="lang-auto">lab_ = torch.tensor((2,))
</code></pre>
<p>will print</p>
<pre><code class="lang-auto">torch.Size([1])
</code></pre>

Umair_Javaid: <p>can u please look at the other question I just posted</p>
Copy weights only from a network's parameters

erilyth: <p>Given two sets of parameters, I want to copy only the weights from one set of parameters to the other. Is it possible to do this?</p>

fmassa: <p>I think you can do something like</p>
<pre><code class="lang-python">params1 = model1.named_parameters()
params2 = model2.named_parameters()

dict_params2 = dict(params2)

for name1, param1 in params1:
    if name1 in dict_params2:
        dict_params2[name1].data.copy_(param1.data)
</code></pre>

chenchr: <p><a class="mention" href="/u/fmassa">@fmassa</a><br>
Hello, can this achieved by load_static_dict? If net1() is build on the base of net2() and some other layer is added to net1(), and I want to finetune net1() using net2()'s weight, can I directly use load_static_dict? Thanks!</p>

dEathEater: <p><a class="mention" href="/u/chenchr">@chenchr</a> I think it works. One can try <code>model.fc4.load_state_dict(model.fc3.state_dict())</code> to update<code>fc4</code> layer’s parameters using the <code>fc3</code> layer.</p>
<p><a class="mention" href="/u/fmassa">@fmassa</a>, However not sure if <code>clone()</code> , <code>state_dict()</code> or deepcopy would be a better choice. It would be great if you can elaborate upon the differences!</p>

aksg87: <aside class="quote no-group" data-username="fmassa" data-post="2" data-topic="5841">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/48/58_2.png" class="avatar"> fmassa:</div>
<blockquote>
<p>params1 = model1.named_parameters() params2 = model2.named_parameters()</p>
</blockquote>
</aside>
<p>Is there a better way to copy layer parameters from one model to another in 2020 (when trying to transfer a trained encoder or something else)?</p>
<p>I created this helper function per the discussion above but it doesn’t seem to be working as expected!</p>
<pre data-code-wrap="python"><code class="lang-python">def copyParams(module_src, module_dest):
    params_src = module_src.named_parameters()
    params_dest = module_dest.named_parameters()

    dict_dest = dict(params_dest)

    for name, param in params_src:
        if name in dict_dest:
            dict_dest[name].data.copy_(param.data)</code></pre>

jose: <p>Any news on this? I am also moving forward implementing this function. Basically, I want to do some operations that will accumulate the gradient information in the form of a delta_params. Then I want to apply it to the original params, and replace the params in the model with original_params + delta_params.</p>

aksg87: <p>I believe that code actually does copy parameters as expected. I had another issue which is why I was getting strange results (I didn’t copy the original model’s decoder).</p>