akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How do i write a custom Data Loader?

username: BadTimeManagement
cooked: <p>Hi, I’m new to pytorch. Is there a guideline to write your own Data Loader?</p>
<p>Thank you.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Here is the official <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">custom data loading tutorial</a>. Also, this question has been answered for many different situations in this forum.</p>
<p>Bests</p>

username: BadTimeManagement
cooked: <p>Sorry, still did not understand data loader.<br>
Is this part the Custom Data Loader in the tutorial?<br>
“”“Face Landmarks dataset.”""</p>
<pre><code>def __init__(self, csv_file, root_dir, transform=None):
    """
    Args:
        csv_file (string): Path to the csv file with annotations.
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    self.landmarks_frame = pd.read_csv(csv_file)
    self.root_dir = root_dir
    self.transform = transform

def __len__(self):
    return len(self.landmarks_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    img_name = os.path.join(self.root_dir,
                            self.landmarks_frame.iloc[idx, 0])
    image = io.imread(img_name)
    landmarks = self.landmarks_frame.iloc[idx, 1:]
    landmarks = np.array([landmarks])
    landmarks = landmarks.astype('float').reshape(-1, 2)
    sample = {'image': image, 'landmarks': landmarks}

    if self.transform:
        sample = self.transform(sample)

    return sample</code></pre>

username: Nikronic
cooked: <p>Actually, this is a custom Dataset which can be used as an input to a DataLoader. If you are looking for custom DataLoader, usually the only parameter you can modify is collate_fn which corresponds to batching.</p>

username: BadTimeManagement
cooked: <p>What if I want to add functionality to data loader, I plan to add a function in data loader that send a request to a CPU cloud via MPI? Is it possible? Thankyou.</p>

username: Nikronic
cooked: <p>Oh, I have never worked with remote servers and clusters so I cannot answer this question, sorry.<br>
I think if you create another topic and add this issue on the title so people can interact better. I was only thinking of batching and loading data in different approaches.</p>
<p>Sorry again, good luck</p>

username: BadTimeManagement
cooked: <p>No worries.Thanks for the help.</p>

username: Dipam_Vasani
cooked: <p>Hey,<br>
If you want to write a custom data loader you can check out my article : <a href="https://towardsdatascience.com/the-pytorch-training-loop-3c645c56665a" rel="nofollow noopener">https://towardsdatascience.com/the-pytorch-training-loop-3c645c56665a</a></p>
<p>However, I would suggest adding the functionality to a custom dataset instead of a data loader.</p>
Are conversions to string differentiable?

username: tiwa
cooked: <p>I’m building a model that generates sequences. Internally, the sequences are a <code>(N,1, length, W)</code> tensor of width-<code>W</code> one-hots, but part of the loss function involves converting sequences to strings (using a dictionary) and passing them as an argument to another PyTorch neural network that returns a scalar. Furthermore, certain one-hots are (deterministically) ignored during this conversion.</p>
<p>Is this okay to do? It isn’t clear to me how gradients would be backpropagated properly, but I’m not getting any errors.</p>

username: albanD
cooked: <p>Hi,</p>
<p>As you mentioned, strings are “discrete” and so gradients for them don’t really make sense. So no you won’t be able to packprop through that op I’m afraid.</p>

username: tiwa
cooked: <p>I thought so! Instead, I’ll need to train a model that can accept a <code>(N,1, *, W)</code> tensor, correct? In order to (deterministically) ignore some one-hots in the original <code>(N,1, length, W)</code> tensor, how should I filter so that gradients are still sensible?</p>
<p>For instance, suppose I have a <code>(1, 1, 10, 3)</code> tensor, and I want to ignore the one-hot <code>(0,1,0)</code>. How would I turn:</p>
<pre><code class="lang-auto">tensor([[[[0., 0., 1.],
          [0., 0., 1.],
          [0., 1., 0.],
          [0., 1., 0.],
          [0., 0., 1.],
          [0., 0., 1.],
          [0., 1., 0.],
          [0., 0., 1.],
          [1., 0., 0.],
          [0., 0., 1.]]]])
</code></pre>
<p>into</p>
<pre><code class="lang-auto">tensor([[[[0., 0., 1.],
          [0., 0., 1.],
          [0., 0., 1.],
          [0., 0., 1.],
          [0., 0., 1.],
          [1., 0., 0.],
          [0., 0., 1.]]]])
</code></pre>
<p>in a gradient-friendly way?</p>

username: albanD
cooked: <p>Well, removing entries is easy. You can simply index the elements you want to keep. And the gradients will flow properly back to these elements.</p>
<p>Be careful though when you generate these one-hot encodings that you do so in a way that returns non-zero gradients. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: tiwa
cooked: <p>Thank you! It isn’t clear to me how to index the tensor in bulk like that. How would I do the indexing (without collecting the indices using a for loop?)</p>

username: albanD
cooked: <p>Sure something like that will work:</p>
<pre><code class="lang-auto">import torch

inp = torch.tensor([[[[0., 0., 1.],
          [0., 0., 1.],
          [0., 1., 0.],
          [0., 1., 0.],
          [0., 0., 1.],
          [0., 0., 1.],
          [0., 1., 0.],
          [0., 0., 1.],
          [1., 0., 0.],
          [0., 0., 1.]]]])

to_remove_line = torch.tensor([0., 1., 0.])


to_remove = (inp == to_remove_line).all(-1)
res = inp[to_remove.logical_not()]

print(res)

</code></pre>
<p>Note that the <code>==</code> can be relaxed if you’re using floating points by checking that the difference is below a given threshold.</p>
What is the utility of requires_grad = False

username: Imene.b
cooked: <p>hy every one<br>
i have found a scrips that put the required grad of the 10  first layers to false,<br>
I am trying to understand way do we put requires_grad = False, i know that is to freeze certain layers in the network and to do not calculate gradient in it, but what is the benefit in the calculation</p>

username: albanD
cooked: <p>Hi,</p>
<p>Setting this to False means that no gradient are needed and so all the autograd state does not need to be saved for this part of the net. So it will reduce the memory consumption of the model (and slightly speed it up).</p>

username: Imene.b
cooked: <p>so it is not about avoiding the  destraction of any  information contained into those layers,<br>
during futur training rounds. because i want to preserve the weights of the first layers of my network, to be used in another case.</p>

username: albanD
cooked: <p>Not sure what you mean by “destruction of any information contained into those layers”.<br>
If you reference saved Tensors for the backward pass. They are only useful in that case. Since you don’t need them here, it is irrelevant.<br>
And otherwise, nothing else is removed from the layers.</p>

username: Imene.b
cooked: <p>the informations that i am talking about are the weights values.  i want to gard this values for an other processing. the goal is to compare the final result</p>

username: albanD
cooked: <p>Yes if these Tensors are not in the optimizer, they won’t get updated.<br>
And if they are and their .grad field is None, they won’t get updated as the backward won’t update the .grad field.</p>

username: Imene.b
cooked: <p>tank you very mutch; you help me a lot</p>
Multiply vector tensor in matrix tensor

username: Mahsa_Afsharizade
cooked: <p>Hi, I have two <strong>tensors</strong>, a matrix A and a vector B,</p>
<pre><code class="lang-auto">A=[[1,2,3],[4,5,6],[7,8,9]]
B=[1,10,100]
</code></pre>
<p>I want to multiply 1*[1,2,3], 10*[4,5,6], and 100*[7,8,9], and get the result as:<br>
<code>C=[[1,2,3],[40,50,60],[700,800,900]]</code></p>
<p>How can I do this multiplication to get <strong>tensor</strong> C as above?<br>
Thanks</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>It is a element wise multiplication which can be done using <code>torch.mul(A, B)</code>. You just need to take care of shapes to have a proper broadcasting.</p>
<pre><code class="lang-auto">
x = torch.tensor([[1,2,3],[4,5,6],[7,8,9]])
y = torch.tensor([1,10,100])

torch.mul(x, y.view(-1, 1))
</code></pre>
<p>Bests</p>

username: Mahsa_Afsharizade
cooked: <p>It works.<br>
Thank you.</p>
How to add noise to image in denoising autoencoders

username: arjun_pukale
cooked: <p>I want to implement denoising autoencoder in pytorch.<br>
when I am in the training loop:</p>
<pre><code class="lang-auto">for inputs,labels in trainloader:
        noised_inputs=add_noise(inputs)
        opt.zero_grad()
        outputs = net(noised_inputs)
        loss = loss_fn(outputs,inputs)
        loss.backward()
        opt.step()
</code></pre>
<p>Please tell me how to implement this add_noise function which can add gaussian noise to the entire batch input of images</p>

username: chetan_patil
cooked: <p>You can use the <code>torch.randn_like()</code> function to create a noisy tensor of the same size of input. Then add it.<br>
In your case ,</p>
<pre><code class="lang-auto">def add_noise(inputs):
     noise = torch.randn_like(inputs)
     return inputs + noise
</code></pre>

username: arjun_pukale
cooked: <p>It worked!!! also we can multiply it with factor like 0.2 to reduce the noise</p>
<pre><code class="lang-auto">def add_noise(inputs):
     noise = torch.randn_like(inputs)*0.2
     return inputs + noise
</code></pre>
<p>Original Image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68fc26730ef15656a2e57a90e5e98a34e3d473d8.png" alt="image" data-base62-sha1="eYJSWDZR17l9lxMljJiizsWywas" width="592" height="108"></p>
<p>Noised Image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b88a5f50c2770f99cefa09715a4c02d1904ed6bf.png" alt="image" data-base62-sha1="qkwjAkKjXJthFhpitZ3krja8hMr" width="592" height="108"></p>
Cuda version of torch.eye()

username: Rocco_Fortuna
cooked: <p>What would be the best way to make an identity matrix directly on GPU?<br>
I couldn’t find a torch.cuda.eye() factory function.<br>
I am currently simply making a <code>FloatTensor</code>, then filling it with zeros with <code>.fill_(0)</code> and finally<br>
using <code>torch.masked_scatter_()</code> to put ones on the diagonal.</p>
<p>Any faster/more elegant way to do this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The <code>eye</code> function takes a <code>device=</code> argument. You can specify <code>"cuda"</code> there. Or provide a device object.</p>
Expected more than 1 value in eval() mode

username: Alister_Maguire
cooked: <p>I have a 1D convolutional model with batch norm layers, and it doesn’t seem to like me predicting on a single sample, even when in eval mode. Predicting on more than 1 sample works fine, but I get the following error from predicting on a single sample:</p>
<p>ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 1024])</p>
<p>I’ve seen lots of posts about handling this error when not in eval mode, which makes sense, but I haven’t seen anything on handling this error when already in eval mode.</p>
<p>Here is a section of the model architecture showing one of the batch norm layers:</p>
<p>(conv6): Conv1d(256, 256, kernel_size=(3,), stride=(1,), padding=(1,))<br>
(maxp6): MaxPool1d(kernel_size=3, stride=1, padding=1, dilation=1, ceil_mode=False)<br>
(fc1): Linear(in_features=2560, out_features=1024, bias=True)<br>
(bn1): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)<br>
(drop1): Dropout(p=0.5, inplace=False)</p>
<p>Any ideas?</p>

username: ptrblck
cooked: <p>Since you are not using the running estimates via <code>track_running_stats=False</code>, the batchnorm layer has to calculate the batch statistics even in <code>eval()</code> mode.<br>
From the docs:</p>
<blockquote>
<p>track_running_stats: a boolean value that when set to <code>True</code>, this<br>
module tracks the running mean and variance, and when set to <code>False</code>,<br>
this module does not track such statistics and always uses batch<br>
statistics in both training and eval modes. Default: <code>True</code></p>
</blockquote>
<p>If you are using this mode, you would need to provide more than a single input value.</p>

username: Alister_Maguire
cooked: <p>Interesting… Thanks for pointing this out. The issue that I’ve had with setting track_running_stats to True is that I get the “Unexpected key(s)” error when loading the state dictionary at a later time:</p>
<p>Unexpected key(s) in state_dict: “bn1.running_mean”, “bn1.running_var”, “bn1.num_batches_tracked”, “bn2.running_mean”, “bn2.running_var”, “bn2.num_batches_tracked”, “bn3.running_mean”, “bn3.running_var”, “bn3.num_batches_tracked”</p>
<p>I’ve seen reports of this before, and most people suggest to either remove the keys (which I can do, but then I seem to be in my current predicament again) or use DataParallel. I don’t believe I’m using DataParallel when training/saving, though… I merely call torch.save(state, file_name). Any suggestions on how to best save and load the model while turning on track_running_stats? I’ve tried converting my model to DataParallel when loading, but that didn’t seem to resolve the key errors.</p>

username: Alister_Maguire
cooked: <p>Okay, I figured out why we weren’t able to load models with track_running_stats on. We having some saving/loading functions that weren’t appropriately handling this setting. It would save with the correct setting for track_running_stats, but it would ignore this variable when loading and default to turning it off. Fixing this resolves the issue. Thanks again!</p>
PyTorch cheat sheet?

username: kaiseryet
cooked: <p>Hey guys,</p>
<p>Is there a pytorch cheat sheet like these for numpy?<br>
<a href="https://blog.finxter.com/collection-10-best-numpy-cheat-sheets-every-python-coder-must-own/" class="onebox" target="_blank" rel="nofollow noopener">https://blog.finxter.com/collection-10-best-numpy-cheat-sheets-every-python-coder-must-own/</a></p>
<p>google brought me some, but I was thinking of maybe something better than existing? Thanks.</p>

username: sailfish009
cooked: <p>google ‘pytorch cheatsheet’ and image search:<br>
                    <a href="https://www.sznajdman.com/wp-content/uploads/2018/02/pytorch-cheat.jpg" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://www.sznajdman.com/wp-content/uploads/2018/02/pytorch-cheat.jpg" width="647" height="500">
          </a>

</p>

username: kaiseryet
cooked: <p>thanks, but I’m looking for an even better one…</p>

username: praxs
cooked: <p>Check the official one at <a href="https://pytorch.org/tutorials/beginner/ptcheat.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/ptcheat.html</a>. This can be viewed in text from <a href="https://pytorch.org/tutorials/_sources/beginner/ptcheat.rst.txt" rel="nofollow noopener">https://pytorch.org/tutorials/_sources/beginner/ptcheat.rst.txt</a></p>
How to correctly upload data into the DataLoader

username: spadel
cooked: <p>I have a dataset with image galaxies and corresponding labels as a .h5 file, that I want to train a classification network with. At first, I want to normalize the images and convert it to grayscale. Afterwards, the data shall be splitted into a training set and a validation set. So far my code looks as follows:</p>
<pre><code class="lang-auto">import h5py
import numpy as np
import matplotlib.pyplot as plt
import random
import torch

# Load dataset
f = h5py.File('Galaxy10.h5', 'r')

# The dataset contains two keys: ans and images
# ans represent the labels with shape (21785,) and the images have shape (21785, 69, 69, 3)
labels, images = f['ans'], f['images']

# As a next step, we want to convert the RGB images to grayscale, normalize them and set the std deviation to 1
images = np.dot(images[..., :3], [0.2989, 0.5870, 0.1140])
images = (images - np.mean(images)) / np.std(images)

from torch.utils.data.dataloader import DataLoader
from torch.utils.data.sampler import SubsetRandomSampler

# At first, we want to split our dataset into a training set and a validation set
validation_split = .2

# Creating data indices for training and validation splits:
dataset_size = len(images)
indices = list(range(dataset_size))
split = int(np.floor(validation_split * dataset_size))
train_indices, val_indices = indices[split:], indices[:split]

# Creating PT data samplers and loaders:
train_sampler = SubsetRandomSampler(train_indices)
valid_sampler = SubsetRandomSampler(val_indices)
</code></pre>
<p>How do I correctly load the images with the labels into the DataLoader now? I assume, that I have to make two separate DataLoaders for training set and validation set.</p>
<p>Thanks for your help!</p>

username: swap
cooked: <p>from torch.utils.data import TensorDataset, DataLoader</p>
<p>BATCH_SIZE = 512</p>
<p>tensor_x = torch.Tensor(images[train_indices])<br>
tensor_y = torch.Tensor(labels[train_indices])<br>
dataset_train = TensorDataset(tensor_x, tensor_y)</p>
<p>train_loader = DataLoader(dataset_train, batch_size=BATCH_SIZE)</p>
<h1>you can iterate</h1>
<p>for i, (x, y) in enumerate(train_loader):<br>
print(“x.shape {}, y.shape {}”.format(x.shape, y.shape))</p>
First batch of Imagenet training is slow with sequential loading

username: iacolippo
cooked: <p>Hello, I’m trying to perform a training from scratch on ImageNet with VGG16. Since the loader in the example is quite slow even after resizing the images and I can’t put them on an SSD, I tried to build a sequential data loader by modifying this code: <a href="https://github.com/BayesWatch/sequential-imagenet-dataloader" rel="nofollow noopener">https://github.com/BayesWatch/sequential-imagenet-dataloader</a>.</p>
<p>My machine is a NC24 instance on Azure (4 x K80 GPU). I’m using Python 3.6, PyTorch 0.3.0 with CUDA 8.0 and cuDNN 7.0.</p>
<p>Some weird things happen:<br>
1 - the first batch takes a lot more time than the others. I’m talking about 2-3 minutes for the first batch vs 3 seconds for the following.<br>
2 - 3 seconds is an improvement over the 7-8 seconds for the old dataloader, but BayesWatch readme talks about 0.5s for one batch.</p>
<p>Do you have any idea why nr. 1 happens? I can share the code if it helps (need to make it a bit more readable before)</p>

username: phiwei
cooked: <p>Did you ever figure this out? I am having a similar issue.</p>
Combining basic and advanced indexing in PyTorch

username: Victory
cooked: <p>Is it possible to combine basic and advanced indexing in PyTorch? I have a 3x128x128x128 array which basically stores a 3D vector at each voxel in a 128x128x128 array of voxels. I want to get the vectors stored at a selected few of those voxels.<br>
I use the following code to get those vectors:</p>
<p><em>offset_vectors = my_array[:,voxel_indices]</em></p>
<p>Here, “voxel_indices” is a tuple of 3 tensors representing the indices of the voxels of interest along the three dimensions (The tuple of tensors is in the same format as returned by PyTorch’s <em>nonzero</em> method call with as_tuple=True).</p>
<p>But, the above call gives me the following error message:<br>
<em>ValueError: only one element tensors can be converted to Python scalars</em></p>
<p>Am I doing something wrong or Pytorch doesn’t allow combining basic and advanced indexing as done above? Numpy does allow such combination (<a href="https://numpy.org/doc/stable/reference/arrays.indexing.html" rel="nofollow noopener">https://numpy.org/doc/stable/reference/arrays.indexing.html</a>).</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Do the index tensors in <code>voxel_indices</code> have the same shape or would you need to broadcast?<br>
In the former case, you should be able to index the original tensor via:</p>
<pre><code class="lang-python">x[:, idx[0], idx[1], idx[2]]
</code></pre>
Dataloader to make batch to minimize padding

username: Mainul
cooked: <p>Hi,<br>
I am using <code>DataSet</code> and <code>DataLoader</code> class to load batches. How can I make this behave like <a href="https://torchtext.readthedocs.io/en/latest/data.html#bucketiterator" rel="nofollow noopener">BuckeIterator</a> from torchtext ?</p>
<p>BuckeIterator load data of almost the same size to create each batch to minimize padding.</p>
Network Architecture of forward method

username: sgaur
cooked: <p>Hi I am trying to figure out how data flow in forward method</p>
<p>Is there is any way I can visualize this below part.</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        x = self.fc(x)

        return x
</code></pre>
<p>Thanks</p>

username: ptrblck
cooked: <p>The data will flow through each module from sequentially.<br>
You could visualize this graph via e.g. <a href="https://github.com/szagoruyko/pytorchviz">torchviz</a>.</p>
Is possible to save PyTorch model as string?

username: Peter_Pirog
cooked: <p>I wonder if possible to save trained PyTorch model as string in text file and load the model from this string?</p>
<p>I try with function torch.save(model.state_dict(), PATH)<br>
but there is nos satisfying results.</p>
<p>Peter</p>

username: ptrblck
cooked: <p>Could you explain a bit which results are not satisfying and how the storage as a string would avoid it?</p>
How to register forward hooks for each module

username: Dawid_S
cooked: <p>I’d like to register forward hooks for each module in my network. I have a working code for one module. The most important part looks this way:</p>
<pre><code class="lang-auto">    def __init__(self, model, layer_name):
        self.hook = model.module.linear1.register_forward_hook(self.hook_fn)
</code></pre>
<p>I don’t know, however, how to access modules using layer_name variable.</p>
<p>So I suspect there’s some better way to do this. First, I’d like to list all the layers names (model.children() or model.named_parameters() don’t do this) and second, I’d like to use them somehow with the attached code.</p>

username: ptrblck
cooked: <p>You could use <code>model.named_modules()</code> to iterate all submodules including their names and store the hooks in a <code>dict</code>:</p>
<pre><code class="lang-python">self.hooks = {}
for name, module in model.named_modules():
    hooks[name] = module.register_forward_hook(self,hook_fn)
</code></pre>

username: Dawid_S
cooked: <p>Cool! I overlooked <code>.named_modules()</code> but I knew there has to be something like that.</p>

username: ooodragon
cooked: <p>Hello ptrblck!<br>
I have an additional question…<br>
I want to save inputs (or feature space) of each modules in a dictionary wise<br>
so… for example if I have module names as model.layer1.conv1, model.layer1.conv2<br>
I would like to save their input in a form of<br>
<code>input['model.layer1.conv1'] = [input]</code><br>
and append them as iteration goes<br>
<code>input['model.layer1.conv1'].append(input)</code><br>
I would like to do it with forward hook<br>
I’m not sure if the code you presented here does the job for me</p>
<p>thanks a lot</p>

username: ptrblck
cooked: <p><code>.named_modules()</code> will return the name and module recursively for the complete model.<br>
If you want to just register specific modules, such as <code>nn.Conv2d</code> but not e.g. <code>BottleNeck</code> or other custom modules, you could use a condition to check for the module via <code>if isinstance()</code>.</p>

username: ooodragon
cooked: <p>thanks!<br>
hm… so named_modules will only give me a specific name of the parameter only when I have it iterated over the module…</p>
<p>is there any way to get a specific id of the module without iterating the whole module??<br>
thanks!</p>

username: ptrblck
cooked: <p>Yes, you could either directly access and index the module via:</p>
<pre><code class="lang-python">print(model.layer1.conv1)
</code></pre>
<p>or use <code>getattr(model, name)</code>.</p>