akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Help needed with Data loading

bolt25: <p>I’m currently working on Image denoising task.<br>
I have two folders with me- one folder contains clean images and the other folder contains noisy images.<br>
NOTE: the images in both the folders are in sequence. i.e. If clean folder has an image 1.png then 1.png is the noisy image corresponding to that clean image.</p>
<p>I’ve selected MSE loss as my loss class<br>
Now the problem I’m facing is that when I’m writing a loop to train my CNN, I have to feed both the output of the model ( output = model(noisy) )  and the clean image</p>
<p><code>loss=criterion(output,clean)</code></p>
<p>I don’t know how to load both the images simultaneously in the training loop.</p>
<p>Can anyone help me out with it.<br>
Thanks in advance!</p>

Nikronic: <p>If, you need to define your own custom dataset, you can load both images in <code>__getitem__</code> method by using <code>CustomDataset</code>. Here is post for such a situation: <a href="https://discuss.pytorch.org/t/custom-dataset-with-some-preprocessing/30385/2" class="inline-onebox">Custom Dataset with some preprocessing</a></p>
<p>Also, <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">dataloading tutorial</a> has complete info about problems with customized data.</p>
<p>Bests</p>

bolt25: <p>Thank you so much!<br>
I understood what I need to do in order to load both the images simultaneously.<br>
Can’t wait to implement this!</p>
<p>P.S according to <a class="mention" href="/u/nikronic">@Nikronic</a>’s suggestion I further watched a video on youtube which explains how to make this type of CustomDataset.<br>
<a href="https://www.youtube.com/watch?v=ZoZHd0Zm3RY" rel="nofollow noopener">Link to the video here!</a></p>
High memory consumption when moving weights to another device

Chame_call: <p>I’ve got a strange behavior moving a network from ‘cpu’ device to ‘cuda’.<br>
When it happens there’s a huge allocation of RAM which is not appropriate for me…</p>
<p>I made a snapshot of memory allocation:</p>
<blockquote>
<h1>Line #    Mem usage    Increment   Line Contents</h1>
<pre><code> 21    227.0 MiB    227.0 MiB       @profile
 22                                 def __init__(self):
 23    227.0 MiB      0.0 MiB           ckpt_dir = 'hair_extraction/pytorch_hair_segmentation/models/pspnet_squeezenet_sgd_lr_0.002_epoch_46_test_iou_0.882.pth'
 24    227.0 MiB      0.0 MiB           network = 'pspnet_squeezenet'
 25    230.3 MiB      3.3 MiB           self.device =  'cuda' if torch.cuda.is_available() else 'cpu'
 26
 27    230.3 MiB      0.0 MiB           assert os.path.exists(ckpt_dir)
 28
 29    230.3 MiB      0.0 MiB           self.test_image_transforms = std_trnsf.Compose([
 30    230.3 MiB      0.0 MiB               std_trnsf.ToTensor(),
 31    230.3 MiB      0.0 MiB               std_trnsf.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
 32                                         ])
 33    248.0 MiB     17.8 MiB           self.net = get_network(network)
 34   2942.6 MiB   2694.6 MiB           self.net.to(self.device)
 35   2952.6 MiB     10.0 MiB           state = torch.load(ckpt_dir, map_location=torch.device(self.device))
 36   2952.6 MiB      0.0 MiB           self.net.load_state_dict(state['weight'])
 37   2952.6 MiB      0.0 MiB           self.net.eval()
</code></pre>
</blockquote>
<p>as can be seen on line 34 there’s a big allocation of memory…<br>
buf when I explicitly define device as ‘cpu’ things go in the right way:</p>
<h1><a name="line-mem-usage-increment-line-contents-1" class="anchor" href="#line-mem-usage-increment-line-contents-1"></a>Line #    Mem usage    Increment   Line Contents</h1>
<pre><code>21    226.4 MiB    226.4 MiB       @profile
22                                 def __init__(self):
23    226.4 MiB      0.0 MiB           ckpt_dir = 'hair_extraction/pytorch_hair_segmentation/models/pspnet_squeezenet_sgd_lr_0.002_epoch_46_test_iou_0.882.pth'
24    226.4 MiB      0.0 MiB           network = 'pspnet_squeezenet'
25    226.4 MiB      0.0 MiB           self.device =  'cpu'
26
27    226.4 MiB      0.0 MiB           assert os.path.exists(ckpt_dir)
28
29    226.4 MiB      0.0 MiB           self.test_image_transforms = std_trnsf.Compose([
30    226.4 MiB      0.0 MiB               std_trnsf.ToTensor(),
31    226.4 MiB      0.0 MiB               std_trnsf.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
32                                         ])
33    243.4 MiB     17.1 MiB           self.net = get_network(network)
34    243.8 MiB      0.4 MiB           self.net.to(self.device)
35    256.3 MiB     12.5 MiB           state = torch.load(ckpt_dir, map_location=torch.device(self.device))
36    256.3 MiB      0.0 MiB           self.net.load_state_dict(state['weight'])
37    256.3 MiB      0.0 MiB           self.net.eval()
</code></pre>
<p>how it can be possible?</p>
How to quantize torch.zeros_like(x)?

juanmanpr: <p>I am getting all kind of errors like:</p>
<p>RuntimeError: Could not run ‘aten::empty.memory_format’ with arguments from the ‘QuantizedCPUTensorId’ backend. ‘aten::empty.memory_format’ is only available for these backends: [CUDATensorId, SparseCPUTensorId, VariableTensorId, CPUTensorId, MkldnnCPUTensorId, SparseCUDATensorId].</p>
<p>when using hacky solutions like nn.quantized.FloatFunctional().mul(x,0)</p>
<p>What is the correct way to quantize this operation?</p>

juanmanpr: <p>sorry, it seems that nn.quantized.FloatFunctional().mul_scalar(x,0) hacky solutions work in pytorch 1.5 (I was using 1.4).</p>
How to deploy model in Java ？

yonxin3344520: <p>我在Ubuntu系统使用java调用model，总是提示"No<br>
suchMethod org.pytorch.NativePeer.forward ", 以下是我的代码。请不吝指教。非常谢谢</p>
<p>I use java to call model in Ubuntu system, always prompt “no”</p>
<p>suchMethod org.pytorch.NativePeer . forward ", here is my code. Please don’t be stingy. Thank you very much！</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a5331d373b6e15f58eb5100defdea37aac614ab.png" alt="image" data-base62-sha1="3KSEN0rzsfOoOCDo2z60vQ6i9XJ" width="529" height="252"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/6022348c58f6be2e1b9a7b6e0dcc26c06a4e7805.png" alt="image" data-base62-sha1="dIr7xsKB8euaHbLLJdycCoAy1SZ" width="623" height="399"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/1/413b1292d672112067c80e62abc5432558bb04e7.png" alt="image" data-base62-sha1="9j3B4tWIAwJEe11ZNV3B8RIprzF" width="604" height="388"></p>
Lightning vs Ignite

Aldebaran: <p>Currently, we have <a href="https://github.com/PyTorchLightning/pytorch-lightning" rel="nofollow noopener">Lightning</a> and <a href="https://github.com/pytorch/ignite#tutorials" rel="nofollow noopener">Ignite</a> as a high-level library to help with training neural networks in PyTorch. Which of them is easier to train in a multi GPU environment?</p>

sree_harsha: <p>I have used PyTorch Lightning. (While I can’t compare the two, as I haven’t used Ignite).</p>
<p>It has been the smoothest experience as far as I have come across, w.r.t multi-GPU training. Changing from a single GPU to a multi-GPU setup is as simple as setting num_gpus in trainer.fit() to as many as you’d like to use.</p>
<p>TPU support is also integrated, where you’d just specify num_tpu_cores, without changing any code.</p>

Aldebaran: <p>Thank you very much for your contribution.<br>
I started using Ignite after read the exciting <a href="https://pytorch.org/ignite/quickstart.html" rel="nofollow noopener">quickstart</a> guide showing the essentials for define and training a simple model. But the provided examples which use multi GPU training do not seem to follow the same simplicity.</p>
<p>Currently, the stable (v0.3.0) release relies only on native PyTorch distributed API where users need to manually set up distributed proc group, wrap model with <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">nn.parallel.DistributedDataParallel</a> and execute the script with torch.distributed.launch tool, or use mp.spawn.</p>

sree_harsha: <p>I would go with Lightning then.</p>
<p>The documentation is pretty clear and readable.</p>
<p><a href="https://pytorch-lightning.readthedocs.io/en/0.7.6/multi_gpu.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch-lightning.readthedocs.io/en/0.7.6/multi_gpu.html</a></p>
What is the relationship between num_workers and the distributed process in DistributedDataParallel?

akashs: <p>What is the relationship between <code>num_workers</code> of the data loader in <code>DistributedDataParallel</code> mode? For example, if the <code>num_workers=8</code> and the number of GPUs is 4, then whether each distributed process in <code>DistributedDataParallel</code> mode will get <code>num_workers 2 or 8</code>?</p>

mrshenli: <p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> for data loader question <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Hi,</p>
<p>They are completely independent.<br>
DataLoader processes will only be used to load data from your Dataset.</p>
<p>The processes for <code>DistributedDataParallel</code> have nothing to do with it.</p>
Model dtype cant set

IGero228: <p>Hi.<br>
Have</p>
<blockquote>
<p>self.lin = nn.Linear(in_dim, hidden_dim).double()</p>
</blockquote>
<p>and when try to use</p>
<blockquote>
<p>out = self.lin(out)</p>
</blockquote>
<p>Receive error:</p>
<blockquote>
<p>RuntimeError: Expected object of scalar type Float but got scalar type Double for argument <span class="hashtag-raw">#2</span> ‘mat1’ in call to _th_addmm</p>
</blockquote>
<p>out.float() fix it but i dont understand why self.lin on float type when i directly set double on it!?</p>

albanD: <p>Do you cast the overall model to a different type after setting <code>self.lin</code> to double by any chance?</p>
Speed Confusion

Meraki: <p>Hi all, I am a new beginner of Pytorch. I found previous disccusion talking about the speed for backward is 1 or 2 times of forward. But my backward() is rather too slow.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Nets(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden1 = nn.Linear(26, 15)
        self.hidden2 = nn.Linear(15, 15)
        self.out = nn.Linear(15, 1)

    def forward(self, x):
        # flow to 1st hidden layer
        x = self.hidden1(x)
        x = torch.tanh(x)

        # flow to 2nd hidden layer
        x = self.hidden2(x)
        x = torch.tanh(x)

        # flow to output layer
        x = self.out(x)
        output = torch.sum(x).view(-1)
        return output

inputs = torch.randn(63,26)

%%timeit
net = Nets()
a = net(inputs)
## 320 µs ± 11.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
a.backward(retain_graph=True)
## 25.3 s ± 519 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
</code></pre>
<p>Overall, is that normal or I could make it a little bit faster? Thanks.</p>

albanD: <p>Hi,</p>
<p>I cannot reproduce it.<br>
Using the modified script here:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import time

class Nets(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden1 = nn.Linear(26, 15)
        self.hidden2 = nn.Linear(15, 15)
        self.out = nn.Linear(15, 1)

    def forward(self, x):
        # flow to 1st hidden layer
        x = self.hidden1(x)
        x = torch.tanh(x)

        # flow to 2nd hidden layer
        x = self.hidden2(x)
        x = torch.tanh(x)

        # flow to output layer
        x = self.out(x)
        output = torch.sum(x).view(-1)
        return output

inputs = torch.randn(63,26)

net = Nets()
start = time.time()
a = net(inputs)
print("fw: ", time.time() - start)
## 320 µs ± 11.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

start = time.time()
a.backward(retain_graph=True)
print("bw: ", time.time() - start)
</code></pre>
<p>I get</p>
<pre><code class="lang-auto">fw:  0.0009152889251708984
bw:  0.0009768009185791016
</code></pre>
<p>which is what I was expected.<br>
Maybe timeit has some weird side effects?</p>
Batch norm and dropout

rzhang63: <p>Hi,</p>
<p>I’m playing with the MC dropout (Yarin Gal) idea which inserts a dropout layer after every weight layer. But for many pretrained models like ResNet, they are using BatchNorm instead of dropout. So does it still make sense to use have both dropout and batchnorm in those models at the same time? Is there a reason why dropout is not used anymore in recent architecture designs?</p>
<p>Thanks!</p>

mailcorahul: <p>Dropouts in general are used in conjunction with fully connected layers. My understanding is that BatchNorm along with helping the model to converge faster, also acts as a regularizer(which serves the purpose).<br>
It ‘could’ be derived from experimental evidence that dropouts may not be well suited for convolutions. That said, there is nothing stopping you in using batchnorm and dropout together.</p>
Masking out gradient before backpropagating

ParikaGoel: <p>I want to mask out the gradients computed by loss function before backpropagating it further. Below I have written a piece of code to explain what I am trying to do:</p>
<pre><code class="lang-python"> import torch
 
 model = torch.nn.Linear(2, 4)
 input = torch.rand(2)
 target = torch.rand([2,2])
 mask = torch.Tensor([[True, False],
                     [True, True]])

 model.train()
 for i in range(3):
     output = model(input)
     output = torch.reshape(output, [2,2])
     loss = torch.nn.functional.mse_loss(target, output)

     loss.backward()
</code></pre>
<p>My mask is of the shape [2,2]. Backward function corresponding to mse loss would give me a grad of shape [2,2]. I want to mask this gradient using the mask tensor before backpropagating it further.</p>
<p>Could someone please give me an insight into how to do this ?</p>
<p>I have followed some topics related to this. But none of them solves my problem</p>

albanD: <p>Hi,</p>
<p>You can do this using a hook which will return the new value to use</p>
<pre><code class="lang-auto">     output = torch.reshape(output, [2,2])
     output.register_hook(lambda grad: grad * mask.float())
     loss = torch.nn.functional.mse_loss(target, output)

</code></pre>

ParikaGoel: <p>Thanks. It worked for me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Multi-class Segmentation in UNet

Asjad_Murtaza: <p>Hi, I am new to PyTorch. I need to train Unet model in PyTorch. The dataset has 3 classes, let’s call them A,B,C. The dataset directory consists of an images folder and three other folders which contain masks for class A,B and C. How do I load these masks for training? Should I use separate dataloaders and train the model for each class one by one or is there any way to do it simultaneously</p>

mailcorahul: <p>Hi,<br>
Shouldn’t there be one mask for every image instead of 3? Each output mask will contain the necessary classes, right?</p>
pytorch java demp

aaronlyt: <p><strong>pytorch</strong> : 1.5 statble<br>
<strong>libtorch</strong> : <a href="https://download.pytorch.org/libtorch/cu102/libtorch-shared-with-deps-1.5.0.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu102/libtorch-shared-with-deps-1.5.0.zip</a></p>
<p><strong>run command</strong> : ./gradlew run --stacktrace --debug</p>
<h3><a name="stop-at-1" class="anchor" href="#stop-at-1"></a>stop at:</h3>
<p>&lt;=========----&gt; 75% EXECUTING [1m 37s]</p>
<blockquote>
<p>:run</p>
</blockquote>
<h3><a name="debug-infomation-2" class="anchor" href="#debug-infomation-2"></a>debug infomation:</h3>
<p>2020-06-10T13:43:15.052+0800 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.<br>
2020-06-10T13:43:15.052+0800 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.<br>
2020-06-10T13:43:15.053+0800 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.</p>
<p>2020-06-10T10:45:56.962+0800 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: STARTED<br>
2020-06-10T10:45:56.962+0800 [INFO] [org.gradle.process.internal.DefaultExecHandle] Successfully started process ‘command ‘/home/ks/software/jdk-9.0.4/bin/java’’<br>
2020-06-10T10:45:56.962+0800 [DEBUG] [org.gradle.process.internal.ExecHandleRunner] waiting until streams are handled…<br>
2020-06-10T10:45:57.206+0800 [ERROR] [system.err] Exception in thread “main” java.lang.NoSuchMethodError: nativeNewTensor<br>
2020-06-10T10:45:57.207+0800 [ERROR] [system.err] at org.pytorch.NativePeer.forward(Native Method)<br>
2020-06-10T10:45:57.207+0800 [ERROR] [system.err] at org.pytorch.Module.forward(Module.java:37)<br>
2020-06-10T10:45:57.207+0800 [ERROR] [system.err] at demo.App.main(App.java:17)<br>
2020-06-10T10:46:02.923+0800 [LIFECYCLE] [org.gradle.cache.internal.DefaultFileLockManager]<br>
2020-06-10T10:46:02.923+0800 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.</p>
Torchvision's transforms function seems to rotate image

Tzeviya: <p>I’m loading images into a <code>dataset</code> class. However, when using torchvision’s <code>transforms</code> functions, it seems to be rotating the images 90 degrees counterclockwise. Is there a reason for this, and a way to prevent it?</p>
<pre><code class="lang-auto">transform = transforms.Compose([transforms.Resize([256, 256]),
									transforms.RandomCrop(224),
									transforms.ToTensor()])

	train_dataset = Images(data_path, transform, 'train')
</code></pre>
<pre><code class="lang-auto">class Images(data.Dataset): 
	def __init__(self, data_path, transform, mode):
		self.examples, self.labels = make_dataset(data_path, mode) 
		self.transform = transform

	def __getitem__(self, idx):

		example = self.examples[idx]
		this_img = Image.open(example)
		this_img.load()

                if self.transform:
			this_img = self.transform(this_img)
</code></pre>
<p>When I debug and print  <code>this_img</code> at this point, it appears rotated. How can I prevent this? Thanks!</p>

Nikronic: <p>Hi,</p>
<p>There is no rotating in your code and transforms will not do that. I think the way you visualize your image has some problems. Here is the way I plot images:</p>
<pre><code class="lang-auto">x = Image.open('cat.jpg')
x.load()
transform = transforms.Compose([transforms.Resize([256, 256]),
									transforms.RandomCrop(224),
									transforms.ToTensor()])
new_x = transform(x)
plt.imshow(new_x.permute((1, 2, 0)))  # permute channels to get numpy style image
</code></pre>
<p>Bests</p>

Tzeviya: <p>You’re right. I was using:</p>
<p><code>plt.imsave('x.png', this_img.transpose(0,2).numpy())</code></p>
<p>to save the image, and I can see now that transposing resulted in the wrong order of dimensions. Thanks so much!</p>
Torchtext - can we save the processed dataset and fields?

yifanwang: <p>Hi,</p>
<p>I’m currently using torchtext, but I found that creating <code>Dataset</code>object and calling Field’s <code>build_vocab</code> takes quite a long time, especially when the tokenizer is complicated. However, I failed saving them with pickle. Is there a way that we can save the processed dataset and fields, so that we can speed up data loading?</p>
<p>Thanks.</p>

Johan_pow: <p>Hi did you find a good way to do it? I have found this way however,</p>
<pre><code class="lang-auto">TEXT = data.Field(sequential=True, tokenize=tokenizer, lower=True,fix_length=200,batch_first=True)
with open("model/TEXT.Field","wb")as f:
     dill.dump(TEXT,f)
</code></pre>
<p>from  <a href="https://stackoverflow.com/questions/53421999/how-to-save-torchtext-dataset" rel="nofollow noopener">https://stackoverflow.com/questions/53421999/how-to-save-torchtext-dataset</a></p>

MFajcik1: <p>But can you currently somehow save dataset?</p>

cb13e917983afd9ad4e7: <p>Currently, Pytorch has provide the interface to save/load the processed vocab. You can directly use the torch.save() and torch.load() to operate the TEXT.Field object.</p>
Cuda always get out of memory in google colabs

rizal_alfarizi: <p>So i have 2DCNN models to classify image, there are just 2 class, i have 300 images each class.<br>
here is my nn module class</p>
<pre><code class="lang-auto">class Flatten(Module):

    def forward(self, input):

        return input.view(input.size(0), -1)

class ActionNet(Module):

    def __init__(self, num_class=4):

        super(ActionNet, self).__init__()

        

        self.cnn_layer = Sequential(

            #conv1

            Conv2d(in_channels=1, out_channels=32, kernel_size=1, bias=False),

            BatchNorm2d(32),

            PReLU(num_parameters=32),

            MaxPool2d(kernel_size=3),

            #conv2

            Conv2d(in_channels=32, out_channels=64, kernel_size=1, bias=False),

            BatchNorm2d(64),

            PReLU(num_parameters=64),

            MaxPool2d(kernel_size=3),

            #flatten

            Flatten(),

            Linear(576, 128),

            BatchNorm1d(128),

            ReLU(inplace=True),

            Dropout(0.5),

            Linear(128, num_class)

        )

    

    def forward(self, x):

        x = self.cnn_layer(x)

        return x
</code></pre>
<p>i split training and test 80%:20%, for more detail here is my training code: <a href="https://pastebin.com/uzBFTrDc" rel="noopener nofollow ugc">https://pastebin.com/uzBFTrDc</a> , when i try to train with colab there is an error:</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.90 GiB total capacity; 15.20 GiB already allocated; 1.88 MiB free; 15.20 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>i use google colab because i don’t have powerfull GPU and implement batch that i dont know wether it’s correct or not, the training data is just 400 image with size 32x32,i think colabs is powerfull enough to train my data, anyone can help me?</p>

ptrblck: <p>I cannot access the training code, but apparently the device memory is not sufficient for your current training.<br>
Do you get this error in the first iteration or after a few iterations or epochs?<br>
In the latter case, do you see an increase in the device memory?<br>
If so, make sure you are not storing tensors, which are attached to the computation graph, such as the <code>loss</code> or <code>output</code> of your model.<br>
If you are storing these tensors in e.g. a <code>list</code>, detach them before storing via:</p>
<pre><code class="lang-python">losses.append(loss.detach())
</code></pre>

rizal_alfarizi: <p>i solved this problem, out of memory is because i didn’t use batch in validation data</p>
Manually select batches from DataLoader

Zador: <p>Hi,<br>
I have been following some pytorch tutorials where they use datasets.ImageFolder and then utils.data.DataLoader to iterate across the data set. This, however is a big problem for me, as the task I am trying to solve strictly requires me to have full knowledge on which specific image I am passing through my neural network.</p>
<p>In my task I am comparing images, and the images are numbered, and so when comparing some images with my net I need to be able to specify that I wish to run image ‘i’ and then image ‘j’ through the neural network.</p>
<p>This is how I am imagining it:</p>
<pre><code class="lang-auto">image_datasets = datasets.ImageFolder(os.path.join(data_dir, 'food'), transform= data_transforms)

dataloaders = torch.utils.data.DataLoader(image_datasets, batch_size=1, shuffle=False, num_workers=0)

output1 = model(dataloaders[i])
output2 = model(dataloaders[j])
</code></pre>
<p>Nothing like this seems to exist. How can I access data of my choice?</p>

ptrblck: <p>If you don’t need the batching, shuffling, or the usage of multiple workers from the <code>DataLoader</code>, you could directly access the <code>image_datasets</code> with the index.<br>
Note that you would need to add a batch dimension to your data and target, since the <code>Dataset</code> will not create this dimension for you.</p>
<p>Alternatively, if you want to use the <code>DataLoader</code>, I would recommend to implement a custom sampler, which gives you the ability to specify which indices are returned in which step.</p>

Zador: <p>The first one worked, thanks for the help!</p>
AttributeError: 'int' object has no attribute 'detach'

hfdp: <p>Anyone got a tip for this? This is the code and error it gives:</p>
<pre><code># Training
trg_loss = [] 
forward_model.train()
with torch.set_grad_enabled(True):
    for data_library['100MHz'], data_library['250MHz'] in training_generator:
        if(core_num&gt;1):
            Parallel(n_jobs=core_num)
            #print(data_library['20MHz'].shape)
            delayed(train)(data_library[prior],data_library[target])
        else:
            new_loss = train(data_library[prior],data_library[target])
            trg_loss.append(new_loss)
        # Parallel(n_jobs=core_num)
        # print(data_library['20MHz'].shape)
        # delayed(train)(data_library[prior],data_library[target])
#print("\nTraining Loss: ", sum(trg_loss).detach().numpy().max()/len(trg_loss))
FILE.write("\nEpoch "+str(epoch)+": Training Loss = " +str(sum(trg_loss).detach().numpy().max()/len(trg_loss)))
</code></pre>
<p>with the error being:</p>
<p><strong>AttributeError: ‘int’ object has no attribute 'detach’</strong></p>

hfdp: <p>Do you have any idea how I can get it to print properly though?</p>

ptrblck: <p>Could you check the type and shape of <code>sum(trg_loss)</code>?<br>
If <code>trg_loss</code> contains tensors, <code>sum(trg_loss)</code> should return another tensor, which shouldn’t raise the error on calling <code>detach()</code>.<br>
However, if <code>new_loss</code> is an int, then this won’t work and you would have to create a tensor first or use the list entries directly. Note that creating a tensor of Ints, will not contain any Autograd history, so that you wouldn’t need to call <code>detach()</code> at all.</p>
ModuleNotFoundError: No module named 'Conv'

alameer: <p>I am getting the above error message. Which module do i need to install to use this module please?</p>
<p>I have tried to install <code>conda install -c conda-forge pytorch_spline_conv</code></p>
<p>But still getting an error message when running the code?</p>
<p>Many Thanks</p>

ptrblck: <p>Is the <code>Conv</code> module specific to <code>pytorch_spline_conv</code> or the PyTorch <code>nn.ConvXd</code>?<br>
Also, you’ve used to category <code>kornia</code>, but it seems to be unrelated to it or is <code>kornia</code> raising this error?</p>

alameer: <p>Thanks! I think it’s specific for nn.ConvXd I have changed the category to vision only.</p>

ptrblck: <p>Could you post the code, which raises this issue, please?<br>
Based on the error message, it seems you are trying to use:</p>
<pre><code class="lang-python">import Conv
&gt; ModuleNotFoundError: No module named 'Conv'
</code></pre>
<p>If you want to use the <code>nn.ConvXd</code> layers, you would have to use:</p>
<pre><code class="lang-python">import torch.nn as nn
conv = nn.Conv2d(...)
</code></pre>

alameer: <p>Many Thanks for your help. It turn out that i haven’t copied the file Conv.py that’s why i was getting this error message. Thanks for your reply.</p>
How conv_transpose2d is implemented?

glenguo06: <p>Hi, recently I need to implement functional.conv_transpose2d() with pure C++ code from scratch, and use cudnn if needed. Where can I follow in detail how this function in PyTorch or ATen is actually implemented?</p>

ptrblck: <p>The general dispatching is done in <a href="https://github.com/pytorch/pytorch/blob/4c4b9916ef806b69413365c7c406cb2a375680bc/aten/src/ATen/native/Convolution.cpp">ATen/native/Convolution.cpp</a>, in particular the cudnn implementation is called in <a href="https://github.com/pytorch/pytorch/blob/4c4b9916ef806b69413365c7c406cb2a375680bc/aten/src/ATen/native/Convolution.cpp#L708-L712">these lines of code</a>.<br>
This will then call into <a href="https://github.com/pytorch/pytorch/blob/4c4b9916ef806b69413365c7c406cb2a375680bc/aten/src/ATen/native/cudnn/Conv.cpp#L1038-L1048">cudnn_convolution_transpose_forward</a>, which is the backward operation of a vanilla convolution.</p>
How to compile binary with libtorch without CUDA

divinho: <p>Is that possible somehow? I looked at the Caffe2 and Torch cmake files and there did not seem to be a straightforward way to do it (but I’m a newbie with C++ compilation and cmake).</p>

albanD: <p>I am not sure what you want. But the <code>NO_CUDA=1</code> flag will disable all cuda stuff.</p>

divinho: <p>I want to have no dependencies on CUDA when using libtorch to compile binaries, as I’m not going to be using a GPU.</p>
<p>I’m now thinking the way to do this is to compile pytorch without CUDA, is that possible? <a href="https://github.com/pytorch/pytorch/issues/3030" rel="nofollow noopener">There’s a year old issue where the answer is no it isn’t</a>.</p>

albanD: <p>As I said above, running the compilation with <code>NO_CUDA=1</code> env variable set will do a compilation and install without any cuda dependency.</p>

divinho: <p>Sorry, and thanks.</p>
<p>How could one compile it without installing it? I don’t want it to overwrite my existing installation, which I assume will happen when doing <code>python3 setup.py install</code>.</p>

albanD: <p>You can do <code>python3 build</code> to only build cpp dependencies.</p>

RuFAI: <p>How do we set  the No_CUDA = 1 flag? In the Cmake file or something else?</p>

b93901190: <p>I set the environment variable NO_CUDA as 1 in my windows OS. It is not working. CMake still asked me cuda stuff.<br>
The error message is following</p>
<p>– Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.18363.<br>
CUDA_TOOLKIT_ROOT_DIR not found or specified<br>
– Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY)<br>
CMake Warning at C:/libtorch/share/cmake/Caffe2/public/cuda.cmake:31 (message):<br>
Caffe2: CUDA cannot be found.  Depending on whether you are building Caffe2<br>
or a Caffe2 dependent library, the next warning / error will give you more<br>
info.<br>
Call Stack (most recent call first):<br>
C:/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)<br>
C:/libtorch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)<br>
CMakeLists.txt:4 (find_package)</p>

b93901190: <p>reply to myself,<br>
just use the libtorch that does not need cuda.<br>
release: <a href="https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-1.5.0.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-1.5.0.zip</a></p>
<p>debug: <a href="https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-debug-1.5.0.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-debug-1.5.0.zip</a></p>

peterjc123: <p>Updated answer:</p>
<pre><code class="lang-auto"># Bash
USE_CUDA=0 python setup.py install
# CMD (Windows)
set USE_CUDA=0
python setup.py install
# Powershell
$env:USE_CUDA="0"
python setup.py install
</code></pre>
TypeError: cannot unpack non-iterable function object

hfdp: <p>Hello,</p>
<p>I get this error ‘TypeError: cannot unpack non-iterable function object’<br>
When trying to run the training which looks as follows:</p>
<p>The mistake seems to be in the line that says Parallel(n_jobs=core_num)(delayed(train)(data_library[prior],data_library[target]))</p>
<pre><code class="lang-auto">&gt;    	# Training
&gt;     trg_loss = [] 
&gt;     forward_model.train()
&gt;     with torch.set_grad_enabled(True):
&gt;         for data_library['20MHz'], data_library['100MHz'], data_library['250MHz'] in training_generator:
&gt;             if(core_num&gt;1):
&gt;                 Parallel(n_jobs=core_num)(delayed(train)(data_library[prior],data_library[target])) 
&gt;             else:
&gt;                 new_loss = train(data_library[prior],data_library[target])
&gt;                 trg_loss.append(new_loss)
&gt;             # Parallel(n_jobs=core_num)
&gt;             # print(data_library['20MHz'].shape)
&gt;             # delayed(train)(data_library[prior],data_library[target])
&gt;     print("\nTraining Loss: ", sum(trg_loss).detach().numpy().max()/len(trg_loss))
&gt;     FILE.write("\nEpoch "+str(epoch)+": Training Loss = " +str(sum(trg_loss).detach().numpy().max()/len(trg_loss)))
</code></pre>
<p>All help is welcome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
thanks!</p>

ptrblck: <p>I don’t know how <code>data_library</code> etc. are defined, but the issue might have been raised by <code>joblib</code> and seems to be similar to <a href="https://stackoverflow.com/questions/59672888/using-joblib-and-getting-cannot-unpack-non-iterable-function-object">this one</a>.</p>