akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Difference in execution time between similar hardwares

username: douglasrizzo
cooked: <p>I have 2 computers, these are their configurations:</p>
<ol>
<li>16 GB RAM, GeForce GTX 1070, i7-8700, Debian testing, Python 3.8.3;</li>
<li>32 GB RAM, GeForce GTX 1070, i7-7700K, Manjaro Linux 20.1, Python 3.8.</li>
</ol>
<p>I run the same experiment in both machines, using PyTorch 1.6.0, CUDA 10.2 and NVIDIA drivers 440.100. In the first machine, one experiment takes 11 hours. In the second, it takes 48 hours.</p>
<p>I am unable to find what’s so significantly different between the two computers that cause such a large difference in execution time. Does anyone have any knowledge about what can cause this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>From checking the CPU <a href="https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=126686,97129">here</a> it looks like one machine has a higher core count and turbo boost frequency compared to the other. This can speed things up a bit.<br>
Also what kind of drive are you using on both machines (if you load your data on the fly from a drive)?</p>

username: douglasrizzo
cooked: <p>Indeed, the slower machine is using a hard drive, while the faster one uses an M.2 drive. However, my data comes from a simulator and is not loaded from any disks, so I don’t believe that is where the slowdown comes from.</p>
<p>Also, while there is indeed a difference in the CPUs, their usage is never maxed out, so I don’t really know if that’s the bottleneck. In any case, I run the same experiments in a server with an Intel Xeon Gold 5118 and a Tesla V100 and the same experiment runs in 28 hours.</p>
<p>I will try and profile my code in some of these machines to see if I find out what’s going on.</p>

username: douglasrizzo
cooked: <p>The problem was that I am using a library for logging my data (Weights &amp; Biases) that I thought worked fully asynchronously and online, but it actually writes the data to a file on disk before logging it to their servers. This created lots of accesses to the hard drive and, in the computers that were not equipped with SSD, slowed the process down quite a bit. Mystery solved.</p>

username: albanD
cooked: <p>Thanks for the update!<br>
Good to know that it was nothing bad on our side <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Libcudart Problem with nightly

username: randomstring
cooked: <p>I am using a package that is trying to import torchcspring.  This package works fine with pytorch 1.6.0 but when I switch to more nightly builds that appear in my ngc based container this import fails with CUDA11 installed.   Where should I report this compatibility issue at?</p>
<pre><code class="lang-auto">import torchcsprng as csprng
  File "/opt/conda/lib/python3.6/site-packages/torchcsprng/__init__.py", line 10, in &lt;module&gt;
    from torchcsprng._C import *
ImportError: libcudart.so.10.2: cannot open shared object file: No such file or directory
</code></pre>

username: tom
cooked: <p>This seems to say that the torchcsprng package you have installed uses CUDA 10.2, so you would need to recompile if you want to use CUDA 11.</p>
<p>Best regards</p>
<p>Thomas</p>

username: randomstring
cooked: <p>Thanks.  Sounds like I need to figure out how to contact the container maintainer, which appears to be no one <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Add additional data such as word features to word embedding

username: keysmash97
cooked: <p>Hi everyone, I’m using pretrain word embedding for nmt task. I have idea about using word features such as named entity to improve nmt quality. Is it possible to concate one hot vector (named entity) to my pretrain word embedding and use it for my nmt model ?<br>
For example:<br>
Given sentence: My name is James .<br>
NE annotated sentence: My|O name|O is|O James|PERSON<br>
With James|PERSON, i will concate one-hot vector of PERSON tag (e.g [1,0,0,0]) to word embedding vector of “James” (e.g [4,5,6]). So result is [4,5,6,1,0,0,0]</p>

username: vdw
cooked: <p>Sure, you can concatenate vectors. For example, say you have</p>
<ul>
<li>
<code>embed</code> with <code>embed.shape = (batch_size, seq_len, embed_dim)</code>
</li>
<li>
<code>custom</code> with <code>custom.shape = (batch_size, seq_len, custom_dim)</code>
</li>
</ul>
<p>You can do:</p>
<p>X = torch.cat([embed, custom], 2)</p>
<p>Then <code>X.shape = (batch_size, seq_len, embed_dim+custom_dim)</code></p>

username: Shandilya21
cooked: <p>yes you can do that,<br>
just add extras embedding of suitable dimension,<br>
embedding = <strong>nn.Embedding(vocab_size, dim)</strong>,<br>
embedding.shape = (batch_size, seq_len, dims)<br>
extended_dim = (batch_size, seq_len, extended_dims)</p>
<p><strong>final = torch.cat([embedding, extended],2)]</strong><br>
the final is the vector with embeddings of name_entity as well. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Dimitrisl
cooked: <p>Hello, does this guarantee that your custom embeddings will train ?</p>

username: mbednarski
cooked: <p>If you will use standard pytorch blocks (contained in nn module) - they will. If you create tensors by yourself that should be updated by autograd you need to wrap them inside <code>nn.Parameter</code>. In any case you can check if parameter is actually updated by checking value of <code>your_custom_parameter.grad</code> after doing <code>loss.backward()</code>. If it will be non-zero that means it will be updated by the optimizer.</p>
Num_samples should be a positive integer value, but got num_samples=0

username: Ahmad_Riza
cooked: <p>Hello everybody, I am new to PyTorch. I have a problem when I tried to train my data. When I run my program there is an error like following:</p>
<p>Traceback (most recent call last):<br>
File “/home/riza/Documents/D E F E C T  D E T E C T I O N/code/GAN/WGAN.py”, line 140, in <br>
dataloader = torch.utils.data.DataLoader(dataset, batch_size=opt.batch_size, shuffle=True, drop_last=True)<br>
File “/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py”, line 176, in <strong>init</strong><br>
sampler = RandomSampler(dataset)<br>
File “/usr/local/lib/python3.5/dist-packages/torch/utils/data/sampler.py”, line 66, in <strong>init</strong><br>
“value, but got num_samples={}”.format(self.num_samples))<br>
ValueError: num_samples should be a positive integer value, but got num_samples=0</p>
<p>Please help me to solve the problem, Thank you</p>

username: ptrblck
cooked: <p>Could you post the definition of your <code>Dataset</code>?</p>

username: Ahmad_Riza
cooked: <blockquote>
<blockquote>
<blockquote>
<p>import os<br>
dirpath = os.getcwd()<br>
print("current directory is : " + dirpath)<br>
current directory is : /media/riza/EA62E27862E248C1/ubuntu_ppt/GAN_final/GAN/Defect/back/crack</p>
</blockquote>
</blockquote>
</blockquote>
<p>That is I checked from my terminal, I directly open terminal from my dataset folder</p>

username: ptrblck
cooked: <p>What kind of <code>Dataset</code> are you using? Is it <code>ImageFolder</code> or a custom implementation?</p>

username: Ahmad_Riza
cooked: <p>custom implementation dataset. It is my own dataset</p>

username: ptrblck
cooked: <p>Could you check, what <code>len(dataset)</code> returns, as it’s apparently returning a length of 0.</p>

username: Ahmad_Riza
cooked: <p>Sorry, I just fixed the problem. when I check len(dataset) the value is 0. Thank you for your help</p>

username: Zubair_Malik
cooked: <p>Ahmad how u fix it? Plzz help me</p>

username: Zubair_Malik
cooked: <p>Brother what should we return to solve this error…plzz help</p>

username: ptrblck
cooked: <p>Make sure the <code>__len__</code> method of your <code>Dataset</code> returns a valid number of samples (&gt;0).<br>
How are you passing (or loading) the data and how are you calculating the length of the <code>Dataset</code>?</p>

username: Ahmad_Riza
cooked: <p>Hello, sorry I just know your message, may I know your listing code?</p>

username: RY_dev
cooked: <p>Hello<br>
I am having the same error<br>
I am using an arabic dataset with utf-8 encoding<br>
Can anyone help me plz ?</p>
Will moving tensor from GPU to CPU to GPU break back-propagation?

username: qiminchen
cooked: <p>Hi,</p>
<p>I have to do <code>torch.inverse()</code> on tensor with a size of <code>4 x 240 x 320 x 3 x 3</code> every iteration during training. Since <code>torch.inverse()</code> supports batch inverse, I guess <code>3 x 3</code> is too small so <code>torch.inverse()</code> is pretty slow on GPU. So I moved the batch from GPU to CPU when doing <code>torch.inverse()</code> and move the output back to GPU for other operations. I’m wondering if <code>GPU-&gt;CPU-&gt;GPU</code> would break the back-propagation as I haven’t got any errors so far?</p>

username: albanD
cooked: <p>Hi,</p>
<p>No moving Tensors across devices is perfectly differentiable. It will all work fine!</p>
Requires_grad in pytorch

username: SAI_VARSHITTHA
cooked: <p><code>For the weights, we set  </code>requires_grad<code>  after the initialization, since we don’t want that step included in the gradient. (Note that a trailing </code>_<code>  in PyTorch signifies that the operation is performed in-place.)</code><br>
I found this in the following link : <a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html#using-torch-nn-functional" rel="nofollow noopener">Pytorch nn tutorial</a><br>
But why is <code>requires_grad</code> not set in the initialisation of weights, unlike biases?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think the reason is in the sentence you linked: " we don’t want that step (I guess the division in that example) included in the gradient".<br>
The initialization is just filling the original values and should not be considered when computing derivatives of the net.</p>
Fastest way of dataloading large datasets

username: Dipayan_Das
cooked: <p>Hello,</p>
<p>I am working a classification problem, which includes the feature generation at first and then classification. Due to my problem constrain, i generate the features first and then train classifier separately. However, dataloading has become the bottleneck of my pipeline and progress.</p>
<p>I save the generated datasamples as list of tensors, as .pt file having dimensions [(50,10,10,10), (1)], the last (1) being the associated label tensor.<br>
I use a standard dataset and dataloader code:</p>
<pre><code class="lang-auto">class LR_Dataset(Dataset):
    
    def __init__(self, filepath):
        self.filepath = filepath
        self.filenames = os.listdir(self.filepath)

    def __len__(self):
    	return len(self.filenames)

    def __getitem__(self, idx):
    	x,y = torch.load(os.path.join(self.filepath,self.filenames[idx]))
    	return x,y
        

def dataloader(filepath, batch_size, num_workers=0):
    dataset = LR_Dataset(filepath)
    return DataLoader(dataset, batch_size=batch_size, num_workers=num_workers)

</code></pre>
<p>Note that: the dataset folder contains ~150,000 .pt files. So i doubt if reading .pt file is taking the time, or is it because the .pt files contain tensors and not numpy arrays.</p>
<p>I am facing a tough deadline. Any help would be greatly appreciated.</p>
<p>Thank you</p>

username: Dipayan_Das
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Please help! <img src="https://discuss.pytorch.org/images/emoji/apple/pray/2.png?v=9" title=":pray:t2:" class="emoji" alt=":pray:t2:"></p>

username: suraj.pt
cooked: <p>I don’t think there is any overhead from using tensors instead of np arrays.</p>
<p>You could try increasing <code>num_workers &gt; 0</code> to use multiprocessing in <code>DataLoader</code></p>

username: ptrblck
cooked: <p>As <a class="mention" href="/u/dipayan_das">@Dipayan_Das</a> explained multiple workers might give you a speedup.<br>
If that’s not helping, you could try to preload the complete dataset, which should take approx. 27GB, if my calculation is right.</p>

username: Dipayan_Das
cooked: <p>Thanks a lot for considering, but I have a limited RAM of 16GB and 8GB GPU. Any other method other than loading whole data to RAM shall do. Additionally, i have indeed experimented with num_workers, however, the effect is not at all significant.</p>
<p>Thank you</p>
Pytorch C++ c10::Error

username: ChristophMoserGreil
cooked: <p>So I have been trying to rebuild the MNIST example from the Pytorch C++ Frontend tutorial.<br>
However, I get a c10::Error at the following line:</p>
<pre><code class="lang-auto">torch::Tensor d_loss_real = torch::binary_cross_entropy(real_output, real_labels);
</code></pre>
<p>real_output and real_labels are simply tensor, and neither of them is null.<br>
This is how the error code reads:</p>
<blockquote>
<p>Unhandled exception at 0x00007FF873E03B29 in MNIST_Example.exe: Microsoft C++ exception: c10::Error at memory location 0x0000000E2C35CED0.</p>
</blockquote>
<p>Weirdly enough, I cannot even step into the function using the debugger (Yes I enabled it, it works on other functions). Wrapping the whole thing in a try catch and printing exception.what() simply gets me</p>
<blockquote>
<p>Unknown Exception</p>
</blockquote>
<p>Some more infos:</p>
<ul>
<li>Windows 10 using Visual Studio</li>
<li>Llibtorch debug version without GPU Support (also I do not have a GPU if that is relevant)</li>
<li>.dll files are in the same folder as .exe</li>
<li>build system is CMake</li>
</ul>
<p>Any tipps would be greatly appreciated.</p>

username: ChristophMoserGreil
cooked: <p>If this helps, I provide the call stack as shown by Visual studio:</p>
<pre><code class="lang-auto"> 	KernelBase.dll!00007ffc0d563b29()	Unknown
 	vcruntime140d.dll!00007ffc018bb1f0()	Unknown
 	torch_cpu.dll!00007ffb9561c8fb()	Unknown
 	torch_cpu.dll!00007ffb955f1886()	Unknown
 	torch_cpu.dll!00007ffb955df815()	Unknown
 	torch_cpu.dll!00007ffb955de778()	Unknown
 	torch_cpu.dll!00007ffb95e8bb75()	Unknown
 	torch_cpu.dll!00007ffb95304f42()	Unknown
 	torch_cpu.dll!00007ffb95305067()	Unknown
 	torch_cpu.dll!00007ffb91bec6f2()	Unknown
 	torch_cpu.dll!00007ffb91beea6c()	Unknown
 	torch_cpu.dll!00007ffb91bec5c6()	Unknown
 	torch_cpu.dll!00007ffb91bf4b01()	Unknown
 	torch_cpu.dll!00007ffb91bae475()	Unknown
&gt;	MNIST_Example.exe!main() Line 165	C++
 	MNIST_Example.exe!invoke_main() Line 79	C++
 	MNIST_Example.exe!__scrt_common_main_seh() Line 288	C++
 	MNIST_Example.exe!__scrt_common_main() Line 331	C++
 	MNIST_Example.exe!mainCRTStartup() Line 17	C++
 	kernel32.dll!00007ffc0fa07bd4()	Unknown
 	ntdll.dll!00007ffc1042ce51()	Unknown

</code></pre>
<p>The line 165 is the one where binary_cross_entropy is called.</p>

username: ChristophMoserGreil
cooked: <p>I have added the .pdb in order to debug torch_cpu.dll and I found that it is looking for a File called “Functions.cpp”. I found that file on Github here:<br>
<a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/templates/Functions.cpp" rel="nofollow noopener">Link</a><br>
But I do not know why the file is not included in my libtorch files and whether it is smart to simply throw it in there manually?</p>
<p>Weirdly enough, according to the pytorch docs, the Functions.h header, where this function is declared, is not actually included by any File called Functions.cpp.</p>
Is there a dataset functionality similar to PyTorch Geometrics preprocess skipping?

username: Sascha
cooked: <p>Hi</p>
<p>so I’ve used <a href="https://pytorch-geometric.readthedocs.io/en/latest/index.html" rel="nofollow noopener">Pytorch-Geometric</a> for a while and have now returned to PyTorch.<br>
Something that I haven’t found in PyTorch, that i have grown very accustomed to in Pytorch-Geometric, is the possibility to skip preprocessing on Datasets, after you’ve done it once.<br>
And i was wondering, if there is a similar functionality here and I’m just missing it, or if i have to do everything myself.</p>
<p>But what am I talking about exactly?<br>
Well, there is a good example of it in their <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html" rel="nofollow noopener">documentation</a> and it is as follows.</p>
<pre><code class="lang-auto">import torch
from torch_geometric.data import InMemoryDataset


class MyOwnDataset(InMemoryDataset):
    def __init__(self, root, transform=None, pre_transform=None):
        super(MyOwnDataset, self).__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self):
        return ['some_file_1', 'some_file_2', ...]

    @property
    def processed_file_names(self):
        return ['data.pt']

    def download(self):
        # Download to `self.raw_dir`.

    def process(self):
        # Read data into huge `Data` list.
        data_list = [...]

        data, slices = self.collate(data_list)
        torch.save((data, slices), self.processed_paths[0])
</code></pre>
<p>So here, the interface provides two things. A directory for the raw data and a directory for the processed data.<br>
The dataset will execute the download function, if the raw-directory is empty.<br>
And when the process method has been executed, the data is being saved into the processed directory. So that, if the data has already been preprocessed, it will just be loaded from a file every time i reload the dataset.</p>
<p>Is there a functionality like that?</p>

username: suraj.pt
cooked: <p>Hi <a class="mention" href="/u/sascha">@Sascha</a>, I believe you can use <code>InMemoryDataset</code> with vanilla PyTorch.</p>
<p>Datasets in PyTorch require only 3 functions [ <code>__init__(), __len__(), __getitem__()]</code>. Since torch_geometric.Dataset contains those 3 functions, you should be able to pass its subclasses to a vanilla PyTorch DataLoader.</p>

username: Sascha
cooked: <p>I’ll try that, thank you.<br>
Although i hoped, that there would have been a way, without me having to install the Pytorch Geometric package into every environment, just because I like their dataset interface.</p>
Linking tensor.grad.data to nodes

username: Omroth
cooked: <p>Hi guys.  I have:</p>
<pre><code class="lang-auto">    result = model(tensor)

    score_max_index = result.argmax()
    score_max = result[score_max_index]

    score_max.backward()

    data = tensor.grad.data.abs()
</code></pre>
<p>Data is</p>
<pre><code class="lang-auto">tensor([0.4954, 0.6926, 0.2542, 1.0316, 0.3206, 0.0126, 0.5587, 0.0943, 2.8202,
        1.4996, 0.6181, 1.6888, 0.4801, 2.1043, 0.5233, 1.4717, 0.5883, 1.8086,
        1.2205, 0.5946, 1.1455, 0.4592, 1.6773, 1.2097, 1.0795, 1.1406, 0.8416,
        1.0722, 1.1076, 0.1615, 0.9181, 1.2613, 1.4448, 1.4703, 0.4576, 1.1672,
        0.5309, 3.9590, 0.5291, 7.3970, 0.4947, 0.4488, 0.8801, 0.1289, 0.3571,
        0.1130, 0.9194, 0.4625, 0.2476, 1.4286, 0.3808])
</code></pre>
<p>My network is:</p>
<pre><code class="lang-auto">self.linear1 = torch.nn.Linear(_input_dimension, _hidden_dimension)
        self.activation = torch.nn.Tanh()
        self.dropout = torch.nn.Dropout()
        self.linear2 = torch.nn.Linear(_hidden_dimension, _hidden_dimension2)
        self.activation2 = torch.nn.Tanh()
        self.linear3 = torch.nn.Linear(_hidden_dimension2, _hidden_dimension3)
        self.activation3 = torch.nn.Tanh()
        self.linear4 = torch.nn.Linear(_hidden_dimension3, _output_dimension)
</code></pre>
<p>How can I determine which member of tensor.grad.data “belongs” to which node or edge in that network?</p>
<p>Thanks,<br>
Ian</p>

username: tom
cooked: <p>It belongs to tensor (if present)?<br>
To get the model’s parameter’s grad, you could loop through <code>for name, p in model.named_parameters()</code> and use <code>p.grad</code> for each.<br>
The .data should probably be avoided.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Omroth
cooked: <p>So p.grad in that instance will be the gradient for the tensor’s run through the network?</p>

username: tom
cooked: <p>It is good to always say it as “the gradient of <em>function</em> with respect to <em>argument to that function</em>” to keep the mathematical nature of things happy, so in this case <code>p.grad</code> will be the gradient of the loss (from loss.backward() - actually the sum of all of them since the beginning or last time you used opt.zero_grad or so) with respect to p.</p>

username: Omroth
cooked: <p>Many thanks Tom, that is exceptionally clear and useful.</p>
<p>Ian</p>
ResNet18 parameterlist is empty

username: Kla
cooked: <p>Hi everyone, I’m trying to implement a resnet18 network for my dataset but I become an error: “optimizer got an empty parameter list”. May someone tell me what I have done wrong?</p>
<p>optimizer = optim.SGD(running_programm.parameters(), lr=0.01, momentum=0.9)</p>
<p>My network looks so:</p>
<p>class Resnet_Speaker_Recognition(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Resnet_Speaker_Recognition, self).<strong>init</strong>()</p>
<pre><code>    resnet_model = models.resnet18(pretrained=True)
    resnet_model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2,2), padding=(3, 3), bias=False)
    num_ftrs = resnet_model.fc.in_features
    resnet_model.fc = nn.Linear(num_ftrs, 1251)

def forward(self,x):
    x=self.resnet_model(x)
    return x</code></pre>

username: tom
cooked: <p>I think you’re missing a bunch of <code>self.</code> in the init.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: If you put triple backticks ``` before and after your code, you’ll get proper formatting.</p>

username: Kla
cooked: <p>Thank you very much, it really helped me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Question about building PyTorch 0.4.0 using CUDA 10

username: wentian_zhao
cooked: <p>In some situations, my colleagues need to run some old object detection code that only supports PyTorch 0.4.0. However, we only have RTX 2080 Ti, which won’t properly run if CUDA 8 is used. Can I compile PyTorch 0.4.0 from source code using CUDA 10?</p>

username: tom
cooked: <p>No worries, I’ll trade you my 1080 Ti against your 2080 Ti for a small fee. <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>More seriously, packages for PyTorch 0.4.2 with CUDA 9.2 can be found on the site, but CUDA 10 seems to be first supported by PyTorch 1.0. So you may have some luck compiling PyTorch 0.4.2 with CUDA 10 or you might look at grabbing fixes from the git repo whenever you hit a compile error (I don’t think there should be too many).</p>
Torch.save(model.state_dict(), PATH) [Errno 13] Permission denied?

username: inkplay
cooked: <p>Made a folder in my drive and when I tried to save the model I get permission denied? How do I fix this? Windows 10 Pytorch 0.40 and the folder was already made.</p>
<p>Edited: I works now -&gt; save.(model.state_dict(), ‘mode.pth’)</p>

username: Sanjay_Krishnamurthy
cooked: <p>Thanks it worked for me</p>

username: ved_prakash
cooked: <p>I was doing the mistake of providing path but not file name.<br>
Silly mistake …</p>
<p>PATH = “/home/folder1/folder2”<br>
torch.save(model.state_dict(), os.path.join(PATH,“filename.pth”))</p>

username: dynamiterdx
cooked: <p>Silly me. Same mistake lol.</p>
Training becomes slower as time goes by

username: Kenzd
cooked: <p>i am training my model(contains conv layers, fc layers ,bigru layers）on two gpus. at the beginning, every batch takes around 5s， but as time goes by , it takes 20s for every batch. and i don’t know why the training time for the same size data will increase.<br>
i use model=nn.DataParallel(model,device_ids=[0,1]) to train my model on two gpus.<br>
and i use torch.cuda.empty_cache() after every batch, but it doesn’t work out.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Kenzd" data-post="1" data-topic="97231">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/67e7ee/48.png" class="avatar"> Kenzd:</div>
<blockquote>
<p>torch.cuda.empty_cache() after every batch, but it doesn’t work out.</p>
</blockquote>
</aside>
<p>This would slow down your code as synchronizing calls are executed to free and reallocate the memory.</p>
<p>Are you seeing an increasing memory usage on the GPU(s)? If so, you might accidentally store the computation graph, which would increase the memory and runtime of your training, e.g. via <code>total_loss += loss</code>. If you store the loss or any other tensor, which is attached to the computation graph, you should <code>.detach()</code> it before.</p>
PyTorch, threading, multiple GPUs

username: MChaus
cooked: <p>Hello!<br>
I have very intense task with matrices. I want to pass a tensor to GPU in a separate thread and get the result of performed operations.<br>
I created a class - <code>Worker</code> with interface <code>compute</code> that do all the work and returns the result. Now, I want to pass 4 class instances along with tensors to separate threads for computing on all my 4 GPUs.<br>
The code:</p>
<pre><code class="lang-auto">workers = [
    Worker(64, device= torch.device('cuda:0')),
    Worker(64, device= torch.device('cuda:1')),
    Worker(64, device= torch.device('cuda:2')),
    Worker(64, device= torch.device('cuda:3'))
    ]
matrices = [tensor1, tensor2, tensor3, tensor4]
output = []
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
    for worker, matr in zip(workers, matrices):
        result = executor.submit(worker.compute, matr)
        output.append(result)
</code></pre>
<p>But <code>output[0].result()</code> throws the following error .</p>
<blockquote>
<p>“CUDA error: an illegal memory access was encountered”</p>
</blockquote>
<p>I think the code inside the class is fine, because everything works on every GPU without threads.<br>
I am new to PyTorch, help me please.</p>
<p>EDIT:<br>
Today, in the morning I found out that the problem may cause custom CUDA kernel I use. Without it everything works. Any way, I appreciate any suggestions and good practices on using threading with PyTorch, because I am not sure my code exploits a good way.</p>

username: JuanFMontesinos
cooked: <p>Why do you need to use multithreading?<br>
You can do everything in the main thread, whether you want to run multiple instances within the same GPU or to run one instance per GPU.</p>

username: MChaus
cooked: <p>I think in this case tasks will perform consequentially not simultaneously.<br>
The thing I want is simultaneous computation on all of my GPUs.</p>
GPU Memory in Eval vs Training

username: NicoAdamo
cooked: <p>Hi there, I’m new to Pytorch and struggling to understand GPU memory management. I have a model which, during training, takes up slightly more memory than my GPU can handle - so I’ve gone ahead and trained it on an AWS server with more virtual memory.</p>
<p>I’m wondering whether I will need the same amount of memory to evaluate the model on the GPU, or whether I could possibly evaluate without running into issues. I know in both cases you have to load the whole model into the GPU, but during evaluation you don’t need to calculate gradients so I’d guess it takes less memory - but by how much, I’m not sure. What do you think?</p>

username: ptrblck
cooked: <p>Your assumption is correct. During evaluation you don’t need to calculate the gradients, which would take memory, and also don’t need to store intermediate activations, which would be needed to calculate the gradients.</p>
<p>To lower the memory usage and not store these intermediates, you should wrap your evaluation code into a <code>with torch.no_grad()</code> block as seen here:</p>
<pre><code class="lang-python">model = MyModel().to('cuda')
with torch.no_grad():
    output = model(data)
</code></pre>
- Illegal memory access was encountered - while no error when using cpu

username: dvirginz
cooked: <p>GPU 2080TI<br>
Driver 430.64<br>
cuda 10.1<br>
pytorch  1.6.0</p>
<p>The following error occurs on the following lines of code, and I could not recreate it artificially</p>
<pre><code class="lang-auto">    l = torch.from_numpy(model.train_dataset.template.Laplacian)
    a = torch.from_numpy(model.train_dataset.template.A)
    evec = template_evecs[0][:,1]
    evals = torch.from_numpy(model.train_dataset.template.evals)
    eval1 = evals[1]
    res = l.matmul(evec) - eval1 * a.matmul(evec)
</code></pre>
<p><code>THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=700 : an illegal memory access was encountered</code></p>
<p>Unfortunately, If I transfer the tensors to <code>.cpu()</code> first, the answer is correct and no errors.<br>
Ideas?</p>

username: Dexter
cooked: <p>My guess is, this happens because problem happens in memory allocation in GPU for complete data insertion. Have you tried it doing in batches and checking whether the same happens for batches?</p>

username: dvirginz
cooked: <aside class="quote no-group" data-username="Dexter" data-post="2" data-topic="96908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dexter/48/28953_2.png" class="avatar"> Dexter:</div>
<blockquote>
<p>er the same</p>
</blockquote>
</aside>
<p>I think I don’t understand you:/ it’s not a complete data insertion rather loading precomputed numpy arrays and moving them to the gpu</p>

username: ptrblck
cooked: <p>Could you rerun your code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the stack trace as well as the line of code, which creates this issue?</p>

username: dvirginz
cooked: <p>I’m sorry ptrblck, but I cannot repreduce anymore, super strange.<br>
Thanks for helping:)</p>
My object detection model is overfitting , how can i solve this problem?

username: malatyali
cooked: <p>i am using retinanet. when i draw loss graph for training and validation loss, after 5 epoch , validation loss is increasing. (i used augmentation and L2)  What do you think about that ?</p>
<p>Thanks,</p>

username: ptrblck
cooked: <p>To counter overfitting you could try to collect more data, increase regularization, lower the model capacity etc.<br>
There is no single answer to this problem, so you might need to try out different approaches.</p>

username: KaiHoo
cooked: <p>Maybe this paper can help: <a href="https://arxiv.org/abs/1902.04103" rel="nofollow noopener">https://arxiv.org/abs/1902.04103</a></p>

username: malatyali
cooked: <p>thanks, i will try . Is there any pipeline for ?</p>

username: malatyali
cooked: <p>thanks, i will read this paper.</p>
Is Conv1D fully supported now?

username: antimora
cooked: <p>Back in Oct 2019 Glow did not support ONNX models with Conv1D . A workaround was to use Conv2D instead. Now I see a recent merge for Conv1D (<a href="https://github.com/pytorch/glow/pull/4317" rel="nofollow noopener">https://github.com/pytorch/glow/pull/4317</a>). Does it mean there is a Conv1D support?</p>

username: jfix
cooked: <p>Yes, it should work!</p>

username: antimora
cooked: <p><a class="mention" href="/u/jfix">@jfix</a>, I can confirm the Conv1D in ONNX is supported. I have a follow up question, however.</p>
<p>Since Conv1D uses Conv2D by transposing/reshaping tensor, can you please confirm transposing and reshaping operations are cheap? I see lots of extra/new reshape and transpose nodes in graph.dot, whereas ONNX does not contain any.</p>
Optimization guide using model-compiler

username: antimora
cooked: <p>I am exploring Glow to optimize my model that has a bunch of conv + batch normalization + relu. And unfortunately I do not see a good guide for this. I looked through available options ( <code>./build/bin/model-compiler -help-list</code> ) but I am lost which I should be applying.</p>
<p>Here is a partial screenshot of the model:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/9436eeec97a955b1062436d35f0fb8a1cd20a75a.png" data-download-href="https://discuss.pytorch.org/uploads/default/9436eeec97a955b1062436d35f0fb8a1cd20a75a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/9436eeec97a955b1062436d35f0fb8a1cd20a75a_2_319x500.png" alt="image" data-base62-sha1="l9amiAkTGS1ldCp4nANXGIHtPOG" width="319" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/9436eeec97a955b1062436d35f0fb8a1cd20a75a_2_319x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/9436eeec97a955b1062436d35f0fb8a1cd20a75a_2_478x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/9436eeec97a955b1062436d35f0fb8a1cd20a75a_2_638x1000.png 2x" data-dominant-color="F1F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">666×1042 60.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does model-compiler automatically apply optimizations like fusing nodes? Or one needs to apply manually?</p>
<p>I am using PyTorch 1.4 and exporting my model in ONNX format.</p>

username: jfix
cooked: <p>Optimizations are done automatically. Some are dependent on the backend you’re running on. In the model snippet you have here I’d expect the BatchNorms to be optimized away (fused into Conv’s weights). You should be able to pass <code>-dump-graph-DAG=graph.dot</code> to see the Glow graph after optimizations are done. Some things such as Relu fusing may be done at lower levels of IR which are a bit harder to see (e.g. might be done in LLVM IR for the CPU backend or other LLVM-based backends).</p>

username: tkg.rahul
cooked: <p><a class="mention" href="/u/antimora">@antimora</a> I am also facing problem in finding good guide/tutorial? Have you got any reference or share your own experience?</p>

username: antimora
cooked: <p>Nothing beyond what Jordan has suggested. I can confirm via graph.dot some of the nodes, such as batch norm, are fused.</p>
Batch learning 4D Data

username: anwu21
cooked: <p>I have 4D data (N, 3, 64, 64) – basically N 64x64 RGB images where N is the set of rotations of the images – and I would like to use DataLoader.  So, the tensor shape I desire is (batch_size, N, 3, 64, 64).  However, it appears I can only feed DataLoader a tensor shape (batch_size, A, B, C).  So I could only get DataLoader to work by setting batch_size = 1, and let N be viewed as the size of the batch.  This approach does work, but it takes a very long time to train.  It would be nice to run multiple batches of (Nx3x64x64) simultaneously using DataLoader.  Is there a way to do this?  If so, could you please provide an example of how to implement it?</p>

username: Samarth_Mishra
cooked: <p>I do not think there is a restriction for <code>torch.utils.data.DataLoader</code>. Maybe your network cannot use 5-D data when it expect 4-D?</p>
<p>Personally, when dealing with sets of different rotations, I have been able to “flatten” (using <code>reshape</code>) those into individual images in a batch so my final mini-batch would have the shape <code>(batch_size x N, 3, 64, 64)</code>. Does this help?</p>

username: anwu21
cooked: <p>When you flatten using reshape, is it correct to assume that the N samples are grouped together during training?  That is, samples from one set of N don’t get mixed with samples from another set of N.  How does one ascertain this?  I suppose I may need to set shuffle=False during training to keep the N samples individually grouped after flattening, but that seems odd to do during training as shuffle is usually True during training and False during test.</p>

username: Samarth_Mishra
cooked: <p>This reshaping is done on the minibatch (5-D) before you input it to the network. <code>shuffle</code> can still be <code>True</code>.<br>
I’m not 100% sure this is what you were asking but just by how reshape works, the first N elements among the N x batch_size would be the different rotations of the first image.</p>

username: anwu21
cooked: <p>It is good to know that the order is kept.</p>
<p>Originally, I would have N x 1 tensor as an output of the network, and I would take a softmax over this tensor.  Now, with an input of (batch_size x N, 3, 64, 64), I now have a tensor of size batch_size*N x 1 and must take a softmax over every N elements (batch_size times).  I can only think of accomplishing this using a loop.  Is there a better way to do this, since looping this way is slow?  Even with looping is faster because of loading more data initially (batch_size x N samples) instead of just N samples.  But with typical batch training, no looping is required at all with DataLoader.</p>

username: Samarth_Mishra
cooked: <p>You can reshape the output back to (batch_size, N) and use a softmax over <code>dim=1</code></p>

username: anwu21
cooked: <p>Thanks!  That works great.</p>
Is there any way to train a shared neural network in one step?

username: chatterboy
cooked: <p>Hello, I’m trying to implement a model that is used multiple times in one step of each epoch. I think the code is like below. How to train such model in proper way? Thank you.</p>
<pre><code class="lang-python">encoder = Encoder()
optimizer = torch.optim.SGD(encoder.parameters())
criterion = torch.nn.MSELoss()
for epoch in range(num_epoch):
    for images, labels in train_loader:
        # in this line, images.shape [b, c, h, w]
        something processing ...
        # in this line, images.shape [n, b, c, h, w]
        outs = []
        for _ in images[:]:
            outs = encoder(_)
        something processing ...
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>

username: chatterboy
cooked: <p>I changed some part of the above code, in which the encoder take a whole tensor as input like below. In this case, are there any differences from above code in learning process?</p>
<pre><code class="lang-python">class Encoder(nn.Module):
    def __init__(self):
        # ... layers to use

    def _forward(self, x):
        # x.shape: [b, c, h, w]
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        x = F.relu(self.bn4(self.conv4(x)))
        return x

    def forward(self, x):
        # x.shape: [n, b, c, h, w]
        r = []
        for _ in x[:]:
            r.append(self._forward(_))
        return torch.stack(r)

...

for epoch in range(num_epochs):
    for images, labels in train_loader:
        # ...
        outs = encoder(images)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>

username: ptrblck
cooked: <p>I guess there shouldn’t be a difference, but since your codes are incomplete I would recommend to run a quick test using a constant input and check the output as well as the gradients for an iteration.</p>
<p>I’m not familiar with your use case, but would it be possible to pass the complete input directly to the model instead of using a loop?</p>

username: chatterboy
cooked: <p>Thank you for reply! Yes, I knew that it is possible to pass entire input to the model by reshaping the dimensions of the input and try to test on your recommendation. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Why does each GPU occupy different memory using DDP?

username: Shanyan_Guan
cooked: <p>I found a strange thing. When I use two GPUs, the memory occupied in the two GPUs is the same. But when I use 4 or 6 GPUs, the memory occupied in GPUs is not exactly the same.</p>
<p>Has anyone ever been in this situation?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c12788a40dabf26eacdf5cf8e75614f5a598aefa.png" data-download-href="https://discuss.pytorch.org/uploads/default/c12788a40dabf26eacdf5cf8e75614f5a598aefa" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c12788a40dabf26eacdf5cf8e75614f5a598aefa.png" alt="image" data-base62-sha1="ryIQQU2mTPgdtOzP3LLk6y6dbaG" width="690" height="399" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">859×497 47 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>A device might run a bit ahead of the others and could thus create another memory footprint.<br>
Also, if you didn’t set e.g. cudnn to the deterministic mode, the kernel selection might slightly vary between the devices.</p>

username: Shanyan_Guan
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, Thanks for your quick reply! I have set cudnn to the deterministic mode. Maybe this is due to the different speeds between each GPU. Will this case affect the training process?</p>

username: ptrblck
cooked: <p>DDP synchronizes the devices if necessary and communicates the gradients etc. between the GPUs.<br>
If one device is a ms faster, it would have to wait, but besides that you shouldn’t see any effects.</p>
How to avoid copying data when using multiprocessing and CUDA?

username: lkc
cooked: <p>I want to train multiple independent models on GPU using multiprocessing. The data is the same for all the processes. A <code>fork</code> method uses Copy-On-Write where if the data is read-only (as is mine), the child processes will not make a copy of the data. From the <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing" rel="nofollow noopener">multiprocessing documentation</a>, I see that CUDA only supports <code>spawn</code> or <code>forkserver</code> methods. However, with these methods, the child process makes separate copies of the data and the memory increases linearly with the number of processes.</p>
<p>How I can have common data used by all the child workers while training on GPU? There is no communication between the processes and the data is read-only. If I don’t use GPU, the <code>fork</code> method works fine, but obviously I’m interested in making it work on the GPU.</p>
<p>I haven’t used <a href="https://pytorch.org/docs/stable/notes/cuda.html#use-nn-parallel-distributeddataparallel-instead-of-multiprocessing-or-nn-dataparallel" rel="nofollow noopener"><code>nn.parallel.DistributedDataParallel</code></a> but I believe it is more suitable for training a single model and distributing the batches across different devices. My use case is different where I am training separate models on single/multiple GPUs and the training data need not be distributed. Is it possible use <code>nn.parallel.DistributedDataParallel</code> here?</p>
Broadcasting Index

username: taha
cooked: <p>I am trying to use a reference matrix to slice another matrix with more dimensions:</p>
<pre><code class="lang-python"># The tensor that we use for indexing
reference = torch.rand(bsize, nfeat)
index = torch.argsort(reference)[:, :bottomk]
# The target tensor that we want to slice
target = torch.rand(bsize, nsample, nfeat)
</code></pre>
<p>How can I perform slicing using <code>index</code>? I want to automatically broadcast the second dimension. Can you suggest me how to do it?</p>
<pre><code class="lang-python"># The for-loop version. But I want a more elegant and faster version 
# that does not loop over batch
for i in range(bsize):
    target[i, :, index[i, :]] = 0
</code></pre>

username: Naruto-Sasuke
cooked: <p>Why target is size of <code>bsize * nsample * nfeat</code>. How do you want to slice when your reference is size of <code>bsize * nfeat</code>. Please give an a example  in detail. Probably I can help with it.</p>

username: taha
cooked: <p>I edited the question to clarify your question.</p>

username: googlebot
cooked: <p>can’t say that gather() is fast, but following should be a loop-less equivalent for reading</p>
<pre><code class="lang-auto">target.gather(-1,index.unsqueeze(1).expand(-1,nsamples,-1))
</code></pre>
<p>for writing, look at Tensor.scatter_ or Tensor.masked_fill_, expanding index/mask similarly</p>