akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Add redundancy to neural network

vainaijr: <p>How do I make my neural network learn same pattern multiple times, that is one neural network learns minor modifications of the same pattern, and stores them in a list, and returns this list to another neural network?</p>
<p>if I do something like this,</p>
<pre><code class="lang-auto">class Model(nn.Module):
  def __init__(self):
    self.lin = nn.Linear(3, 3)
  def forward(self, x):
    out = self.lin(x)
    return out, out
</code></pre>
<p>then it would return two outputs, but both of them would be same, I want redundancy, but not exactly same patterns, only minor modifications.</p>

paganpasta: <p>I’m not sure why you are trying to do, because there might be a better way to achieve the desired output. However, one way to achieve ‘minor’ modifications is by adding some random noise(possibly Gaussian) to X in the forward() method call.</p>
Gradient of accumulative loss is not equal to sum of gradients of single losses!

alibsa: <p>Why accumulating gradients show us different value than summing them manually? Is it because of round-off errors?</p>
<p>Code:</p>
<pre><code class="lang-auto">class LogisticRegression(torch.nn.Module):
	def __init__(self, input_dim, output_dim):
		super(LogisticRegression, self).__init__()
		self.linear = torch.nn.Linear(input_dim, output_dim)

	def forward(self, x):
		outputs = self.linear(x)
		return outputs

batch_size = 128

seed = 0
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

train_dataset = dsets.MNIST(root='./dataMNIST', train=True, transform=transforms.ToTensor(), download=True)
test_dataset = dsets.MNIST(root='./dataMNIST', train=False, transform=transforms.ToTensor(), download=True)
trainloader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)
testloader = torch.utils.data.DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)

device = torch.device('cuda:0')

input_dim = 784
output_dim = 10

net = LogisticRegression(input_dim, output_dim)
net = net.to(device)
criterion = nn.CrossEntropyLoss(reduction = 'sum')

for data, target in trainloader:
	data = Variable(data.view(-1, 28 * 28))
	target = Variable(target)
	data, target = data.to(device), target.to(device)
	output = net(data)
	cost = criterion(output, target)
	cost.backward()

total_grad = 0.0
for p in net.parameters():
	total_grad += p.grad.data
	break

net.zero_grad()

for data, target in trainloader:
	data = Variable(data.view(-1, 28 * 28))
	target = Variable(target)
	data, target = data.to(device), target.to(device)
	output = net(data)
	cost = criterion(output, target)
	cost.backward()
	for p in net.parameters():
		total_grad -= p.grad.data
		break
	net.zero_grad()

print(total_grad.norm(2).item())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">0.029222624376416206
</code></pre>
<p>Thank you!</p>

albanD: <p>Hi,</p>
<p>This is most likely due to precision errors of floats.<br>
Do you see the same behavior if you run with double by setting <code>torch.set_default_tensor_type(torch.DoubleTensor)</code>.</p>

alibsa: <p>Thanks, I tried it before, and the output was: 5.152470202661044e-11<br>
I was thinking may be there is another problem.</p>
<p>I think you are right, and it is because of the floating number precision.</p>
The tensor from pytorch c++ can print() but can't clone()?

Chunsheng_Guo: <p>a tensor created in pytorch C++ and return to python pytorch. It  use clone() to display error: segmentation fault (core dumped)</p>

albanD: <p>Hi,</p>
<p>This is most likely because there is some problem the way the Tensor was returned. Could you provide a small code sample that reproduce the issue?</p>
Converting class tensor from segmentation to image

Glock: <p>I have a softmax output of a segmentation network which after argmax is of size (batch_size, Width, Height) and consists of (0,1,2,3) which are classes assigned to each pixel in image. Now I’m trying to convert this tensor into RGB image where each class is assigned diffrent color. I came up with this code</p>
<pre><code class="lang-auto">def convert_tensor_to_RGB(network_output):
    converted_tensor = torch.zeros([network_output.size(0), 3, network_output.size(1), network_output.size(2)])
    x = {
        0: torch.Tensor([0, 0, 0]),
        1: torch.Tensor([255, 0, 0]),
        2: torch.Tensor([0, 0, 255]),
        3: torch.Tensor([0, 255, 0]),
    }
    for i in range(network_output.size(0)):

        sample = network_output[i, :, :]
        for j in range(len(x)):
            converted_tensor[i, sample == j] = x[j]
    return converted_tensor
</code></pre>
<p>but I get an error: IndexError: The shape of the mask [540, 960] at index 0 does not match the shape of the indexed tensor [3, 540, 960] at index 0<br>
So is there any efficient and fast way to do this convertion?</p>

phan_phan: <p>On the line<br>
<code>converted_tensor[i, sample == j] = x[j]</code><br>
you just forgot to index on the <em>channel</em> dimension:<br>
<code>converted_tensor[i, :, sample == j] = x[j]</code><br>
but then <code>converted_tensor[i, :, mask]</code> would be shaped <code>[3, ?]</code> whereas <code>x[j]</code> is shaped <code>[3]</code>, so you would probably need to unsqueeze it:</p>
<pre><code class="lang-auto">converted_tensor[i, :, sample == j] = x[j].unsqueeze(1)
</code></pre>
<p>EDIT:<br>
A much better way to do this would be to use <a href="https://pytorch.org/docs/stable/nn.functional.html#embedding" rel="nofollow noopener"><code>torch.nn.functional.embedding()</code></a>.<br>
Put your color table in a <code>[4, 3]</code> float tensor:</p>
<pre><code class="lang-auto">x = torch.FloatTensor([[0, 0, 0], [255, 0, 0], [0, 0, 255], [0, 255, 0]])
</code></pre>
<p>Then you can directly have your result with:</p>
<pre><code class="lang-auto">converted_tensor = torch.nn.functional.embedding(network_output, x).permute(0, 3, 1, 2)
</code></pre>
<p>No need for a loop.</p>

Glock: <p>Thank you, it works great.</p>
Use linux based pytorch model in windows c++ code

Fariha1123: <p>I am looking forward to the best way to use my linux based pytorch model to the c++ program in windows.<br>
Can someone please guide me to the best options?</p>

tom: <p>You can follow the <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">export to C++ tutorial</a>, the export format is the same on Windows and Linux, so you can make the step Python on Linux to C++ on Windows.</p>
<p>Best regards</p>
<p>Thomas</p>

Fariha1123: <p>Thanks, I tried following the tutorial as it is but i am facing issues in building the application via cmake.</p>
<p>This is my error report:</p>
<p>D:\work\torchscript\example-app\example-app\build&gt;cmake -DCMAKE_PREFIX_PATH=D:/work/torchscript/libtorch …<br>
– Caffe2: CUDA detected: 10.0<br>
– Caffe2: CUDA nvcc is: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/bin/nvcc.exe<br>
– Caffe2: CUDA toolkit directory: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0<br>
CMake Error at D:/work/torchscript/libtorch/share/cmake/Caffe2/public/cuda.cmake:74 (message):<br>
Caffe2: Couldn’t determine version from header: Change Dir:<br>
D:/work/torchscript/example-app/example-app/build/dCYT4/CMakeFiles/CMakeTmp</p>
<p>Run Build Command(s):C:/Program Files (x86)/Microsoft Visual<br>
Studio/2017/Professional/MSBuild/15.0/Bin/MSBuild.exe cmTC_c9077.vcxproj<br>
/p:Configuration=Debug /p:Platform=x86 /p:VisualStudioVersion=15.0 /v:m &amp;&amp;<br>
Microsoft ® Build Engine version 15.9.21+g9802d43bc3 for .NET Framework<br>
Copyright © Microsoft Corporation.  All rights reserved.<br>
C:\Program Files (x86)\Microsoft Visual<br>
Studio\2017\Professional\Common7\IDE\VC\VCTargets\Microsoft.Cpp.Platform.targets(65,5):<br>
error MSB8020: The build tools for Visual Studio 2010 (Platform Toolset =<br>
‘v100’) cannot be found.  To build using the v100 build tools, please<br>
install Visual Studio 2010 build tools.  Alternatively, you may upgrade to<br>
the current Visual Studio tools by selecting the Project menu or<br>
right-click the solution, and then selecting “Retarget solution”.<br>
[D:\work\torchscript\example-app\example-app\build\dCYT4\CMakeFiles\CMakeTmp\cmTC_c9077.vcxproj]</p>
<p>Call Stack (most recent call first):<br>
D:/work/torchscript/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)<br>
D:/work/torchscript/libtorch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)<br>
CMakeLists.txt:4 (find_package)</p>
<p>– Configuring incomplete, errors occurred!<br>
See also “D:/work/torchscript/example-app/example-app/build/CMakeFiles/CMakeOutput.log”.<br>
See also “D:/work/torchscript/example-app/example-app/build/CMakeFiles/CMakeError.log”.</p>

tom: <p>So here I must admit that I know next to zero about building things for windows, but if you have the 2010 build tools installed (have you?), you might need to include them into the PATH.<br>
To me, this seems to be something around how to build general things on Windows more than PyTorch specific.</p>
<p>Best regards</p>
<p>Thomas</p>
Torchscript custom ops with CUDA

soldierofhell: <p>Hi,<br>
I want to replace custom c++/cuda ops with custom torchscript c++/cuda ops to be able to export model from python to c++. Currently the <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html" rel="noopener nofollow ugc"> EXTENDING TORCHSCRIPT WITH CUSTOM C++ OPERATORS</a> tutorial only handle C++ usecase, but in the end states:</p>
<blockquote>
<p>You are now ready to extend your TorchScript models with C++ operators that interface with third party C++ libraries, <strong>write custom high performance CUDA kernels</strong>, or implement any other use case that requires the lines between Python, TorchScript and C++ to blend smoothly.</p>
</blockquote>
<p>On the other hand in <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-mixed-c-cuda-extension" rel="noopener nofollow ugc">Writing a Mixed C++/CUDA extension</a> tutorial it states that there’s some magic goin’on <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>The general strategy for writing a CUDA extension is to first write a C++ file which defines the functions that will be called from Python, and binds those functions to Python with pybind11. <strong>Furthermore, this file will also  <em>declare</em>  functions that are defined in CUDA ( <code>.cu</code> ) files. The C++ functions will then do some checks and ultimately forward its calls to the CUDA functions.</strong> In the CUDA files, we write our actual CUDA kernels. <strong>The  <code>cpp_extension</code>  package will then take care of compiling the C++ sources with a C++ compiler like  <code>gcc</code>  and the CUDA sources with NVIDIA’s  <code>nvcc</code>  compiler.</strong> This ensures that each compiler takes care of files it knows best to compile. Ultimately, they will be linked into one shared library that is available to us from Python code.</p>
</blockquote>
<p>Question is how to do it for torchscript op?</p>
<p>Regards,</p>

tom: <p>You can do the very same - write your kernel in .cu. Put the op registration in the .cpp.<br>
I usually recommend to use load/load_inline with <code>is_python_module=False</code> as the first step to move from C++ extensions to Custom Ops.</p>
<p>Best regards</p>
<p>Thomas</p>

soldierofhell: <p>Thank you Tom. I was not sure who’s gonna trigger compilation of .cu source.<br>
BTW torchvision recently introduced torch.ops so I have to analyze it</p>
<p>Regards,</p>
Something wrong when i tried to use two gpus in a node

zyyupup: <p>I use the  command line below to run the script(<a href="https://github.com/NVIDIA/DALI/blob/master/docs/examples/pytorch/resnet50/main.py" rel="nofollow noopener">pytorch example of dali</a>)</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=1 train_imagenet_with_dali.py -t
</code></pre>
<p>when the nproc_per_node=1,it work. But when nproc_per_node=2,there is a error.</p>
<pre><code class="lang-auto">*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
=&gt; creating model 'resnet50'
=&gt; creating model 'resnet50'
Traceback (most recent call last):
  File "/home/zyy/anaconda3/envs/python36/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/zyy/anaconda3/envs/python36/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/zyy/anaconda3/envs/python36/lib/python3.6/site-packages/torch/distributed/launch.py", line 246, in &lt;module&gt;
    main()
  File "/home/zyy/anaconda3/envs/python36/lib/python3.6/site-packages/torch/distributed/launch.py", line 242, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/zyy/anaconda3/envs/python36/bin/python', '-u', 'train_imagenet_with_dali.py', '--local_rank=1', '-t']' died with &lt;Signals.SIGSEGV: 11&gt;.
</code></pre>

spanev: <p>Hi <a class="mention" href="/u/zyyupup">@zyyupup</a>,</p>
<p>Can you try this please:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=2 main.py -a resnet50 --dali_cpu --fp16 --b 32 --static-loss-scale 128.0 --workers 4 --lr=0.4 ./ 2&gt;&amp;1
</code></pre>
<p>And give us full repro step and your environment (Anaconda, PyTorch, DALI versions)</p>

zyyupup: <p>Thank you for your reply. I tried the commamd  and the same error accured again. My environment is:</p>
<ul>
<li>Anaconda(python):3.6.8</li>
<li>Pytorch:1.2.0+cuda9.2</li>
<li>DALI:0.13.0</li>
<li>System:ubuntu16.04 with 2 gpus</li>
</ul>
<p>I only changed the path of the dataset in the code and then run it.</p>

JanuszL: <p>DALI example is based on an older version of PyTorch APEX example - <a href="https://github.com/NVIDIA/apex/tree/master/examples/imagenet" rel="nofollow noopener">https://github.com/NVIDIA/apex/tree/master/examples/imagenet</a>. You can try to run it as well to check if this may be the DALI fault.</p>
Help regarding neural network training

busra: <p>Yapay sinir ağı egitimi için data setim var ama bunu nasıl aktaracağımı bilmiyorum excelden. Yardımcı olabilir misiniz?</p>

ptrblck: <p>Hi Busra,</p>
<p>I’ll add the translation of your text here, so that we can help better:</p>
<blockquote>
<p>I have a data set for artificial neural network training, but I don’t know how to transfer it. Can you help me?</p>
</blockquote>
<p>If you want to fine tune a model, have a look at <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">this tutorial</a>. <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html">This tutorial</a> might also be a good starter.</p>
<p>PS: feel free to use some translation service (e.g. Google translate) to post your question. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

busra: <p>I will investigate.Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Splitting Tensor for different loss functions

Krish: <p>I have a tensor <code>x</code> of shape <code>[batch_size, 2]</code>. I want to split it in shapes <code>[batch_size, 1]</code> and <code>[batch_size, 1]</code> then use one as a classification head and one as a regression head.</p>
<p>Is there a way to do this or will I have to use different tensors?</p>

phan_phan: <p>Use the tensors <code>x[:, 0]</code> and <code>x[:, 1]</code>.</p>

Krish: <p>I did that but I am asking that if I pass these to BCE loss and MSE loss respectively, will the grad be stored correctly so that optimisation can happen properly?</p>

phan_phan: <p>Yes ! Say you have <code>loss_1</code> and <code>loss_2</code> (computed for example with <code>x[:, 0]</code> and <code>x[:, 1]</code>).<br>
Then if you call <code>.backward()</code> on a combination of the two losses, for example</p>
<pre><code class="lang-auto">loss_sum = 0.5 * loss_1 + 0.5 * loss_2
loss_sum.backward()
</code></pre>
<p>optimization can happen properly.</p>
Are models on Pytorch Hub screened for errors?

SpandanMadan: <p>I’m planning on using pytorch hub segmentation models for my research project. The models look fine overall, but I’m wondering if they have been screened for possible errors? I know that they are pooled in by other people on github, but is the pytorch team taking the time to screen them? Or is it an automated process with no checks?</p>
<p>Essentially, Do models on pytorch hub have high credibility, like tensorflow hub?</p>
<p>Thanks a lot for the initiative!</p>

ptrblck: <p><a class="mention" href="/u/ailzhang">@ailzhang</a> might know more about the process in PyTorch Hub.</p>

smth: <p>Yes, absolutely.<br>
We carefully vet and review them to the best extent we can.<br>
If they are on <a href="http://pytorch.org/hub" rel="nofollow noopener">pytorch.org/hub</a> they are screened.</p>

SpandanMadan: <p>Amazing! Thanks <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/ptrblck">@ptrblck</a>, the research community thanks you for making our job easier through Pytorch in so many ways and continually improving it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Batch size larger than dataset size

HaziqRazali: <p>How can i configure the dataloader to accept a batch size that is larger than the dataset size? Is it possible for the dataloader to continue sampling from the dataset?</p>
<pre><code class="lang-auto">uti_va_loader = torch.utils.data.DataLoader(uti_va_data, 
                                            batch_size=args.batch_size, 
                                            shuffle=True, 
                                            num_workers=0, 
                                            drop_last=False,
                                            pin_memory=torch.cuda.is_available())
</code></pre>
<p>This line is completely skipped if the batch size is larger than the dataset size.</p>
<pre><code class="lang-auto">for batch_idx, data in enumerate(uti_va_loader): 
    print(data.size())
</code></pre>

JuanFMontesinos: <p>It’s not about the dataloader but dataset. Dataset is data-agnostic and it just iterates over a list of indices whose length is set at <strong>len</strong> dataset method.</p>
<p>How do you expect the dataloader to accept larget batch size if it cannot load non existing data? It will throw errors.</p>
<p>Anyway if you artificially enlarge the number that dataset.<strong>len</strong> returns you will be able to.</p>
<p>If what you wanna do is create a kind of infinite loop you can use built-in itertools’ repeat which allows you to iterate a iterator as many times as you want.<br>
<a href="https://docs.python.org/2/library/itertools.html#itertools.repeat" class="onebox" target="_blank" rel="nofollow noopener">https://docs.python.org/2/library/itertools.html#itertools.repeat</a></p>

HaziqRazali: <blockquote>
<p>Anyway if you artificially enlarge the number that dataset. <strong>len</strong>  returns you will be able to.</p>
</blockquote>
<p>Thank you. That give me an idea to simply take the modulo of <code>dataset.len</code>, allowing me to set a batch size larger then the size of the dataset. I still needed to set <code>__len__</code> to return a larger number, either the length of the dataframe or the batch size.</p>
<ol>
<li>Set the length of the dataset to be the max over the dataset length or the batch size</li>
</ol>
<pre><code class="lang-auto">def __len__(self):
    return max(len(self.df),args.batch_size)
</code></pre>
<ol start="2">
<li>Take the modulo idx by the <strong>actual</strong> length of the data</li>
</ol>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        idx = idx % self.data_len
</code></pre>
<p>Template below</p>
<pre><code>class uti_dataset(torch.utils.data.Dataset):
    def __init__(self, args, data_path):
    
    # load dataset
    self.df = pd.Dataframe()
    ...
    self.data_len = len(self.df)
            
def __len__(self):
    return max(len(self.df),args.batch_size)

def __getitem__(self, idx):
    idx = idx % self.data_len
    filenames = self.df.iloc[idx]["filepaths"]
    # load and transform data
    # ...
    return images
</code></pre>
Hidden size gru

slavavs: <p>My model</p>
<pre><code class="lang-auto">class Model(torch.nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        #self.fc1 = nn.Linear(3,32)
        self.gru = nn.GRU(3, 256, 3)
        #nn.init.xavier_uniform_(self.gru.weight)
        self.fc3 = nn.Linear(256, 1)
        nn.init.xavier_uniform_(self.fc3.weight)
    def forward(self, x):
        batch = x.size(0)
        #out = self.fc1(out)
        out = torch.transpose(x,0,1)
        hidden = self.__init__hidden(batch)
        out,hidden = self.gru(out, hidden)
        #print(hidden.size()) #torch.Size([3, 128, 256])
        out = self.fc3(hidden)
        return out
    def __init__hidden(self, batch):
        hidden = torch.zeros(3, batch, 256).to(device)
        return hidden
</code></pre>
<p>I need the fc3 layer to accept not 3 sizes, but the last one. How to write this?</p>

phan_phan: <p>In your example, <code>hidden[-1]</code> is the hidden state for the last step, for the last layer.<br>
It is shaped <code>[batch_size, hidden_size]</code>, so</p>
<pre><code class="lang-auto">self.fc3(hidden[-1])
</code></pre>
<p>will do fine.</p>
Install Pytorch Error

siriushou: <p>I download pytorch from the official website. However, when I imported pytorch on my local windows computer, I got the following error as below. It seems torchvision suffered malfunction. What does it mean? Can someone know how to fix it? Thank you.</p>
<hr>
<p>ImportError                               Traceback (most recent call last)<br>
 in ()<br>
3 import torch<br>
4 import torch.nn as nn<br>
----&gt; 5 import torchvision<br>
6 import torchvision.datasets as dataset<br>
7 import torchvision.transforms as transforms</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torchvision_<em>init</em>_.py in ()<br>
1 from torchvision import models<br>
----&gt; 2 from torchvision import datasets<br>
3 from torchvision import ops<br>
4 from torchvision import transforms<br>
5 from torchvision import utils</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torchvision\datasets_<em>init</em>_.py in ()<br>
----&gt; 1 from .lsun import LSUN, LSUNClass<br>
2 from .folder import ImageFolder, DatasetFolder<br>
3 from .coco import CocoCaptions, CocoDetection<br>
4 from .cifar import CIFAR10, CIFAR100<br>
5 from .stl10 import STL10</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torchvision\datasets\lsun.py in ()<br>
----&gt; 1 from .vision import VisionDataset<br>
2 from PIL import Image<br>
3 import os<br>
4 import os.path<br>
5 import six</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torchvision\datasets\vision.py in ()<br>
1 import os<br>
2 import torch<br>
----&gt; 3 import torch.utils.data as data<br>
4<br>
5</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data_<em>init</em>_.py in ()<br>
2 from .distributed import DistributedSampler  # noqa: F401<br>
3 from .dataset import Dataset, IterableDataset, TensorDataset, ConcatDataset, ChainDataset, Subset, random_split  # noqa: F401<br>
----&gt; 4 from .dataloader import DataLoader, _DatasetKind, get_worker_info  # noqa: F401</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in ()<br>
10 import torch.multiprocessing as multiprocessing<br>
11 from . import IterableDataset, Sampler, SequentialSampler, RandomSampler, BatchSampler<br>
—&gt; 12 from . import _utils<br>
13 from torch._utils import ExceptionWrapper<br>
14 import threading</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data_utils_<em>init</em>_.py in ()<br>
12<br>
13 # old private location of the ExceptionWrapper that some users rely on:<br>
—&gt; 14 from torch._utils import ExceptionWrapper  # noqa: F401<br>
15<br>
16</p>
<p>ImportError: cannot import name ‘ExceptionWrapper’</p>
Python gramma "grad_h[h < 0] = 0" how to implement in libtorch?

suzhongmou: <p>hello ，i meet a problem in libtorch<br>
when i want to implement the tutorial follow ,i dont konw how to implament<br>
" grad_h[h&lt;0] = 0 " sentense.</p>
<pre><code class="lang-auto">import numpy as np

N, D_in, H, D_out = 64, 1000, 100, 10

x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)
learning_rate = 1e-6
for t in range(500):
    h = x.dot(w1)
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)
    loss = np.square(y_pred - y).sum()
    print(t, loss)
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h &lt; 0] = 0
    grad_w1 = x.T.dot(grad_h)
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2
</code></pre>

tom: <p>I think you are looking for <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.masked_scatter_" rel="nofollow noopener"><code>Tensor.masked_scatter_</code></a>.</p>
<p>Viele Grüße</p>
<p>Thomas</p>
Iterating through the generator crashes when I add normalization

mjabri: <p>Hello,</p>
<p>I had a piece of code that was working fine but when I added a normalization step in my “transforms”, it crashed.</p>
<p>Here’s my code for creating the generator :</p>
<pre><code class="lang-auto">data_transforms_train = transforms.Compose([
    transforms.Resize((512, 512)),
    transforms.RandomVerticalFlip(),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.8979], std=[0.3025])
])
    
params = {'batch_size': 1,
          'shuffle': True,
          'num_workers': 0}

training_set = Dataset(partition['train'], labels, transform=data_transforms_train)
training_generator = DataLoader(training_set, **params)
</code></pre>
<p>Again if I remove the line “transforms.Normalize(mean=[0.8979], std=[0.3025])”, everything goes back to working fine.</p>
<p>Any suggestion on what could be causing this ?</p>
<p>Thanks !</p>

albanD: <p>Hi,</p>
<p>What is the error that you get? Have you tried putting the normalize before the <code>ToTensor()</code> ?</p>

mjabri: <p>Hello <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>I try running the following in a jupyter notebook :</p>
<pre><code class="lang-auto">for test_images, test_labels in training_generator:  
    sample_image = test_images[0]   
    sample_label = test_labels[0]
</code></pre>
<p>And I just get a message that the kernel crashed and it will be restarting.</p>

albanD: <p>It is very hard to say without more informations.<br>
You might want to run it outside of the notebook to get more informations.</p>
Build extensions with/without CUDA

dashesy: <p>What does <code>torch.cuda.is_avaialble()</code> result mean? Does it mean the CUDA tools (nvcc,…) is available, or does it mean CUDA code can run (nvidia GPU/drivers/…)?<br>
I would like to build extensions if CUDA tools are available even if no GPU is present. It seems like I could <a href="https://github.com/microsoft/pytorch_od/issues/1" rel="nofollow noopener">import CUDaExtension only to fail when building starts</a>.</p>
<p>My current workaround is to <a href="https://github.com/microsoft/pytorch_od/commit/f3c27100b0ca04b03c441adc96afab05b2d57afb" rel="nofollow noopener">try-catch everything</a>.</p>
<pre><code class="lang-python">try:
    from torch.utils.cpp_extension import CUDAExtension
    import torch
    assert torch.cuda.is_available(), "No CUDA found"
except (ImportError, OSError, AssertionError) as e:
    CUDAExtension = None
    print("No CUDA was detected, building without CUDA error: {}".format(e))
</code></pre>
<p>But there should be a standard way?</p>
DataLoader .to_sparse() speed-up

mshuaibi: <p>My collate function for my dataloader involves the construction of a sparse matrix by <code>.to_sparse()</code>. The <code>.to_sparse()</code> call is the bottleneck in the data loading process and slows down training significantly. Is there anyway to get around this? I’m thinking of saving the sparse matrix for each batch with <code>torch.save()</code> and then load that instead of calling <code>.to_sparse()</code> every time. Not exactly sure how to go about though - such that a unique hash is constructed for a batch. Thanks!</p>
Bidirectional LSTM for classification

ironv: <p>I am using a bidirectional LSTM for a binary classification model on text sequences.</p>
<pre><code class="lang-auto">self.rnn = nn.LSTM(embed_size, hidden_size, batch_first=True, bidirectional=True)
out,(ht,ct) = self.rnn(X_packed)
print(ht.shape)
</code></pre>
<p>for <code>bs=64</code>, <code>hidden_size=128</code>, the dimension of <code>ht</code> is <code>2 x 64 x 128</code>.  This is then pushed to a FC layer and finally passed through a <code>sigmoid</code> activation function.</p>
<p>Should the input to the FC layer be <code>ht[-1]</code> i.e. <code>64 x 128</code>  or a concatenated version of the two  <code>torch.concat([ht[0],ht[-1]],dim=1)</code> i.e. <code>64 x 256</code>?</p>

phan_phan: <p>Here <code>ht[0]</code> corresponds to the last output of the <em>forward lstm</em>, and <code>ht[1]</code> corresponds to the last output of the <em>backward lstm</em>.<br>
You used a bidirectional lstm, so you might as well use the output of both directions! Your solution <code>torch.concat([ht[0],ht[-1]],dim=1)</code> seems correct.</p>
Extract element from a matrix

linyu: <p>Now, I have a float tensor B, that is B = torch.randn(2,3,4), and I also have a bool tensor C, that is the same size as B and its element is true or false. I want to extract the values in B, whose location is the true values of C. What should I do?</p>

ptrblck: <p>You could just index <code>B</code> with the BoolTensor:</p>
<pre><code class="lang-python">B = torch.randn(2, 3, 4)
C = torch.randint(0, 2, (2, 3, 4)).bool()
res = B[C]
</code></pre>
Expand the line

slavavs: <pre><code class="lang-auto">class NetDataset(Dataset):

    def __init__(self):
        xy = np.loadtxt('data_2d.txt', delimiter=';', dtype=np.float32)
        self.len = xy.shape[0]
        self.x_data = torch.from_numpy(xy[:, 0:-1])
        self.y_data = torch.from_numpy(xy[:, [-1]])

    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    def __len__(self):
        return self.len
</code></pre>
<p><code>self.x_data = torch.from_numpy(xy[:, 0:-1])</code><br>
it gives me the following view [0.1 0.4 0.5 0.2 0.8 0.5]</p>
<p>How can i get this [[0.1 0.4 0.5][0.2 0.8 0.5]]?</p>

albanD: <p>Hi,<br>
What is <code>xy[:, 0:-1]</code> supposed to do?<br>
Also <code>xy[:, [-1]]</code>?</p>

slavavs: <pre><code class="lang-auto">dataset = NetDataset()
train_loader = DataLoader(dataset=dataset, batch_size=128, shuffle=True)
</code></pre>
<p>…</p>
<pre><code class="lang-auto">for i, (inputs, labels) in enumerate(train_loader):
y_pred = model(inputs)
loss = criterion(y_pred, labels)
</code></pre>

slavavs: <aside class="quote no-group" data-username="slavavs" data-post="1" data-topic="57186">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/slavavs/48/15429_2.png" class="avatar"> slavavs:</div>
<blockquote>
<p>[[0.1 0.4 0.5][0.2 0.8 0.5]]</p>
</blockquote>
</aside>
<p>I am currently using nn.linear. I want to use nn.gru [[0.1 0.4 0.5][0.2 0.8 0.5]] - the sequence is 2</p>

albanD: <p>Sorry, what I meant was,<br>
What do you expect <code>xy</code> to be here?<br>
And what do you expect <code>xy[:, 0:-1]</code> to give you?<br>
I am not familiar with such notation.</p>
Backward on iterative parameter updates

boskova: <p>Dear all,</p>
<p>The core of my architecture consists of encoding the features from the input image, and then iteratively regressing the incremental changes of output parameters, where the regressor regresses on features and the current value of the parameters, in each iteration.  The loss is backpropagated in every iteration, and the step on the regressor optimizer is taken in every iteration likewise. The step for the optimizer for the feature encoder is done only once, at the end of the of the last iteration. The goal is to optimize the encoder parameters only once, after the incremental regression optimization has been performed.<br>
The main part of the code to do this is shown below:</p>
<p>current_parameters = torch.zeros(size, device=self.device, dtype=torch.float)<br>
features = self.encoder(images)</p>
<p>for n in range(self.n_regresor_iterations):</p>
<pre><code>self.optimizer_encoder.zero_grad()
self.optimizer_param_regressor.zero_grad()
parameters = self.param_regressor(features, current_parameters)
current_parameters += parameters

loss_total = loss_fn(current_parameters, ground_truth_parameters)

if n &lt; self.n_regresor_iterations - 1:
    loss_total.backward(retain_graph=True)
else:                
    loss_total.backward()

self.optimizer_param_regressor.step()
if n == self.n_regresor_iterations - 1:
    self.optimizer_encoder.step()
</code></pre>
<p>return loss_total.item()</p>
<p>This seems to be training just fine, however, I have been unsuccessfully looking for possible optimizations:<br>
Is it possible to modify this code so that I can get rid of retain_graph=True above?<br>
It increases the memory consumption proportionally to the number of iterations.<br>
Do you see any other problems with this code?</p>
<p>Thank you in advance</p>
fMRI data loader

Rozgns: <p>Hi everyone,</p>
<p>I am trying to set up a network to train on fMRI task data (sequential volumes) along side with T1 (single volume). fMRI data has 1200 volumes for one single subject. What is a smart and efficient way to set up this data loader to avoid memory issues? Help me brain storming on this.</p>
<p>Thank you in advance,<br>
Roza</p>

JuanFMontesinos: <p>It depends on you resources. The fastest way is always using numpy arrays with memory map but ofc that’s raw data and you need hard disk to store it. If your data is compressed then you will have to pay the price of reading and decoding.</p>

albanD: <p>If your data is too big to fit in memory. You can use a Dataset/Dataloader similar to what is used for Imagenet: Every time a sample is needed, it is read on the disk, preprocessed then returned. The dataloader <code>num_workers</code> argument allow you to have multiple processes doing this loading/preprocessing on the side to make sure it is not a bottleneck.</p>
<p>In particular, you can implement your own <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset" rel="nofollow noopener">torch.utils.data.Dataset</a>. Which requires to implement a <code>__getitem__</code> to retrieve one sample from your dataset and <code>__len__</code> that tells the length of your dataset.<br>
You should do loading and any required preprocessing in the <code>__getitem__</code> method and return one sample ready to be forwarded in your network.<br>
Then you can use the basic dataloader with few workers to make that dataset loading parallel !  Note that you should experiment with the number of workers and find the best one by experiment.</p>
Deep Learning methods for Text Generation

OverclockRo: <p>Greetings to everyone,</p>
<p>I want to design a system that is able to <strong>generate stories</strong> or poetry based on a large dataset of text, <strong>without being needed to feed a text description/start/summary as input at inference time.</strong></p>
<p>So far I did this using <strong>RNN</strong> 's, but as you know they have a lot of flaws. My question is, what are the best methods to achieve this task at the time? I searched for possibilities using <strong>Attention mechanisms</strong> , but it turns out that they are fitted for translation tasks.</p>
<p>P.S. I know about GPT-2, Bert, <strong>Transformer</strong> , etc., but all of them need a text description as input, before the generation and this is not what I’m seeking. I want a system able to generate stories from scratch after training.</p>
<p>Thanks a lot!</p>

vdw: <p>What do you mean by “from scratch”? Say you have an encoder-decoder model, you have to give the decoder <em>something</em> to generate the text output. The only thing I see would be that you sample a random vector from your hidden/latent space, give it to the decoder and hope for the best. I tried that a bit with a Variational Autoencoder for sentences, and the results are questionable at best (but I cannot exclude that I made mistakes)</p>

OverclockRo: <p>What I mean is that it should be able to generate text like the RNN does.<br>
I give it a “” token and then I refeed its outputs, when it comes to inference time.<br>
GPT-2 requires a starting phrase, for example.</p>
<p>My dream model would be A kind of Encoder-Decoder model where encoder starts form  and then expands with each generated token in the decoder, while the decoder would be able to use attention mechanisms on the Encoder, or so to say, over the entire generated text.</p>
<p>I hope I explain in such a way to be understandable.</p>

zhangguanheng66: <p>I would suggest you to take a look at the <a href="https://github.com/pytorch/examples/tree/master/word_language_model" rel="nofollow noopener">word language model</a>. During inference, you need to provide a random number as input.</p>
Element-wise multiplication of a vector and a matrix

dnnagy: <p>I’m searching the net for a multiplication which is applied between an 1-d tensor anf an n-d tensor.<br>
Expected behaviour:</p>
<pre><code class="lang-auto">A = torch.tensor(np.array([[[1., 0., 0.],
                            [0., 1., 0.],
                            [0., 0., 1.]],
                           [[2., 0., 0.],
                            [0., 2., 0.],
                            [0., 0., 2.]]]))

v = torch.tensor(np.array([2.0, 1.0]))

torch.element_wise(v, A)
</code></pre>
<p>expected result:</p>
<pre><code class="lang-auto">tensor([[[2., 0., 0.],
         [0., 2., 0.],
         [0., 0., 2.]],
        [[2., 0., 0.],
         [0., 2., 0.],
         [0., 0., 2.]]])
</code></pre>

dnnagy: <p>I found a solution, but I wonder if there exist any simpler solution?</p>
<pre><code class="lang-auto">def ev(a, b):
    N = b.shape[0]
    shp = b.shape[1:]
    return torch.mul(a, b.view(N, -1).transpose(0,1)).transpose(0,1).view(N, *shp)

R = ev(v, A) # gives the right answer
</code></pre>

albanD: <p>You can do the following:</p>
<pre><code class="lang-auto">v.view(-1, 1, 1).expand_as(A) * A
</code></pre>
<p>Note that the automatic broadcasting can take care of the expand and so you can simply do:</p>
<pre><code class="lang-auto">v.view(-1, 1, 1) * A
</code></pre>

albanD: <p>That works as well <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I usually prefer the version with functions compared to advance indexing because I know when I get a view of the data and when I get a copy.</p>
KL divergence results different from numpy

razvanc92: <p>I’ve been trying to implement the KL divergence using tf/pytorch and numpy. So far (tks to <a class="mention" href="/u/nikronic">@Nikronic</a>) the tf and pytorch results are similar but the numpy version is quite off, and I can not find any reason why. One thing that I’ve noticed is that if the preds and labels are containing one array repeated multiple times (np.broadcast_to(np.random.uniform(0., 1., (11,), (64, 12, 300, 11))) the results are very similar. Any help will be more than appreciated.</p>
<pre><code class="lang-auto">preds = np.random.uniform(0., 1., (64, 12, 300, 11))
labels = np.random.uniform(0., 1., (64, 12, 300, 11))


preds_tf = tf.distributions.Categorical(probs=tf.convert_to_tensor(preds))
labels_tf = tf.distributions.Categorical(probs=tf.convert_to_tensor(labels))
tf_res = tf.reduce_mean(tf.distributions.kl_divergence(preds_tf, labels_tf))

preds_torch = torch.distributions.Categorical(probs=torch.from_numpy(preds))
labels_torch = torch.distributions.Categorical(logits=torch.from_numpy(labels).log())
torch_res = torch.mean(kl_divergence(preds_torch, labels_torch))

np_res = np.mean(np.sum(preds * np.log((preds / labels)), axis=-1))
print(tf_res.numpy(), torch_res.item(), np_res)
</code></pre>
Multiprocessing with custom collate_fn problems: AttributeError: Can't pickle local object

timbmg: <p>Hi, I implemented my own collate_fn as I need to pad data up to a variable length. The length up to which the batch has to be padded is also determined in this function.</p>
<p>When I set the num_workers &gt; 0 in the DataLoader, it runs fine on my cpu. However, und gpu I get the following error:<br>
<code>RuntimeError: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCCachingAllocator.cpp:507</code><br>
I read several similar threads, where it is recommended to set the start method for multiprocesing like this:<br>
<code>torch.multiprocessing.set_start_method('forkserver')</code><br>
Now if I do that, I get the following error on both my cpu and gpu:<br>
<code>AttributeError: Can't pickle local object 'MyDataset.get_collate_fn.&lt;locals&gt;.collate_fn'</code><br>
Maybe important to know is that I pass the collate_fn to the Dataloader like this:</p>
<pre><code class="lang-python">class MyDataset(Dataset)
    ...
    @static_method
    def get_collate(device):
        def collate_fn(batch):
            ...
            batch = batch.to(device)
        return collate_fn

data_loader = DataLoader(MyDataset(), collate_fn=MyDataset.get_collate(device))
</code></pre>

wenjieyang: <p>same issue. Do you have a solution?</p>

Oliver_Batchelor: <p>Same thing here, only with set_start_method(‘spawn’)  or ‘forkserver’, ‘fork’ start method is fine. Pytorch 1.2, Linux</p>
Shared weights with model parallelism

shelpuk: <p>Our setup involves initial part of the network (input interface) which run on separate GPU cards. Each GPU gets its own portion of data (model parallelism) and process it separately.</p>
<p>Each input interface, in turn, it itself a complex nn.Module. Every input interface can occupy one or several cards (say, interface_1 runs on GPU 0 and 1, interface_2 - on GPU 2 and 3 and so on).</p>
<p>We need to keep the weights of these input interface the same all over the training. We also need them to run in parallel to save training time which is already weeks for our scenario.</p>
<p>The best idea we can think of was initializing the interfaces with the same weights and then average the gradients for them. As the interfaces are identical, updating same weights with the same gradients should keep them the same all over the training process thus achieving desired “shared weights” mode.</p>
<p>However, I cannot find any good solution for changing values of these weights and their gradients represented as Parameter in PyTorch. Apparently, PyTorch does not allow to do so.</p>
<p>Our current state is: if we copy.deepcopy the ‘parameter.data’ of the “master” interface and assign it to ‘parameter.data’ of the “slave” interface, the values are indeed changed but .to(device_id) does not work and keeps them at the “master” device. However, we need them to move to a “slave” device.</p>
<p>Could someone please tell me if it is possible at all or, if not, if there is a better way to implement shared weights along with the parallel execution for our scenario?</p>

ptrblck: <p>Based on your description, I assume some <code>parallel</code> methods could probably help in your use case.<br>
The usage of the <code>.data</code> attribute is not recommended and you could try to adapt some synchronization work flows from <a href="https://github.com/pytorch/pytorch/blob/d5298b6e666c2110aaac94e0d35413bdde911c5b/torch/nn/parallel/distributed.py#L494" rel="nofollow noopener">DDP</a>.</p>

shelpuk: <p>You are right, the whole system is the variation of model parallelism. We were also thinking about data parallelism across the interfaces but that does not work as in this case PyTorch puts all input data on GPU 0 before distributing it across parallel nodes. For our scenario, the input data is so large that it takes more than entire 32Gb of GPU 0 memory itself.</p>

ptrblck: <p>In that case, try to also stick to the DDP approach (I would generally recommend DDP instead of DataParallel).</p>
Conda install of pytroch doesn't work

smu226: <p>Hello! I created an environment with conda and I want to install pytorch in it, but it doesn’t work. After I get inside my environment with <code>source activate env_name</code> I tried this: <code>conda install pytorch torchvision -c pytorch</code> (I also tried it like this: <code>conda install -c pytorch pytorch torchvision</code>) but I am getting this error:</p>
<pre><code>Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
Solving package specifications: ......
Error: Could not find some dependencies for pytorch: mkl &gt;=2018, cudatoolkit &gt;=9.0,&lt;9.1, blas * mkl, cudatoolkit &gt;=10.0,&lt;10.1, cudatoolkit &gt;=9.2,&lt;9.3, blas * openblas, cudnn 7.0.*, cudatoolkit 9.*

Did you mean one of these?

    pytorch, pytorch-gpu, pytorch-cpu

Did you mean one of these?

    cudatoolkit

You can search for this package on anaconda.org with

    anaconda search -t conda cudatoolkit 9.*

 (and similarly for the other packages)
</code></pre>
<p>Here are my installed packages:</p>
<pre><code>backports                 1.0                      py34_0  
backports.shutil-get-terminal-size 1.0.0                     &lt;pip&gt;
decorator                 4.0.11                   py34_0  
get_terminal_size         1.0.0                    py34_0  
ipython                   4.2.0                    py34_0  
ipython-genutils          0.1.0                     &lt;pip&gt;
ipython_genutils          0.1.0                    py34_0  
libgfortran               1.0                           0  
numpy                     1.9.2                    py34_0  
openssl                   1.0.2l                        0  
path.py                   10.0                     py34_0  
pexpect                   4.2.1                    py34_0  
pickleshare               0.7.4                    py34_0  
pip                       9.0.1                    py34_1  
ptyprocess                0.5.1                    py34_0  
python                    3.4.5                         0  
readline                  6.2                           2  
scipy                     0.16.0               np19py34_0  
setuptools                27.2.0                   py34_0  
simplegeneric             0.8.1                    py34_1  
six                       1.10.0                   py34_0  
sqlite                    3.13.0                        0  
tk                        8.5.18                        0  
traitlets                 4.3.1                    py34_0  
wheel                     0.29.0                   py34_0  
xz                        5.2.3                         0  
zlib                      1.2.11                        0  
</code></pre>
<p>What should I do? Thank you!</p>

ptrblck: <p>It seems you are using Python3.4. Could you create a new conda environment with Python &gt;= 3.5 and try to install PyTorch again?</p>
Autograd/backward in RNN

chiho_Shin: <p>Hello,</p>
<p>I’m developing a framework and especially, RNN modules for capstone project using C++.</p>
<p>To develop our project, I just see how pytorch works related with calculating gradients of RNNs.</p>
<p>While trying to find source code, especially in /pytorch/torch/nn/modules/rnn.py<br>
there is a variable named _rnn_impls with _VF.rnn_tanh and _VF.rnn_relu.<br>
In addition, in forward function in RNN class, the results are returned using impl function.</p>
<p>However, I could not find what the VF means and how to use it.<br>
I find there is a VF class in module, but it is very short code and not enough information for me.<br>
Is that related with backend c source code ?<br>
Could you help me with how it works and finding related source codes?</p>
<p>Thanks.</p>