akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Updating consecutive masks

username: Gabriele_Orlando
cooked: <p>Hello,<br>
I am having some troubles dealing with masks. What I do is the following:</p>
<ol>
<li>I start with a (very) large set of data points.</li>
<li>I mask them and I take only the relevant ones.</li>
<li>I make some calculations</li>
<li>I mask them and I take only the relevant ones<br>
…</li>
</ol>
<p>basically I don’t know at the very beginning which data points are relevant and which ones are not, and in order to avoid useless calculations I perform consecutive masking operation. At the end of the run, I want to merge all the masks.</p>
<p>I was working with pytorch 1.7 until a couple of hours ago and I used to do something like this:</p>
<p>fullMask = torch.ones(shapeTensor).bool()<br>
fullMask[fullMask] = first_level_mask<br>
fullMask[fullMask] = second_level_mask<br>
…</p>
<p>with pytorch 1.9 this doesn’t work anymore and I get the “some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation.” error. I would definitely avoiding spawning .clone() everywhere since I am dealing with very large tensors.</p>
<p>I was wondering if someone can help me in finding the cleanest way to perform this operation.</p>
<p>Thank you in advance for the help.</p>
<p>Gabriele</p>
AttributeError: 'DataLoader' object has no attribute 'persistent_workers'

username: Hristo_Petkov
cooked: <p>Hello! Just upgraded from my old version of pytorch to 1.9. I ran some code I am using which involves DataLoaders. In the old version of Pytorch it works as expected. In this one I get this error:</p>
<p>AttributeError: ‘DataLoader’ object has no attribute ‘persistent_workers’</p>
<p>I’ve been searching in the source code for the class to see if there is such an attribute there and there is. <strong>dict</strong> also proves that there is such an attribute. So what gives?</p>
<p>Any help would be appreciated!</p>

username: ptrblck
cooked: <p>Could you double check the installed PyTorch version via <code>torch.__version__</code>, please, as this attribute should be in <code>1.9.0</code> and I can also use it?</p>

username: Hristo_Petkov
cooked: <p>It is version 1.9.0 yes I already checked that!</p>
<p>The torch.<strong>version</strong> command yields the result 1.9.0</p>
<p>Also when I check the attributes of the Dataloader I get this:</p>
<p>{‘dataset’: &lt;torch.utils.data.dataset.TensorDataset object at 0x000001188463B790&gt;, ‘num_workers’: 0, ‘pin_memory’: False, ‘timeout’: 0, ‘worker_init_fn’: None, ‘_DataLoader__multiprocessing_context’: None, ‘_dataset_kind’: 0, ‘batch_size’: 100, ‘drop_last’: False, ‘sampler’: &lt;torch.utils.data.sampler.SequentialSampler object at 0x000001188463B700&gt;, ‘batch_sampler’: &lt;torch.utils.data.sampler.BatchSampler object at 0x000001188463B610&gt;, ‘generator’: None, ‘collate_fn’: &lt;function default_collate at 0x000001188A783DC0&gt;, ‘_DataLoader__initialized’: True, ‘_IterableDataset_len_called’: None}</p>
<p>So as you can see there is something wrong here. Not sure why though?</p>

username: ptrblck
cooked: <p>I’m still unsure, what might be causing the issue, as this code snippet works fine for me:</p>
<pre><code class="lang-python">import torch
from torch.utils.data import Dataset, DataLoader, TensorDataset

print(torch.__version__)
dataset = TensorDataset(torch.randn(100, 1), torch.randn(100, 1))
loader = DataLoader(dataset, batch_size=25, num_workers=2, persistent_workers=True)

for data, target in loader:
    print(data.shape)
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">1.9.0+cu111
torch.Size([25, 1])
torch.Size([25, 1])
torch.Size([25, 1])
torch.Size([25, 1])
</code></pre>
<p>Are you seeing the same issue using my code?</p>
Multi processing problem in windows 10 platform

username: Torki
cooked: <p>I read a lot on the Internet about the multiprocessor problem with using Dataloader in Windows.</p>
<p>The solutions are here:<br>
1-use if clause to cover for data loader loop.<br>
2-use pickle version 4.<br>
3-set DEFAULT_PROTOCOL in pickle to 4<br>
4-set num_worker=0</p>
<p>For the solution <span class="hashtag">#1</span>,2,3:<br>
The problem persist again after changes.</p>
<p>For the solution <span class="hashtag">#4:</span><br>
Code executed but it’s very slow due to num_worker=0</p>
<p>I check the multiprocessing problem in windows with bellow minimal famous code:</p>
<p>from multiprocessing import Process, freeze_support</p>
<p>def foo():<br>
print (‘hello’)</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
freeze_support()<br>
p = Process(target=foo)<br>
p.start()</p>
<p>after execution this error occurred.<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “D:\ProgramData\Anaconda3\envs\Torki\lib\multiprocessing\spawn.py”, line 116, in spawn_main<br>
exitcode = _main(fd, parent_sentinel)<br>
File “D:\ProgramData\Anaconda3\envs\Torki\lib\multiprocessing\spawn.py”, line 126, in _main<br>
self = reduction.pickle.load(from_parent)<br>
AttributeError: Can’t get attribute ‘foo’ on &lt;module ‘<strong>main</strong>’ (built-in)&gt;</p>
<p>OS: Windows 10<br>
anaconda<br>
torch version= 1.9<br>
pickle version=4</p>
<p>can anyone help me?</p>
<h1>
<a name="ptrblckhttpsdiscusspytorchorguptrblck-1" class="anchor" href="#ptrblckhttpsdiscusspytorchorguptrblck-1"></a><a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a>
</h1>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn after 22k iterations

username: erenovic
cooked: <p>Hello everyone, I keep getting this error but couldn’t find the reason. The error occurs during the training steps, but it appears over 22k iterations after the training starts.</p>
<pre><code class="lang-auto">def train_one_step(data, model, optimizer, device):
    # data is a list of images
    
    model.train()

    avg_loss = torch.zeros((1)).to(device)
    avg_loss2 = torch.zeros((1)).to(device)
    
    x = data[0]
    hidden_state = None

    for i in range(1, len(data)):
       x, hidden_state, state_loss2  = model(x, data[i], hidden_state, train=True)

        avg_loss += calculate_loss(x, data[i])
        avg_loss2 += state_loss2


    avg_loss /= len(data)
    avg_loss2 /= len(data)
    
    loss = 10 * avg_loss +  avg_loss2

    optimizer.zero_grad()
    print(loss.requires_grad)
    loss.backward()
    optimizer.step()
    
    return avg_loss.item(), avg_loss2.item(), loss.item()
</code></pre>
<p>Here calculate loss is a function simply calculating the means squared error using torch.mean((x - data[i])**2). As I have previously stated, the model is successfully trained for 22k iterations but afterwards loss.requires_grad suddenly turns to False (I have observed that it was True in previous iterations).<br>
I did not provide any further info about my model but are there any reasons for requires_grad to turn False without manually setting it?</p>

username: ptrblck
cooked: <p>Detaching e.g. the output from the computation graph would set the <code>requires_grad</code> attribute to <code>False</code>, which should not happen “automatically” after a specific number of iterations.<br>
Are you using conditions inside the <code>forward</code> method, which could pick another code path and could either directly detach tensors or use non-differentiable operations, or any data-dependent control flow which could do the same?</p>

username: erenovic
cooked: <p>Hey Piotr, thanks for the swift answer, as I had no further explanation, I tried to debug it myself. The error was completely on me, I failed when I was loading the data and it somehow caused this error.  But again, thanks for the help.</p>
How can I handle the Cuda out of memory error?

username: S_M
cooked: <p>Hi, I am trying to run the ResNet18 model using GTX 1080 GPU but the Cuda out-of-memory error arises. I reduced the baches size to 4 and use</p>
<pre><code class="lang-auto">    learn=0
    gc.collect()
    torch.cuda.empty_cache()
</code></pre>
<p>but they didn’t work.<br>
could anyone give me some advice?</p>

username: ptrblck
cooked: <p>Clearing the cache wouldn’t avoid the OOM issue and could just slow down your code, so you would either need to reduce the batch size more, lower the memory usage of the model (e.g. less/smaller layers), reduce the spatial size of the input, or use <code>torch.utils.checkpoint</code> to trade compute for memory.</p>

username: S_M
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="127580">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.utils.checkpoint</p>
</blockquote>
</aside>
<p>what kind of help torch.utils.checkpoint can do?</p>

username: S_M
cooked: <p>could you please give me an example code that shows how <code>torch.utils.checkpoint</code> is used about my problem?</p>

username: artyom-beilis
cooked: <aside class="quote no-group" data-username="S_M" data-post="1" data-topic="127580">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/s_m/48/19733_2.png" class="avatar"> S_M:</div>
<blockquote>
<p>Hi, I am trying to run the ResNet18 model using GTX 1080 GPU but the Cuda out-of-memory error arises. I reduced the baches size to 4 and use</p>
</blockquote>
</aside>
<p>1080 is more than enough to handle batch size of 64 with ease on 1080. It is relatively small network.</p>
<p>Look for cuda memory leak somewhere else , maybe you keep referencing/copying some tensors you don’t need etc.</p>

username: S_M
cooked: <p>It seems to be reasonable. How can I check it? Is there any toolbox or method?</p>
Finite differences along specific dimension

username: aRI0U
cooked: <p>I need to compute the approximate derivative of a tensor <code>x</code> using finite differences, i.e. <code>x[1:] - x[:-1]</code>. My problem is that my input tensor is multidimensional and that I need to compute those finite differences along different dimensions. I would thus need a generic function <code>f(x, dim)</code> that basically do the following:</p>
<pre><code class="lang-auto">if dim == 0:
    return x[1:] - x[:-1]
if dim == 1:
    return x[:,1:] - x[:,:-1]
if dim == 2:
    return x[:,:,1:] - x[:,:,:-1]
if dim == -1:
    return x[...,1:] - x[...,:-1]
etc...
</code></pre>
<p>The only trick I found was to use <code>index_select</code>, but it is much slower to define indices and iterate over them than using slices, as you can see below:</p>
<pre><code class="lang-auto">$ python3 -m timeit -s 'import torch; a = torch.randn(5, 2048, 5)' 'a[:,1:,:] - a[:,:-1,:]'
10000 loops, best of 3: 75.4 usec per loop
$ python3 -m timeit -s 'import torch; a = torch.randn(5, 2048, 5)' 'indices = torch.arange(a.size(1)); torch.index_select(a, 1, indices[1:]) - torch.index_select(a, 1, indices[:-1])'
10 loops, best of 3: 26.8 msec per loop
</code></pre>
<p>Does it exist a clean and fast way to solve this problem?</p>

username: tom
cooked: <p>You can use <code>narrow</code> to your advantage:</p>
<pre><code class="lang-python">a = torch.randn(5, 2048, 5)
print(((a[:,1:,:] - a[:,:-1,:])-(a.narrow(1, 1, a.size(1)-1)-a.narrow(1, 0, a.size(1)-1))).abs().max().item())
%timeit a[:,1:,:] - a[:,:-1,:]
%timeit a.narrow(1, 1, a.size(1)-1)-a.narrow(1, 0, a.size(1)-1)
</code></pre>
<p>gets me</p>
<pre><code class="lang-auto">0.0
14.3 µs ± 190 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
11.9 µs ± 15.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
</code></pre>
<p>so by going to PyTorch directly I even shave off a couple of µs.</p>
<p>Best regards</p>
<p>Thomas</p>

username: aRI0U
cooked: <p>It works perfectly! Thanks a lot for your answer!</p>

username: 335621
cooked: <p>What if the a[1:,:,:] - a[:-1,:,:]?</p>
How do you multiply two sparse matrices?

username: samsepiol
cooked: <p>I have two sparse tensors <code>a</code> and <code>b</code> of sizes <code>(A, B, C, D)</code> and <code>(A, B)</code> respectively.</p>
<p>How do I multiply the two of them?</p>
<p>I don’t want to copy <code>b</code> across a new third and fourth dimension <code>C</code> and <code>D</code> times because that wastes memory and also takes compute time. I also don’t want to convert both to dense and back to sparse (although converting <code>b</code> to dense at some point would be fine; it’s just <code>a</code> that takes time to convert).</p>
Model Interpretability using Captum

username: kountaydwivedi
cooked: <p>While using any model interpretability method, should we pass the train set or the test set as input parameter?</p>

username: googlebot
cooked: <p>As I understand it, base scenario is with the training set - you check which features are discovered as important during the <strong>training</strong>. There may be some advanced uses on a held-out dataset (e.g. comparision plots or exploring generalization problems), but that’s more serious scrutinizing than basic X “columnwise” importance checking.</p>
Implementing OpenCL backend for pytorch

username: artyom-beilis
cooked: <p>I started developing a library that implements common DL operations in OpenCL.<br>
It is somewhat similar to cudnn/miopen with addition of providing a library for inference and basic training.</p>
<p><a href="https://github.com/artyom-beilis/dlprimitives" rel="noopener nofollow ugc">https://github.com/artyom-beilis/dlprimitives</a></p>
<p>The project is in very early stages but it is already:</p>
<ul>
<li>Outperforms existing OpenCL DL implementations: plaidml and caffe/opencl-branch by 150-200% on Nvidia and AMD platforms: <a href="https://github.com/artyom-beilis/dlprimitives/blob/master/docs/summary.md" rel="noopener nofollow ugc">https://github.com/artyom-beilis/dlprimitives/blob/master/docs/summary.md</a>
</li>
<li>It is validated on alexnet, resnet18/50, vgg16 and mobilenet on nVidia, AMD and Intel GPUs</li>
</ul>
<p>I’m looking for a way to create a custom backend for pytorch. It is clear for me that it is lots of work, but it is technical part. The big, critical and complex part is writing good high performance kernels that worth something is already in very good shape.</p>
<p>I looked into “out-of-source” backend but the documentation is lacking and I wonder if there is some template and some minimal useful backend that I can “rewrite” for OpenCL.</p>
<p>Any tutorials and pointers will be appreciated</p>