akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Does C++ cudnn_convolution_backward respect Python's cuda stream?

rinkujadhav2013: <p>I’m using C++ <code>cudnn_convolution_backward</code> to implement a custom op. I’m making use of streams in there. My question is that if I call <code>cudnn_convolution_backward</code> inside a stream context manager, will the cudnn backward pass run on that stream or not?<br>
I looked at function’s <a href="https://github.com/pytorch/pytorch/blob/v1.10.2/aten/src/ATen/native/cudnn/ConvShared.cpp#L393" rel="noopener nofollow ugc">implementation in C++</a> but couldn’t find anything there.  Python <code>set_stream</code> is a wrapper around something but I couldn’t find the original source code it wraps and I don’t see anything setting up cudnn stream here either: <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/cuda/Stream.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Stream.cpp at master · pytorch/pytorch · GitHub</a></p>

ptrblck: <p>The cuDNN calls use their cuDNNHandle, which uses the current stream as seen <a href="https://github.com/pytorch/pytorch/blob/61d6c4386459441710fb4cfa2929a3f77e95e5f7/aten/src/ATen/cudnn/Handle.cpp#L48">here</a>.</p>
Any plans on distributed training for large models?

weberxie: <p>Hi team,</p>
<p>I have noticed the PyTorch new feature DistributedTensor, it provides a new way to build a large model. And the new feature TorchDyname of PyTorch 2.0 provides a new to capture graphs. So are there any plans for PyTorch to capture the whole graph of a single GPU and <strong>automatically</strong> translate it to distributed mode with <strong>model parallel and pipeline parallel</strong> based on the GPU resources?</p>
<p>Thanks, any response will be appreciated.</p>

rvarm1: <p>cc <a class="mention" href="/u/wanchaol">@wanchaol</a> <a class="mention" href="/u/aazzolini">@aazzolini</a></p>

wanchaol: <p><a class="mention" href="/u/weberxie">@weberxie</a> Thanks for the question! Yes we are actively exploring and building out the compiler based distributed training stack with DTensor and PyTorch 2.0. We will release it as prototype to use once we finish the building blocks.</p>
Torch.jit.script RuntimeError: Function was not annotated as having type None, but does not return along all paths:

Rachel1: <p>I met below error when doing torch.jit.script(model) on Pop!_OS with python3.8. It’s using torch 1.6.0.<br>
The same code worked on CentOS with python3.6. But Pop!_OS with python3.8 is where I need to run the code. Looking forward for any inputs.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/envs/lib/python3.8/site-packages/torch/jit/__init__.py", line 1516, in script
    return torch.jit._recursive.create_script_module(obj, torch.jit._recursive.infer_methods_to_compile)
  File "/envs/lib/python3.8/site-packages/torch/jit/_recursive.py", line 318, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/envs/lib/python3.8/site-packages/torch/jit/_recursive.py", line 376, in create_script_module_impl
    create_methods_from_stubs(concrete_type, stubs)
  File "/envs/lib/python3.8/site-packages/torch/jit/_recursive.py", line 292, in create_methods_from_stubs
    concrete_type._create_methods(defs, rcbs, defaults)
RuntimeError: 
Function was not annotated as having type None, but does not return along all paths:
  File "/scripts/nn_models.py", line 47
    def forward(self, x: Tensor, hidden: Tuple[(Tensor, Tensor)]) -&gt; Tuple[(Tensor, Tuple[(Tensor, Tensor)])]:
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  &lt;--- HERE
        """Neural net forward pass. It is automatically called when use instance of LSTMnet.```
</code></pre>

ptrblck: <p>Could you update PyTorch to the latest release and see if you would still see the same error?<br>
If so, could you post a minimal, executable code snippet to reproduce the error, please?</p>

Rachel1: <p>Thank you for your response!<br>
This problem seems not a problem of pytorch. The problem is related to the compiling to .pyc.</p>
DataParallel yields different results with different runs on same machine

jay24: <p>Hi, I am using nn.DataParallel for distributed training on a machine with 4 A100 GPUs. I got different results (e.g., MAE, RMSE) when running the same piece of code on the same machine. However, my code yielded exactly the same results when I use only one GPU (set the number of visible GPU number to 1). Anyone have ideas why I got different results when using nn.DataParallel? Thank you.</p>

ptrblck: <p>Did you already set all deterministic flags and seeded the code as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>?</p>

jay24: <p>Yes, I think so. Here is what I did in my code, let me know if I missed anything.</p>
<pre><code class="lang-auto">def set_seed(seed):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

set_seed(args.seed)
</code></pre>
<p>To further clarify this, when I set os.environ['CUDA_VISIBLE_DEVICES]=‘0’ (training on single GPU), I can repeat the results with different runs. However, when I put os.environ['CUDA_VISIBLE_DEVICES]=‘0,1,2,3’ (training with 4 GPUs) in my code, different results will be obtained with different runs.</p>

Yanli_Zhao: <p>DataParallel will involve communication synchronization using NCCL collectives, as far as I know, NCCL collectives may not be deterministic? <a href="https://github.com/NVIDIA/nccl/issues/157" class="inline-onebox" rel="noopener nofollow ugc">how to make NCCL deterministic for different size for the same thing? · Issue #157 · NVIDIA/nccl · GitHub</a></p>

jay24: <p>Any suggestions on how to solve this problem? And will DDP also encounter this problem? Thank you.</p>
Help installing pytorch with cuda on geforce 2070 Super

Cro_Tech: <p>Hi, I’m trying to use pytorch with cuda. I have 2070 Super. I’ve installed cuda toolkit (v 10.2), and I’ve installed pytorch using pytorch docs:</p>
<p><code>conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=10.2 -c pytorch</code></p>
<p>But when I try to use cuda I get an error</p>
<blockquote>
<p>AssertionError: Torch not compiled with CUDA enabled</p>
</blockquote>

eqy: <p>What does <code>torch.__version__</code> show in your interpreter? Have you tried checking that there were no preexisting PyTorch installs on your system?</p>

Cro_Tech: <p>Here’s the output, and I’ve created new conda environment so there should not be other torch installs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8bd079926c82ec6aff11a586fa8619531033e6a.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8bd079926c82ec6aff11a586fa8619531033e6a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8bd079926c82ec6aff11a586fa8619531033e6a.png" alt="image" data-base62-sha1="zurpe6jKt7ctZmhPQSkhDU9b1Ng" width="690" height="111" data-dominant-color="2F333B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1088×176 5.74 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Remove zeros from tensor but keeping dimensions

MagedSaeed: <p>Suppose I have the following tensor:</p>
<pre><code class="lang-python">a = torch.Tensor([[1,2,0],[0,-1,0]])
</code></pre>
<p>I want to remove zeros but keep the dimensions. The results should be:</p>
<pre><code class="lang-auto">[[1,2],[-1]]
</code></pre>
<p>Is there an easy way to achieve this?</p>

ptrblck: <p>You could try to use the experimental <code>NestedTensor</code> support as seen here:</p>
<pre><code class="lang-python">a = torch.tensor([[1,2,0],[0,-1,0]])

a_list = [a_[a_!=0.] for a_ in a]
b = torch.nested.as_nested_tensor(a_list)
print(b)
# nested_tensor([
#   tensor([1, 2]),
#   tensor([-1])
# ])
</code></pre>

Sam_Lerman: <p>I had no idea this existed! Does it provide efficiency optimizations under the hood for things like batch-multiplications or is it just a semantic convenience? Seems like it’d be useful for dynamic sequence lengths instead of masking maybe?</p>

akuysal: <p>Masking will produce the following output ([ 1.,  2., -1.]) which is not the expected one.</p>
<p>import torch<br>
a = torch.Tensor([[1,2,0],[0,-1,0]])</p>
<p>mask = (a != 0)</p>
<p>b = torch.masked_select(a, mask)<br>
print(b)</p>

Sam_Lerman: <p>Oh, I was asking ptrblck about nested tensors.</p>

ptrblck: <p>I don’t know what exactly is supported using <code>NestedTensor</code>s as this class is still in its beta/experimental state. The Transfomer layers are providing some nested tensor functionality as seen <a href="https://github.com/pytorch/pytorch/blob/6faa6c68e8b76fb68f3a2b2783685102d0e87c00/aten/src/ATen/native/nested/NestedTensorTransformerFunctions.cpp">here</a> and the <code>fastpath</code> uses it too as described in <a href="https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/#taking-advantage-of-the-fastpath">this blog post</a>.</p>

MagedSaeed: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  for your reply. I was not aware such method exists. However, since it is beta, this is really suggesting that this approach is an anti-pattern approach. There should be better ways to handle my issue.</p>

ptrblck: <p>I don’t think there is any other way of handling tensors with different shapes besides the (beta) <code>NestedTensor</code> approach or just a plain Python <code>list</code> to store all objects.</p>

MagedSaeed: <p>No <a class="mention" href="/u/ptrblck">@ptrblck</a> , I mean my issue where I was thinking to utilize this nested tensor approach. Luckily, I found other ways to deal with it.</p>
<p>Again, Thanks Much,.</p>
What is happening under the hood when calling torch.stack?

samsja: <p>Hey all,</p>
<p>What is happening under the good when calling <code>torch.stack</code> ?</p>
<p>Is this equivalent to preallocating a tensor of the correct size and then to add the data one by one ?</p>
<p>I did some small experiment which would suggest yes (though I am not looking at memory metrics)</p>
<pre><code class="lang-python">import torch
N = 1000


data_list = [ torch.zeros(3, 224 ,224) for _ in range(N)]
%%timeit
data_stacked = torch.stack(data_list)
&gt;&gt;&gt; 79.9 ms ± 3.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

data_list = [ torch.zeros(3, 224 ,224) for _ in range(N)]
%%timeit
data_stacked = torch.empty((N, *data_list[0].shape))
for i, data in enumerate(data_list):
    data_stacked[i] = data

&gt;&gt;&gt; 90.5 ms ± 12.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
</code></pre>
Pruning in pytroch

Settur_Anantha_Sai_J: <p>I get that there is a pruning option in PyTorch. All it does it make some of the weights 0. My doubt is altho they are zero those edges still exist right? Won’t that affect the backpropagation algorithm?</p>

eqy: <p>I believe it depends on the implementation; if you are referring to functions in <code>torch.nn.utils.prune</code>, I would expect that it would behave as expected in the truly “sparse” setting of removing weights as a mask is used for computation rather than simply zeroing the weights.</p>
<p>Consider the following toy example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.zeros(10, requires_grad=True)
&gt;&gt;&gt; a.sum().backward()
&gt;&gt;&gt; a.grad
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
&gt;&gt;&gt; a2 = torch.zeros(10, requires_grad=True)
&gt;&gt;&gt; mask = torch.round(torch.rand(10, requires_grad=False))
&gt;&gt;&gt; mask
tensor([1., 0., 0., 0., 1., 1., 0., 1., 1., 0.])
&gt;&gt;&gt; (a2*mask).sum().backward()
&gt;&gt;&gt; a2.grad
tensor([1., 0., 0., 0., 1., 1., 0., 1., 1., 0.])
&gt;&gt;&gt;
</code></pre>
RuntimeError: Not compiled with video_reader support, to enable video_reader support, please install ffmpeg (version 4.2 is currently supported) and build torchvision from source

Pratyush: <p>Hi, I was trying to read a video file using <a href="http://torchvision.io" rel="noopener nofollow ugc">torchvision.io</a>, i have followed what’s given in the docs and i get the following error-<br>
CODE ----<br>
reader=torchvision.io.VideoReader(video_path,‘video’)<br>
reader.seek(2.0)<br>
frame=next(reader)<br>
ERROR-<br>
RuntimeError: Not compiled with video_reader support, to enable video_reader support, please install ffmpeg (version 4.2 is currently supported) and build torchvision from source.</p>
<p>Can anyone tell how to debug this in colab?</p>

ptrblck: <p>As the error message explains the pre-built binaries do not seem to support the video reader, so you would need to isntall the <code>ffmpeg</code> dependency and build <code>torchvision</code> from source.</p>
Gaussian filter for images

sank: <p>How can i implement a gaussian filter on a image tensor after the last convolutional layer as a post processing step?</p>

ptrblck: <p>You can create a <code>nn.Conv2d(..., bias=False)</code> layer and set the weights to the gaussian weights with:</p>
<pre><code class="lang-python">conv = nn.Conv2d(..., bias=False)
with torch.no_grad():
    conv.weight = gaussian_weights
</code></pre>
<p>Then just apply it on your tensor.</p>

Dimitris_Lolis: <p>Hello,</p>
<p>I am trying to do as you suggest but i’m kinda stuck and here is a simple example.</p>
<pre><code class="lang-auto">import torch
from torch import nn
from scipy.ndimage import gaussian_filter
import numpy as np


def g_difference(image, kernel, sigma1=3, sigma2=5):
    channels_in, channels_out = (image.shape[1], image.shape[1])
    print(channels_out)
    diffs = gaussian_filter(image.cpu().detach().numpy(), sigma=sigma2) - gaussian_filter(image.cpu().detach().numpy() , sigma=sigma1)
    conv2d = nn.Conv2d(channels_in, channels_out, kernel_size=kernel, bias=False)
    with torch.no_grad():
        conv2d.weight = torch.nn.Parameter(torch.FloatTensor(diffs).cuda())
    return conv2d


x = torch.randn([1, 1, 1600, 500])

output = g_difference(x, 1)

print(output(x.cuda()).shape) #  the output is [1,1,1,1]
</code></pre>

micklexqg: <p>thanks for your solution. i wonder  how to set the gaussian sigma and mean in nn.Conv2d or F.conv2d ?</p>

Blupon: <p>you need to set the weights either manually or using another function, for example from PIL or from scipy, see the code samples here <a href="https://discuss.pytorch.org/t/is-there-anyway-to-do-gaussian-filtering-for-an-image-2d-3d-in-pytorch/12351/19" class="inline-onebox">Is there anyway to do gaussian filtering for an image(2D,3D) in pytorch? - #19 by Jeffrey_Danowitz</a></p>
Setting different weight-decay values for parameters within one FSDP unit

orchidmajumder: <p>Hi, I wanted to confirm my understanding regarding the current behavior of PyTorch FSDP (native) when it comes to setting separate optimization parameters for different layers/modules within an FSDP wrapped module.</p>
<p>As a concrete example, I am following <a href="https://pytorch.org/blog/efficient-large-scale-training-with-pytorch/" rel="noopener nofollow ugc">this</a> tutorial of running a HuggingFace T5 model with FSDP and I use the <code>transformer_auto_wrap_policy </code> with <code>T5Block</code> being the <code>transformer_layer_cls </code> as suggested here. In this case, if I understand correctly, it is not possible to set weight-decay corresponding to the parameters of the LayerNorm layers present within  each <code>T5Block</code> to a value different from the weight-decay that’s passed to the optimizer? In particular, I’d like to have the weight-decay value to be 0.0 for these parameters following standard practice. Without FSDP, I can filter out such parameters by name or type and send two separate parameter groups to the Optimizer. With FSDP, each <code>T5Block</code> is presented as a flat parameter which prevents us from doing so. While using FSDP, I can potentially wrap each individual LayerNorm to be its own FSDP unit but it will require modifying the <code>T5Block</code> module and also will create a much higher number of FSDP units, thereby reducing compute/communication efficiency.</p>
<p>I have looked into this FSDP method called <code>summon_full_params</code> but I am not sure if I can create such parameter groups using the unflattened parameters returned by this function and pass it to the torch.optim.AdamW optimizer. If not, then is there any other way to achieve this behavior with FSDP at this moment?</p>

rvarm1: <p>Hi,</p>
<p>You should be able to enable this behavior with <code>use_original_parameters=True</code> which is required to be set when multiple parameter groups are used.</p>

orchidmajumder: <p>Great, thanks a lot - it works with PyTorch nightly.</p>
Model paralleism training on two node

yuri123: <p>Hi,<br>
Is it possible to train any model with model parallelism on two machine?<br>
I understand If I want to do model parallelism on multi machine with DDP, I need to use RPC, right?<br>
I want to use DDP so I read this tutorial. but I think it requires four machine.</p>
<p><a href="https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html#" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html#</a></p>
<p>Thank you.</p>

rvarm1: <p>Hi,</p>
<p>The above tutorial (and general Distributed APIs) can work on an arbitrary no. of machines.</p>
Expand() memory savings

kelbeda: <p>Hi,<br>
I am fairly new to PyTorch so sorry for a newbie question.<br>
How does PyTorch handles calling <code>.to(device)</code> when the object this methods is called on is created by calling <code>expand(...)</code> on a tensor? Does it create an underlying (small/minimal) tensor on a device and then a view or copies the view as it was a tensor, resulting in allocating much more device’s memory?<br>
If the former is the case, does it copy just the tensor data in the given view or the whole underlying tensor the view points to?<br>
If the former is the case, it would make sense to use expand after the tensor is copued to device right?</p>

huahuanZ: <pre data-code-wrap="python3"><code class="lang-plaintext">import torch
x = torch.randn(3,2,1)

x.stride()
# (2, 1, 1)

y = x.expand(-1, -1, 4)
y.stride()
# (2, 1, 0)

x_gpu = x.to(0)
x_gpu.stride()
# (2, 1, 1)

y_gpu = y.to(0)
y_gpu.stride()
# (8, 4, 1)
</code></pre>
<p>I guess when calling <code>to()</code> method, torch just create a contiguous tensor on the new device, which would probably consuming more memory than the original view.</p>
Mean and Sum reduction of the same loss does not produce vectors with angle 0

M4ximal: <p>Hi,</p>
<p>So I have two batches of 3 images in (3x256x256) and want to calculate the gradient wrt image-wise l1 distances. The first batch f_x is actually the output of some larger UNet wrt to an input x and I want to backpropagate through it to get the gradient of the l1 distance wrt x.</p>
<p>So in code:</p>
<pre><code class="lang-auto">%x and f_x have shape (bs, 3, 256, 256)
f_x = f(x) 
d = torch.nn.functional.l1_loss(f_x , init_image, reduction='sum')
grad = torch.autograd.grad(d, x, retain_graph=True)[0]
</code></pre>
<p>Importantly, my gradient is normalized before I do anything with it so I do not care about scale but only orientation. I noticed some weird behaviours with the gradient so instead, I tried to do:</p>
<pre><code class="lang-auto">%x and f_x have shape (bs, 3, 256, 256)
f_x = f(x) 
d = torch.nn.functional.l1_loss(f_x , init_image, reduction='mean')
grad = torch.autograd.grad(d, x, retain_graph=True)[0]
</code></pre>
<p>Like expected the value of d in the first case is equal to the value in the second case times bs * 3 * 256 * 256. So I expected the cosine similarity between the individual gradients in the two grad matrices to be 1, after all differentiation is linear. So I was actually very surprised when the cosine turned out to be approximately 0.6. Like I said, I do not need scaling information so working with mean is fine for me, however I would like to understand which of the two calculations I can trust more and what causes this issue.</p>
<p>Thank’s for your help</p>

ptrblck: <p>It seems I get the expected results using your pseudo-code snippet:</p>
<pre><code class="lang-python">bs = 2
x = torch.randn(bs, 3, 256, 256, requires_grad=True)
f_x = x * 2
init_image = torch.randn_like(x)

d = torch.nn.functional.l1_loss(f_x , init_image, reduction='sum')
grad1 = torch.autograd.grad(d, x, retain_graph=True)[0]


d = torch.nn.functional.l1_loss(f_x , init_image, reduction='mean')
grad2 = torch.autograd.grad(d, x, retain_graph=True)[0]

print(((grad1 / x.nelement()) - grad2).abs().max())
# tensor(0.)

out = nn.CosineSimilarity()(grad1, grad2)
torch.allclose(out, torch.ones_like(out))
# True
</code></pre>
Can I use same pooling layer?

laro: <p>Pooling layers dosn’t have paramaters to learns.</p>
<p>So if I build the following network:</p>
<pre><code class="lang-auto">import torch
import torch.nn          as nn
import torchinfo


class PoolModel(nn.Module):
    
    def __init__(self):
        super().__init__()

        self.GRU1      = nn.GRU(input_size = 2, hidden_size = 32, num_layers = 2, batch_first=True)
        self.Pooling1  = nn.AvgPool1d(kernel_size = 2, stride = 2)
        self.GRU2      = nn.GRU(input_size = 2, hidden_size = 16, num_layers = 2, batch_first=True)
        
        # do I need the second pooling layer or can I use Pooling1 ?
        # self.Pooling2   = nn.AvgPool1d(kernel_size = 2, stride = 2) 
        

    def forward(self, X):
        
        output, h = self.GRU1(X)
        output    = self.Pooling1(output)
        output, h = self.GRU2(X)
        
        output    = self.Pooling1(output)
        # or:
        #output    = self.Pooling2(output)
        
        return output

torchinfo.summary(PoolModel(), (512, 20, 2))  

==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
PoolModel                                [512, 20, 8]              --
├─GRU: 1-1                               [512, 20, 32]             9,792
├─AvgPool1d: 1-2                         [512, 20, 16]             --
├─GRU: 1-3                               [512, 20, 16]             2,592
├─AvgPool1d: 1-4                         [512, 20, 8]              --
==========================================================================================
Total params: 12,384
Trainable params: 12,384
Non-trainable params: 0
Total mult-adds (M): 126.81
==========================================================================================
Input size (MB): 0.08
Forward/backward pass size (MB): 3.93
Params size (MB): 0.05
Estimated Total Size (MB): 4.06
==========================================================================================
</code></pre>
<p>Can I use the same pooling layer (nn.AvgPool1d) twisc or do I need to create a new one (nn.AvgPool2d) ?</p>

ptrblck: <p>You can reuse the same layer and don’t need to initialize a new object for each call.</p>
Convert scipy csr matrix to torch sparse tensor

DanielC: <p>Hi there,<br>
I got a problem when I’m trying to convert a SciPy sparse matrix to torch form. Here is my code:</p>
<p>when I execute the following code:</p>
<pre><code class="lang-auto">w_csr = ss.load_npz(f'../data/w_csr.npz')
w_csr_tensor = torch.sparse_csr_tensor(torch.from_numpy(w_csr.indptr),
                                       torch.from_numpy(w_csr.indices),
                                       torch.from_numpy(w_csr.data))
</code></pre>
<p>where <code>w_csr</code> is a <code>SciPy</code> sparse matrix with shape <code>6400 x 6400</code>, everything work well. However, I need to manipulate  the <code>w_csr</code> by doing a matrix multiplication:</p>
<pre><code class="lang-auto">import scipy.sparse as ss
def re_order_w(w_csr):
    n_sample = w_csr.shape[0]
    I = ss.eye(n_sample, format='coo')
    # for debug, I didn't apply any change on the eye matrix
    w = I @ w_csr
    return w
</code></pre>
<p>Then when I try to create the sparse tensor by the same way:</p>
<pre><code class="lang-auto">w = re_order_w(w_csr)
w_csr_tensor = torch.sparse_csr_tensor(torch.from_numpy(w.indptr),
                                       torch.from_numpy(w.indices),
                                       torch.from_numpy(w.data))
</code></pre>
<p>The program return following <code>RunTimeError</code>:</p>
<pre><code class="lang-auto">RuntimeError: `col_indices[..., crow_indices[..., i - 1]:crow_indices[..., i]] for all i = 1, ..., nrows are sorted and distinct along the last dimension values` is not satisfied.
</code></pre>
<p>Which is weird, since I basically did nothing on the <code>Scipy</code> sparse matrix. Does anyone can explain the reason?</p>
<p>Appreciate any reply.</p>
Copy weights of some layers

zoythum: <p>I have two models that are built with the same structure except for a single layer placed in the middle of the second net which is not present in the first one, like this:</p>
<blockquote>
<p>Model1:<br>
layer1<br>
layer2<br>
layer3</p>
</blockquote>
<blockquote>
<p>Model2:<br>
layer1<br>
CustomLayer<br>
layer2<br>
layer3</p>
</blockquote>
<p>I trained the first model and stored the weights in a file, I would now like to initialise the second model with these weights and perform some predictions with it.<br>
To do so I instantiate the model and pass some parameters that are required by the layer in the middle but then I don’t know how to manually copy the weights from the first model into the second one.<br>
I don’t think that I can use the usual load_state_dict function since the two models have different structures but I would like to do so while ignoring the single different layer.</p>
<p>Does anyone know if there is a way to manually copy the weights? In keras I would do something like</p>
<pre><code class="lang-auto">for layer_name in model_1:
     model_2.get_layer(layer_name).set_weights(model_1.get_layer(layer_name).get_weights())
</code></pre>
<p>Does PyTorch have something similar?</p>
<p>EDIT:<br>
It was easier than expected, the solution seems to be the following</p>
<pre><code class="lang-auto">with torch.no_grad():
   model2.layer_name.weight.copy_(model1.layer_name.weight)
   model2.layer_name.bias.copy_(model1.layer_name.bias)
</code></pre>
<p>This process must be applied to each layer.</p>

ptrblck: <p>Your approach looks correct, but note that you could also use <code>.load_state_dict()</code> on each submodule e.g. via:</p>
<pre><code class="lang-python">model2.layer_name.load_state_dict(model1.layer_name.state_dict())
</code></pre>
<p>in case you would prefer this approach.</p>
[typing] why is import aliasing preferred to __all__ to explicitly export module variables?

ringohoffman: <p>Why is import aliasing preferred to <code>__all__</code> to explicitly export module variables? What is the reason for this preference? Is this documented anywhere?</p>
<p><a href="https://github.com/pytorch/pytorch/pull/80955/files" rel="noopener nofollow ugc">nn.Parameter, nn.DataParallel</a><br>
<a href="https://github.com/pytorch/pytorch/pull/87619/files" rel="noopener nofollow ugc">torch.testing.assert_close</a></p>

ptrblck: <p>Based on <a href="https://github.com/microsoft/pylance-release/issues/2953">this linked issue</a> it seems <code>pyright</code> complained about this import and claimed <code>nn.Parameter</code> wasn’t exported.<br>
<a href="https://github.com/microsoft/pylance-release/issues/2953#issuecomment-1168408943">This post</a> describes the issue in more detail.<br>
<code>pyright</code> also describes its expectations <a href="https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#library-interface">here</a>.</p>

ringohoffman: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, from what you linked I suppose <a href="https://github.com/microsoft/pylance-release/issues/2953#issuecomment-1168956296" rel="noopener nofollow ugc">this is <em><strong>a</strong></em> reason</a>:</p>
<blockquote>
<p>The <code>__all__</code> variable designates “symbols that should be imported by a wildcard import”, so it often overlaps with the notion of “symbols that are meant to be publicly visible from a module”, but there are cases where the former is a subset of the latter.</p>
</blockquote>
<p>I still wonder if there is an official PyTorch position on this… like if there is some documented reason we are not wanting to expose these variables thru <code>import *</code>.</p>

ptrblck: <p>The corresponding PyTorch issue is <a href="https://github.com/pytorch/pytorch/issues/80433">here</a> describing the other linked issues raised by <code>pyright</code>. If I interpret <a class="mention" href="/u/alband">@albanD</a> 's response correctly there were no arguments <em>against</em> fixing the <code>pyright</code> complains.</p>

albanD: <p>Another data point is the public API definition that we have: <a href="https://github.com/pytorch/pytorch/wiki/Public-API-definition-and-documentation" class="inline-onebox">Public API definition and documentation · pytorch/pytorch Wiki · GitHub</a></p>
<p>But for these, yes we’re happy to fix them!</p>
What can I do to improve my model based on the loss curves I obtained?

Maryam_Khaliji: <p>I have designed a Neural Network model and plotted the loss curves on two different datasets. With more iterations, the gap between validation and train loss grows, but the AUC score of validation increases. Is this model acceptable? If not, how can I improve it?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f0b097d4bccb07c5339d212eb4542f4a7eb8088.png" data-download-href="https://discuss.pytorch.org/uploads/default/9f0b097d4bccb07c5339d212eb4542f4a7eb8088" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f0b097d4bccb07c5339d212eb4542f4a7eb8088_2_690x184.png" alt="image" data-base62-sha1="mGXz0myC8caFFiIoNkZvLlVU6VO" width="690" height="184" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f0b097d4bccb07c5339d212eb4542f4a7eb8088_2_690x184.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f0b097d4bccb07c5339d212eb4542f4a7eb8088_2_1035x276.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f0b097d4bccb07c5339d212eb4542f4a7eb8088.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1090×292 30.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Custom multiprocessing data generation

ramazan: <p>Hi,</p>
<p>I’m trying to figure out how to create a custom dataloader with a following capabilities:</p>
<ol>
<li>It should take a list of datasets and a list of number of workers for each dataset</li>
<li>When generating a batch, it should perform parallelized data generation using specific number of workers for each dataset.</li>
<li>It should create child processes (workers) just <strong>once</strong> and re-use them all along.</li>
<li>It should support samplers like a usual Dataloader.</li>
</ol>
<p>Motivation:<br>
Since I have several dataset classes with different generation types: one is fast – just loading images from disk, the other one is slower – uses GAN on GPU to generate samples, I need to dedicate different number of workers for each dataset. And since second dataset have heavy GAN, we want to spawn workers just once.</p>
<p>For example, suppose we have:</p>
<pre><code class="lang-auto">datasets = [FileDataset(), GanDataset()]
n_workers = [2, 4]
custom_dl = CustomDataloader(datasets, n_workers)
</code></pre>
<p>Every time next(iter(custom_dl)) called, dataloader should send async requests for datasets to start generating and wait until datasets are done.</p>
<p>So, what is a good way to write a dataloader with such logic?</p>
Create mask tensor from lengths without for loop

papyrus: <p>Hi everyone,</p>
<p>I would like to create a mask tensor filled with ones where values should be masked, that masks a part of the input sequences depending on their lengths, used with a Transformer architecture. <code>lengths</code> is a 1D tensor containing the length of all input sequences in a batch. The code works but I would like to do it without a for loop. Any ideas?</p>
<p>Thanks a lot</p>
<pre><code class="lang-auto">torch::Tensor create_mask(torch::Tensor &amp;lengths,
                                            const int percentage)
{
  int max_len = torch::max(lengths).item&lt;int&gt;();
  auto mask_src = torch::zeros({lengths.size(0), max_len});

  for(int i = 0; i &lt; lengths.size(0); i++)
  {
    int current_length = lengths.index({i}).item&lt;int&gt;();

    int nb_masked = current_length * percentage / 100;

    int start_index =
        torch::randint(0, current_length - nb_masked, {1}).item&lt;int&gt;();

    mask_src.index_put_({i, Slice(start_index, start_index + nb_masked)}, 1);
  }

  return mask_src;
}
</code></pre>
How do I encrypt the libtorch model?

RogierBlog: <p>I want to encrypt my libtorch model when it is commercially available. What should I do? Does anyone can help me?</p>

mhbassel: <p>I think you can either encrypt the model file as it is and then add a decryption functionality by modifying the source code.<br>
Or you can use <a href="https://github.com/facebookresearch/CrypTen" rel="noopener nofollow ugc">CrypTen</a>.</p>
<p>It really depends on your use case and the way you make it commercially available.</p>
<p>I am only aware of these two approaches since I am working on a similar project. If you have another ideas feel free to share them.</p>
DLL backend_with_compiler initialization routine failed

luislo1970: <p>Hi,</p>
<p>I tackle with a lot of difficulties to compile Pytorch  libraries on my Windows 10, it is a PC with a NVIDIA QUADRO K4200 (with compute capability. 3.0). The CUDA driver on the graphic board is CUDA 11.4.94 but the runtime library library is 10.2.89. I have installed VS 2019 with MSVC v140-VS 2015 c++ build tools (v14.00), MSVC v142-VS 2019 (v14.27) and (v14.28-16.8), also I installed MSVC v140 - VS 2015 C++ build tools (v14.00) and last Windows 10 SDK. Honestly, I don’t know if any of these VS libraries are redundant or not for the compilation of the pytorch. After 6 hours of compilation (set CMAKE_GENERATOR=Visual Studio 15 2017) I installed pytorch library on python and I see follwing error:</p>
<p>c:\Users\PC\python</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “C:\Users\PC\anaconda3\envs\rnnenv\lib\site-packages\torch_<em>init</em>_.py”, line 125, in <br>
raise err<br>
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed. Error loading “C:\Users\PC\anaconda3\envs\rnnenv\lib\site-packages\torch\lib\backend_with_compiler.dll” or one of its dependencies.</p>
</blockquote>
</blockquote>
</blockquote>
<p>Other print-out is the following:</p>
<blockquote>
<blockquote>
<blockquote>
<p>import ctypes<br>
ctype.CDLL(r"C:\Users\PC\vcomp140.dll")<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
NameError: name ‘ctype’ is not defined<br>
ctypes.CDLL(r"C:\Users\PC\vcomp140.dll")<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “C:\Users\PC\anaconda3\envs\rnnenv\lib\ctypes_<em>init</em>_.py”, line 382, in <strong>init</strong><br>
self._handle = _dlopen(self._name, mode)<br>
FileNotFoundError: Could not find module ‘C:\Users\PC\vcomp140.dll’ (or one of its dependencies). Try using the full path with constructor syntax.<br>
ctypes.CDLL(r"C:\Users\PC\anaconda3\vcomp140.dll")<br>
&lt;CDLL ‘C:\Users\PC\anaconda3\vcomp140.dll’, handle 7ff8a7ec0000 at 0x26561a54eb0&gt;</p>
</blockquote>
</blockquote>
</blockquote>
<p>Please, any idea before starting again all the installation process?</p>
<p>Br/Luis</p>

jackchen: <p>Hi, I bumped into the same error while building PyTorch from source on my Windows (Installation on Mac was successful btw). Did you figure out how to solve it? Thank you!</p>

Chenoille: <p>Hello, I am facing the same issue! What is the solution to that?<br>
Thank you</p>

Kareem_Shaik: <p>Any sugestion to fix for this error “OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed. Error loading “C:\Users\PC\anaconda3\envs\rnnenv\lib\site-packages\torch\lib\backend_with_compiler.dll” or one of its dependencies.”</p>

Chenoille: <p><a class="mention" href="/u/kareem_shaik">@Kareem_Shaik</a>  You should check this <a href="https://github.com/peterjc123/pytorch_dll_load_smoketests" rel="noopener nofollow ugc">page</a>.</p>
Update only a middle layer of a neural network

iman.mirzadeh: <p>Hi,</p>
<p>Say I have a network, like the figure below.<br>
During the training, I want to freeze layer 1 and 3 and only update weights on layer 2. Is it possible?</p>
<p>I don’t know how to do because according to the Autograd documentation, if at least one the inputs of a node in the computation requires grad, then that node requires grad in the graph (which makes sense!).</p>
<p>But I need to replace only a middle layer of the neural network and don’t touch the rest.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9d937dd50f7030804cdbdc9fcab105cd48bffe28.png" alt="Untitled%20Diagram%20(4)" data-base62-sha1="mtYXDYaJpUiurvDNI7sm59J6FIs" width="543" height="103"></p>
<p>Thanks.</p>

albanD: <p>Hi,</p>
<p>You can go through every parameters that you don’t want to update and set them not to require gradients:</p>
<pre><code class="lang-auto">for p in layer2.parameters():
    p.requires_grad = False
</code></pre>
<p>This means that no gradients will be computed for these parameters.<br>
As further optimization, you can give to your optimizer only the layer you want to optimize such that it won’t check every other parameters that won’t have gradients:</p>
<pre><code class="lang-auto">my_opt = torch.optim.SGD(model.layer2.parameters(), other_args)
</code></pre>

iman.mirzadeh: <p>Thanks for your answer.<br>
But, in order to calculate layer2 gradients, I think we need to calculate layer3 gradients. If I set layer3 grads to False, how it would be possible?</p>

albanD: <p>You will set the layer3’s parameters gradients to False. It means that they won’t ask for gradients. But the input to layer3, coming from layer2 will have <code>requires_grad=True</code> because these gradients are needed for layer2’s parameters. And so the output of layer3 will require gradients but during the backward pass, only the gradients wrt to input of layer3 will be computed, not wrt its parameters.</p>
<p>If you were looking at layer2, its input does  not require gradients, but its parameters does. And so its output will as well. During the backward pass though, only gradients wrt the parameters will be computed and not wrt the input.</p>

iman.mirzadeh: <p>Another thing regarding this question:<br>
What if I have batchnorm in layer3 ?<br>
Should I manually call .eval() for layer3?</p>

albanD: <p><code>.eval()</code> has nothing to do with gradient computation but with how layer behaves. Batcnorm will use saved statistics in eval mode, dropout will be an identity etc.</p>

jia_lee: <p>Thus, if we want to freeze some layers (these layers include Batchnorms), we should set this part of model to .eval() for safe. Am I right?</p>
RuntimeError: adaptive_max_pool2d

Federico_Digiacomo: <p>Hi all, I’m using MONAI to build a U-Net for a segmentation task. I want to crop the input images with “CropForegroundd” as follows:</p>
<pre><code class="lang-auto">    monai_load = [
      LoadImaged(keys=["image","segmentation"],image_only=False,reader=PILReader()),
      EnsureTyped(keys=["image", "segmentation"], data_type="numpy"),
      AddChanneld(keys=["segmentation","image"]),
      RepeatChanneld(keys=["image"],repeats=3),
      AsChannelFirstd(keys=["image"], channel_dim = 0), 
      ]
      
    monai_transforms =[
      AsDiscreted(keys=["segmentation"],threshold=0.5),
      CropForegroundd(keys=["image","segmentation"], source_key="image", margin=20),
      Resized(keys=["image","segmentation"],spatial_size=(240,240)),
      ToTensord(keys=["image","segmentation"]),
      ]
      
    train_transforms = Compose(monai_load  + monai_transforms)
    train_ds = IterableDataset(data = train_data, transform = train_transforms)
    train_loader = DataLoader(dataset = train_ds, batch_size = batch_size, num_workers = 0, pin_memory = True)
</code></pre>
<p>During training, it stops everytime at 7th iteration (no matter starting from 0) and returns the following error:</p>
<pre><code class="lang-auto">RuntimeError: adaptive_avg_pool2d(): Expected input to have non-zero size for non-batch dimensions, but input has sizes [1, 3, 0, 0] with dimension 2 being empty
</code></pre>
<p>My input images are MRI images (240x240).</p>
<p>Thanks in advance for your feedback</p>
<p>Federico</p>
How to iteratively remove non-optimal directories during HPO to save space

Slowat_Kela: <p>Hello,</p>
<p>Could I ask,</p>
<p>When I run pytorch lightning with HPO it creates a lot of directories like this:</p>
<pre><code class="lang-auto">'train_fn_ff307114_1350_activation_function=nn_ReLU_inplace_True,c_hidden=64,dp_rate_linear=0.6747,learning_rate=0.0002,num_layers=_2022-12-21_17-51-28'
'train_fn_ff4d9f16_599_activation_function=nn_LeakyReLU_inplace_True,c_hidden=2056,dp_rate_linear=0.5871,learning_rate=0.0035,num_l_2022-12-21_13-26-00'
'train_fn_ffa6c2a4_892_activation_function=nn_ReLU_inplace_True,c_hidden=512,dp_rate_linear=0.6173,learning_rate=0.0116,num_layers=_2022-12-21_15-06-17'
'train_fn_ffe7cd72_1684_activation_function=nn_Tanh,c_hidden=2056,dp_rate_linear=0.6520,learning_rate=0.0032,num_layers=4,optimizer_2022-12-21_19-45-25'
</code></pre>
<p>Each dir containing a checkpoint, log etc.</p>
<p>Is there a way to optimize this for space? For example, if a directory is found to have a better val_acc (my metric of interest for HPO) then we don’t need to keep the less better val_acc HPO attempt, so can I delete these as I go through the HPO process?</p>
<p>My code for running HPO is:</p>
<pre><code class="lang-auto">
def run_ray(metric='val_acc', mode='max',num_samples=4000,config_dict={}, checkpoint_file_name="full_run_2_hemolysis_ray_ckpt",config_file='full_run_2_hemolysis/hemolysis/best_config.txt',local_dir='full_run_2_hemolysis/hemolysis/runs/'):

    hyperopt_search = HyperOptSearch(metric=metric, mode=mode)

    #change from gpu {"gpu": 1}
    tuner = tune.Tuner(tune.with_resources(train_fn,{"gpu": 1}), tune_config=tune.TuneConfig(num_samples=num_samples,search_alg=hyperopt_search),param_space=config_dict,run_config= RunConfig(local_dir=local_dir))
    results = tuner.fit()   
    best_result = results.get_best_result(metric=metric, mode=mode) 

    config_file = open(config_file, 'a')
    config_file.write(str(best_result.config) + '\n')

    best_checkpoint = best_result.checkpoint
    path = os.path.join(str(best_checkpoint.to_directory()), checkpoint_file_name)
    print(path)

    model = GraphLevelGNN.load_from_checkpoint(path)
    config_file.write(str(best_result.log_dir))
    config_file.close()

    return best_result.log_dir,model
</code></pre>
<p>I’m not sure if I’ve given enough information, I just didn’t want to overload with irrelevent details. Or also maybe this is more a question about ray, let me know if it’s not appropriate.</p>
IndexError: Target 25 is out of bounds

ammar_siddiqui: <p>I have a custom custom company dataset which has 14 features and 1 output label having 6 classes [9,12,15,18,21]. I have built a linear model using following definition:</p>
<pre><code class="lang-auto">class HourPredictor(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(in_features=14, out_features=64)
        self.fc2 = nn.Linear(in_features=64, out_features=32)
        self.output = nn.Linear(in_features=32, out_features=5)
    
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.output(x)
        return x
</code></pre>
<p>But when i try to train the model it is giving me this index error:</p>
<p>IndexError Traceback (most recent call last) File :9</p>
<p>File ~/.envs/.vas/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs) 1190 # If we don’t have any hooks, we want to skip the rest of the logic in 1191 # this function, and just call forward. 1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks 1193 or _global_forward_hooks or _global_forward_pre_hooks): → 1194 return forward_call(*input, **kwargs) 1195 # Do not call functions when jit is used 1196 full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>File ~/.envs/.vas/lib/python3.10/site-packages/torch/nn/modules/loss.py:1174, in CrossEntropyLoss.forward(self, input, target) 1173 def forward(self, input: Tensor, target: Tensor) → Tensor: → 1174 return F.cross_entropy(input, target, weight=self.weight, 1175 ignore_index=self.ignore_index, reduction=self.reduction, 1176 label_smoothing=self.label_smoothing)</p>
<p>File ~/.envs/.vas/lib/python3.10/site-packages/torch/nn/functional.py:3026, in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing) 3024 if size_average is not None or reduce is not None: 3025 reduction = _Reduction.legacy_get_string(size_average, reduce) → 3026 return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)</p>
<p>IndexError: Target 25 is out of bounds.</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects target tensors to contain class indices in the range <code>[0, nb_classes-1]</code> while your target apparently contains an index of <code>25</code>, which is wrong.<br>
Map the targets to <code>[0, nb_classes-1]</code> and it should work.<br>
Also, you are explaining to deal with 6 classes but the last output layer uses <code>out_features=5</code> which also seems to be wrong.</p>

ammar_siddiqui: <p>It worked for me. Thank you.</p>
PyTorch, AMD GPU and Windows

leinmi: <p>Hello,</p>
<p>Is there any active development done to make PyTorch support AMD GPU’s on Windows?<br>
AI image generation is quite popular currently, but PyToch supports only Nvidia cards with CUDA.<br>
Any info if/when AMD GPU’s are supported?</p>
<p>Regards,<br>
Mika</p>