akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Local Inference of saved PyTorch models

username: sarthakpati
cooked: <p>I am interested in performing local (no server or cloud) inference of saved PyTorch models that I can “deploy” (for example, using PyInstaller) to machines that do not have any dependencies. Can someone point me in the correct direction for this?</p>
<p>My goal is to be able create a command-line executable that I can share with collaborators who do not have any idea about Python/programming/Docker/containers and would simply like to infer on the models I send (with the correct data, of course).</p>
<p>Requirements from end product:</p>
<ul>
<li>No “cloud” or remote processing - data privacy is a critical requirement and processing needs to be local on Windows and/or Linux machines using CPU (GPU would be a huge plus but not required)</li>
<li>No need for user to install anything (no Python/Docker engine/server/etc.) as this needs to be catered to a non-tech audience</li>
</ul>
<p>Please comment if further clarifications are needed.</p>
<p>Thanks in advance.</p>

username: marksaroufim
cooked: <p>Sounds like you just need to create some CLI tool which you can create with argparse. I like typer but that has a few more dependencies.</p>

username: mmursalin
cooked: <p>Hi <a class="mention" href="/u/sarthakpati">@sarthakpati</a>,<br>
I have a similar situation. Did you solve the issue? Could you please share the soloution?<br>
Thank you.</p>

username: sarthakpati
cooked: <p>Apologies for the late response. I unfortunately do not have a solution for this, yet.</p>
PyTorch LTS release schedule?

username: cdahms
cooked: <p>Is there a PyTorch LTS release schedule?</p>
<p>I see <a href="https://github.com/pytorch/pytorch/releases" class="inline-onebox" rel="noopener nofollow ugc">Releases · pytorch/pytorch · GitHub</a> the first LTS release (1.8.2) was released Aug 17, 2021 but <a href="https://pytorch.org/enterprise-support-program" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Enterprise | PyTorch</a> or anywhere else I can’t find any info on future LTS releases or a LTS release schedule.</p>

username: sarthakpati
cooked: <p>I would be very interested in finding this information as well.</p>
Torch sparse tensor and torch.where function

username: Max_Unhold
cooked: <p>Hello,</p>
<p>I would like to use the <code>torch.where</code> function on a sparse tensor.<br>
Is there any torch.sparse function implemented for that (or will be)?<br>
In the other case: Is there a work-around?</p>
<p>Example</p>
<pre><code class="lang-auto">i = torch.LongTensor([[0, 2], [1, 0], [1, 2]])
v = torch.FloatTensor([3,      4,      0    ])
x = torch.sparse.FloatTensor(i.t(), v, torch.Size([3,3]))

# Working
torch.where(x.to_dense()&gt;0,1,0)
# Not working
torch.where(x&gt;0,1,0)
</code></pre>
What is the synchronise function for mac mps?

username: Bin_Ye
cooked: <p>I want to let cpu to wait until the mac Neural engine tasks to finish, what function should I use? I know for CUDA, I can use torch.cuda.synchronize(), but what function is it for mps?</p>
<p>I have the following code:</p>
<pre><code class="lang-auto">import torch
if torch.has_mps:
    device = torch.device("mps")
else:
    device = torch.device("cpu")

print("using", device, "device")

import time

matrix_size = 32*512

x = torch.randn(matrix_size, matrix_size)
y = torch.randn(matrix_size, matrix_size)

print("***********cpu speed***************")
start = time.time()
result = torch.matmul(x,y)
print(time.time()-start)
print("verify device: ", result.device)


x_mps = x.to(device)
y_mps = y.to(device)
# torch.cuda.synchronize()


for i in range(3):
    print("***********mps speed***************")
    start = time.time()
    result_mps = torch.matmul(x_mps,y_mps)
    # torch.cuda.synchronize()
    print(time.time()-start)
    print("verify device: ", result_mps.device)
</code></pre>
Combining Conditional Masks

username: J_Johnson
cooked: <p>Suppose we have some random distribution of integer values between 1 and 3.</p>
<pre><code class="lang-auto">A=torch.round(torch.rand(10000)*2)+1
</code></pre>
<p>And we want two masks combined in the following order:</p>
<pre><code class="lang-auto">cond1=A!=2.0

B=A[cond1]

cond2=B[:-1]!=B[1:]

all_but_firstB=B[1:]

C=torch.cat([B[0].view(-1), all_but_firstB[cond2]])
</code></pre>
<p>How would I go about combining the two masks as one to apply to the original vector A? Such that:</p>
<pre><code class="lang-auto">A[one_mask]=C
</code></pre>

username: J_Johnson
cooked: <p>I was able to tackle this with using a vector range, applying the masks to that range, then applying that range as a mask to A. As follows:</p>
<pre><code class="lang-auto">data_size=10000
A=torch.round(torch.rand(data_size)*2)+1

b=torch.arange(data_size)

cond1=A!=2.0

b = b[cond1]

cond2 = A[cond1][:-1]!=A[cond1][1:]

all_but_firstb=b[1:]

b=torch.cat([b[0].view(-1), all_but_firstb[cond2]])

print(torch.all(A[b]==C))
</code></pre>
NVIDIA A100 80G very slow during training

username: Ch-rode
cooked: <p>I just bought an A100 80G NVIDIA GPU, but I am not able to use it since it is very slow. It seems to be 5x time slower than an A100 40G, when I do a bert model fine tuning.<br>
Those are my venv information and the nvdia-smi out:</p>
<pre><code class="lang-auto">PyTorch version: 1.12.0+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Rocky Linux release 8.6 (Green Obsidian) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10)
Clang version: Could not collect
CMake version: version 3.20.2
Libc version: glibc-2.28

Python version: 3.9.7 (default, Sep 16 2021, 13:09:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.18.0-372.9.1.el8.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA A100 80GB PCIe
Nvidia driver version: 515.48.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] torch==1.12.0+cu116
[pip3] torchaudio==0.12.0+cu116
[pip3] torchvision==0.13.0+cu116
[conda] blas                      1.0                         mkl  
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.20.3           py39hf144106_0  
[conda] numpy-base                1.20.3           py39h74d4b33_0  
[conda] numpydoc                  1.1.0              pyhd3eb1b0_1

</code></pre>
<pre><code class="lang-auto"> NVIDIA-SMI 515.48.07    Driver Version: 515.48.07    CUDA Version: 11.7  
</code></pre>
<p>Do you know by chance what can be the problem? In this configuration is impossible to use it.</p>
<p>Thank you !</p>

username: ptrblck
cooked: <p>We would need more information to be able to debug this issue as we are not seeing these slowdowns on 80GB vs. 40GB A100s internally on BERT.</p>

username: Ch-rode
cooked: <p>Sure, sorry. Which type of information I can give you?</p>

username: Ch-rode
cooked: <p>I solved using this torch version details in my env:</p>
<pre><code class="lang-auto">torch==1.12.0+cu116
torchaudio==0.12.0+cu116
torchvision==0.13.0+cu116
</code></pre>
How to make gradient flow back through torchvision.transforms

username: joohyunglee
cooked: <p>I would like to include transform mechanism within the loss function. However, it looks like that gradient won’t flow back through transforms. Is there a way to make gradient flow back through a set of <code>torchvision.transforms</code>? I will use <code>transforms.RandomResizedCrop()</code> and <code>transforms.RandomHorizontalFlip()</code>.</p>

username: ptrblck
cooked: <p><a href="https://github.com/kornia/kornia"><code>kornia</code></a> provides differentiable transformations, so you could check it.<br>
CC <a class="mention" href="/u/edgarriba">@edgarriba</a></p>

username: joohyunglee
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Thank you for the suggestion!!! I just wonder if <code>torchvision.transforms.xxx</code> are differentiable transformations since they inherit <code>torch.nn.Module</code>, e.g., <code>torchvision.transforms.RandomResizedCrop</code>.</p>

username: ptrblck
cooked: <p>Yes, I think if the transformation accepts tensors they might be differentiable. E.g. <code>RandomResizedCrop</code> should be a <code>crop</code> and <code>resize</code> operation, which should not detach the tensor from the computation graph.</p>
How to check gradients?

username: BobKim
cooked: <p>Hello!<br>
I want to know how to check the gradient value while training.<br>
Im training the ResNet-34 in CIFAR-10 (image classification task)<br>
I think that storing all gradients requires too much memory storage.<br>
So, i want to check the mean &amp; variance of the gradients at every epoch.<br>
How can i do this??<br>
(Attach the main script below)</p>
<hr>
<p>def main():<br>
parser = get_parser()<br>
args = parser.parse_args()<br>
train_loader, test_loader = build_dataset(args)<br>
device = ‘cuda’ if torch.cuda.is_available() else ‘cpu’<br>
if args.resume:<br>
ckpt = load_checkpoint(ckpt_name)<br>
start_epoch = ckpt[‘epoch’]</p>
<pre><code>    curve = os.path.join('curve', ckpt_name)
    curve = torch.load(curve)
    train_losses = curve['train_loss']
    test_accuracies = curve['test_acc']
else:
    ckpt = None
    start_epoch = -1
    train_losses = []
    test_accuracies = []

net = build_model(args, device, ckpt=ckpt)
criterion = nn.CrossEntropyLoss()

optimizer = Adam(args, net.parameters())
start_time = time.time()

for epoch in range(start_epoch + 1, args.total_epoch):
    start = time.time()
    train_loss, train_acc = train(net, epoch, device, train_loader, optimizer, criterion, args)
    test_loss, test_acc = test(net, epoch, device, test_loader, criterion)
    end = time.time()
    print('Time {}'.format(end-start))

    train_losses.append(train_loss)
    test_accuracies.append(test_acc)

end_time = time.time()
print('End Time: {}'.format(end_time - start_time))
</code></pre>

username: eqy
cooked: <p>You should be able to access the <code>.grad</code> attribute of your model tensors (e.g., via <code>net.parameters()</code>) and use that to compute the statistics you wish.</p>

username: BobKim
cooked: <p>Thanks! ill try this</p>
How to efficiently get the K-th largest values in parallel?

username: 111414
cooked: <p>I have a tensor <code>data</code> with a shape of <code>[b, c]</code>, and a tensor <code>K</code> with a shape of <code>[b]</code>. How can I efficiently get the K-th largest values in parallel? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<p>The following code shows what I need:</p>
<pre><code class="lang-python">import torch

b, c = 10, 6
data = torch.randn(b, c)
K = torch.randint(low=1, high=c, size=(b,))
result = []
for i in range(b):
    result.append(data[i].kthvalue(int(K[i])).values)
result = torch.tensor(result)

print(data)
print(K)
print(result)
</code></pre>
<p>Since the two APIs <code>topk</code> and <code>kthvalue</code> only require an integer <code>k</code> as the ranking parameter, I do not know how to implement the above parallel version efficiently.</p>
<hr>
<p><strong>P.S.</strong></p>
<p>I am using the above process in the <code>forward</code> method of my neural network. So an efficient implementation is really appealing for me.</p>
<p>And now I am using the above trivial implementation, which may become slow when the batch size <code>b</code> is large. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

username: eqy
cooked: <p>I think some version of <code>gather</code> can achieve this (note the off-by-one indexing change):</p>
<pre><code class="lang-auto">import torch
import time

b, c = 10, 6
data = torch.randn(b, c)
K = torch.randint(low=1, high=c, size=(b,))
result = []

t1 = time.time()
for i in range(b):
    result.append(data[i].kthvalue(int(K[i])).values)
result = torch.tensor(result)
t2 = time.time()

result2 = torch.gather(torch.sort(data)[0], 1, (K-1).unsqueeze(1)).squeeze()
t3 = time.time()

print(data)
print(K)
print(result)
print(result2)
print(t2 - t1, t3 - t2)
</code></pre>
<pre><code class="lang-auto">tensor([[-0.1341,  0.0766, -1.1398, -0.9177,  1.4444,  0.2596],
        [ 1.5937,  0.2464, -0.5563, -0.5010, -0.5085,  0.0087],
        [-1.2250, -0.9519, -0.1867,  0.1060, -1.1537, -0.5210],
        [-0.0450, -1.7052, -0.4110,  0.2150, -0.1462,  1.1294],
        [-1.4275, -0.0268, -1.6176,  0.4198,  0.4801,  1.3176],
        [ 1.0741, -0.6166,  0.3744, -1.0931, -0.1258, -0.0549],
        [-1.3071, -0.7423,  0.3884,  0.6894, -1.7652,  0.9175],
        [ 0.5899,  1.5174,  1.0743,  0.5756,  0.4595, -1.4915],
        [ 1.7524, -0.4349, -0.7347,  0.2961, -0.1927, -0.7609],
        [-1.1059, -0.0274, -1.3685, -0.3434, -0.8812,  0.3865]])
tensor([1, 5, 3, 4, 5, 4, 5, 5, 1, 1])
tensor([-1.1398,  0.2464, -0.9519, -0.0450,  0.4801, -0.0549,  0.6894,  1.0743,
        -0.7609, -1.3685])
tensor([-1.1398,  0.2464, -0.9519, -0.0450,  0.4801, -0.0549,  0.6894,  1.0743,
        -0.7609, -1.3685])
0.00020599365234375 0.00012087821960449219
</code></pre>
Replace layers in model by another and with extra parameters

username: deJQK
cooked: <p>Hi, I am trying to replace layers in a defined model with another type of layer, but with some extra parameters. An example is something like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

from torchvision.models import resnet18

class ModA(nn.Module):
  def __init__(self, layer):
    super(ModA, self).__init__()
    self.layer = layer
    self.paraA = nn.Parameter(torch.tensor(1.0))

  def forward(self, input):
    return self.layer(self.paramA * input)

net = resnet18(weights="IMAGENET1K_V2")
layer_names = []
layers = []
for n, m in net.named_modules():
  if isinstance(m, nn.Conv2d):
    layer_names.append(n)
    layers.append(ModA(m))

for n, m in zip(layer_names, layers):
  setattr(net, n, m)

net.to('cuda:0')
net = torch.nn.parallel.DistributedDataParallel(net, device_ids=[gpus])
</code></pre>
<p>It will report some error messages like <a href="https://github.com/pytorch/pytorch/issues/43259" rel="noopener nofollow ugc">this</a>.</p>
<p>If I do not have the extra parameter <code>self.paraA</code> it will work fine (but will print some message like <a href="https://discuss.pytorch.org/t/reducer-buckets-message-with-4-gpus/100096">this</a> if I still do this module change). It seems to me that the reason is although I changed the module manually in the model, I have not “inserted” the extra parameter <code>self.paraA</code> into <code>model.parameters</code>. I wonder what is the most appropriate practice to replace layers in a predefined model with custom modules. Thanks.</p>

username: deJQK
cooked: <p>The reason is replacing layers by <code>setattr</code> with layername does not work for <code>nn.Sequential</code>, so in the original code the <code>nn.Conv2d</code> layers inside <code>nn.Sequential</code> are not replaced correctly, and if print elements in <code>model.named_modules()</code> and <code>model.named_parameters()</code>, there will be an extra module with the same name for each <code>nn.Conv2d</code> in <code>nn.Sequential</code>, and the final model does not have the <code>nn.Conv2d</code> replaced. The extra parameters are not included in <code>nn.named_parameters()</code>, either.</p>
<p>One workaround solution, although not elegant, is as below:<br>
replace this part</p>
<pre><code class="lang-auto">for n, m in zip(layer_names, layers):
  setattr(net, n, m)
</code></pre>
<p>by</p>
<pre><code class="lang-auto">for n, m in zip(layer_names, layers):
  levels = n.split('.')
  if len(levels) &gt; 1:
    mod_ = model
    for l_idx in range(len(levels) - 1):
      if levels[l_idx].isdigit():
        mod_ = mod_[eval(levels[l_idx])]
      else:
        mod_ = getattr(mod_, levels[l_idx])
    setattr(mod_, levels[-1], m)
  else:
    setattr(model, n, m)
</code></pre>
Delete entry from sparse tensor

username: Max_Unhold
cooked: <p>Hello, I would like to be able to add and delete elements in an<br>
<a href="https://pytorch.org/docs/stable/sparse.html" rel="noopener nofollow ugc">pytorch sparse tensor</a></p>
<p>While I did not find a way to add an element by assigning it <code>s[0,1]  = 3</code>, I discovered that you can just add it by an addition:</p>
<pre><code class="lang-auto">i = [[0, 1],[2, 0]]
v = [3, 4]
a1 = torch.sparse_coo_tensor(i, v, (5, 5))
i = [[3],[3]]
v = [10]
a2 = torch.sparse_coo_tensor(i, v, (5, 5))
assigned_tensor = (a1 + a2).coalesce()
print("assigned tensor ",assigned_tensor)
</code></pre>
<p>I did not find a way to delete an element from a sparse tensor either, and if I try to subtract an element in the same way like</p>
<pre><code class="lang-auto">i = [[0, 1],[2, 0]]
v = [3, 4]
a1 = torch.sparse_coo_tensor(i, v, (5, 5))
i = [[0],[2]]
v = [3]
a2 = torch.sparse_coo_tensor(i, v, (5, 5))
added_tensor = (a1 - a2).coalesce()
print("deleted tensor ",added_tensor)
</code></pre>
<p>the element does not vanish but the value becomes zero.</p>
<p>How do I delete an entry from the sparse tensor?<br>
(Noticing <a href="https://github.com/pytorch/pytorch/issues/31742" rel="noopener nofollow ugc">this discussion</a> about deleting 0 values in a sparse tensor).</p>
Help understanding how this function works

username: BaruchG
cooked: <p>Hi all - Can someone please explain to me what this function does and what a valid input for it would be? <a href="https://github.com/Lightning-AI/lightning-bolts/blob/bcbbf6ab6c36430946dd8a416ddc7e697e8507fc/pl_bolts/metrics/aggregation.py#L7" class="inline-onebox" rel="noopener nofollow ugc">lightning-bolts/aggregation.py at bcbbf6ab6c36430946dd8a416ddc7e697e8507fc · Lightning-AI/lightning-bolts · GitHub</a> ?</p>
<pre><code class="lang-auto">def mean(res, key):
    # recursive mean for multilevel dicts
    return torch.stack([x[key] if isinstance(x, dict) else mean(x, key) for x in res]).mean()
</code></pre>
<p>I’m trying to test it out but it’s clearly not the same as torch.mean() and doesn’t seem to accept tensors natively.  I also couldn’t find information about it in the docs.  I’m struggling to understand how it’s actually working due to its recursive nature, so an explanation of what is going on would also be helpful. Thanks!</p>

username: BaruchG
cooked: <p>I got some help on the lightning channel and a valid input would be:</p>
<pre><code class="lang-auto">t = torch.tensor([[100., 100., 200., 200.]])
x = {"test":t}
mean([x], "test")
</code></pre>
Builtin training loop

username: kaoutar
cooked: <p>Hello,<br>
I know that pytorch doesn’t have a builtin fit method like keras, and one should implement it by himself, my question is why? i’ve read somewhere that its implementation in the core  would need lots of things, but what are those things?, if you can’t tell in details, plz just give a clear overview with some examples, i really need to understand these so called “obstacles”. Thank you</p>

username: ptrblck
cooked: <p>PyTorch supports various higher-level APIs which provide a <code>fit</code> or similar methods, such as Ignite, Lightning, etc.<br>
You can see the standard training loop e.g. <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#update-the-weights">here</a>:</p>
<pre><code class="lang-python"># in your training loop:
optimizer.zero_grad()   # zero the gradient buffers
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()    # Does the update
</code></pre>
<p>You can of course wrap it in another method if it’s more convenient for you or just use any higher-level API which would fit your use cases.</p>

username: kaoutar
cooked: <p>Thank you for your reply, but i think i am misunderstood here , i am not asking how to do it, instead, i am referring to the last reply in this <a href="https://discuss.pytorch.org/t/training-a-model-via-a-train-method/58567/5">thread</a>, please take a look to know what i mean</p>
Filter nan values based on two tensors

username: Luca_Pamparana
cooked: <p>I have 2 pytorch tensors say as follows:</p>
<pre><code class="lang-auto">x = torch.Tensor([float('nan'), 2.0, 5.0, 6.0])
y = torch.Tensor([float('nan'), float('nan'), 3.0, 4.0])
</code></pre>
<p>I want to filter both the arrays where entries that are NAN in either of the two arrays are filtered out i.e. in this case,</p>
<pre><code class="lang-auto">filtered_x = [5., 6.]
filtered_y = [3., 4.]
</code></pre>
<p>Just wondering how one can do it with <code>pytorch</code>. I am also using a rather old version of pytorch <code>1.0.1.post2</code></p>

username: ptrblck
cooked: <p><code>tensor.is_finite</code> should work:</p>
<pre><code class="lang-python">x = torch.Tensor([float('nan'), 2.0, 5.0, 6.0])
y = torch.Tensor([float('nan'), float('nan'), 3.0, 4.0])

idx = x.isfinite() &amp;  y.isfinite()
out = torch.stack((x[idx], y[idx]))
print(out)
# tensor([[5., 6.],
#         [3., 4.]])
</code></pre>
<p>I don’t know if this method is available in the old <code>1.0.1</code> version so you might need to update or write a method to check for finite values.</p>

username: Luca_Pamparana
cooked: <p>Thank you. I could just use <code>torch.isfinite()</code></p>
NaN in Pearson Correlation Coefficient loss function

username: talhaanwarch
cooked: <p>Hi, I am trying to use Pearson loss function instead of MSE.<br>
Here is the loss function</p>
<pre><code class="lang-auto">class Neg_Pearson_Loss(nn.Module):   
    #https://stackoverflow.com/a/19710598/11170350
    def __init__(self):
        super(Neg_Pearson_Loss,self).__init__()
        return
    def forward(self, X, Y):       
        assert not torch.any(torch.isnan(X))
        assert not torch.any(torch.isnan(Y))
        # Normalise X and Y
        X = X-X.mean(1)[:, None]
        Y = Y- Y.mean(1)[:, None]
        # Standardise X and Y
        X = (X/ X.std(1)[:, None])+1e-5
        Y =(Y/ Y.std(1)[:, None])+1e-5
        #multiply X and Y
        Z=(X*Y).mean(1)
        Z=1-Z.mean()
        return Z
</code></pre>
<p>But it gives nan loss.<br>
I have normalized the label and feature between 0 and 1. The same code work for mse/mae loss function but give nan on Pearson loss function.<br>
Any idea</p>

username: ptrblck
cooked: <p>You could be dividing by a zero <code>stddev</code> and might want to move the <code>1e-5</code> eps value into the denominator.</p>
CTCDecoder on GPU

username: patrickvonplaten
cooked: <p>Congrats for releasing the <a href="https://pytorch.org/audio/main/models.decoder.html#module-torchaudio.models.decoder" class="inline-onebox" rel="noopener nofollow ugc">torchaudio.models.decoder — Torchaudio nightly documentation</a> for beam search decoding!</p>
<p>One question: Is it supported to run the decoder on GPU?</p>

username: ptrblck
cooked: <p>I don’t think this module works on the GPU, as <a href="https://github.com/pytorch/audio/blob/03a0d68e9ab06d7e1fbf84280a65d104a07abcff/torchaudio/models/decoder/_ctc_decoder.py#L211-L215">these checks</a> would fail and also the docs show the <code>CPU</code> label.</p>
Torch.stack() and torch.cat() taking too long

username: AjeelAhmed1998
cooked: <p>Hey guys,</p>
<p>The dataset is a collection of around 1500 three-dimensional(RGB) images with resolution 640x480 and I’m stacking them up for standardization.</p>
<p>I’m using <code>torchvision.datasets.ImageFolder</code> for fetching the dataset mostly because I don’t have a .csv file to map the images to their label.</p>
<pre><code class="lang-auto">dataset = ImageFolder(root=path, transform=transform)
stacked_dataset = torch.stack([im for im, _ in dataset], dim=3)
</code></pre>
<p>For transform I’m only using <code>T.ToTensor()</code></p>
<p>I have waited for more than 15 minutes and it still kept going.<br>
I also tried using <code>torch.cat()</code> with <code>unsqueeze</code> like so :</p>
<pre><code class="lang-auto"> stacked_dataset = torch.cat([img_t.unsqueeze(3) for img_t, _ in dataset], 3)
</code></pre>
<p>but it also had the same result.</p>
<p>Bringing down the resolution through <code>T.Resize()</code> worked but I would prefer to not bring down the resolution.</p>
<p>I could see from the Activity Monitor that it was taking up almost 12GB of memory (MBP M1Pro).</p>

username: ptrblck
cooked: <p>I would recommend to profile the code and check which part actually takes the majority of time as I would guess the data loading is slower than a <code>torch.stack</code> or <code>torch.cat</code> operation.</p>
How to compute tensor offset averages?

username: Aldebaran
cooked: <p>Consider a batch representation of labels (concatenated into a sentence) as shown below.</p>
<pre><code class="lang-python">batch_size = 2 # two set of labels as two sentences
max_length = 64 
rpr_dimension = 768

labels_rpr = torch.rand((batch_size, max_length, rpr_dimension))
labels_rpr # torch.Size([2, 64, 768])
# tensor([[[0.4031, 0.3784, 0.9089,  ..., 0.9796, 0.7666, 0.7783],
#          [0.9969, 0.2751, 0.0540,  ..., 0.8257, 0.4371, 0.2242],
#          [0.0208, 0.9337, 0.7295,  ..., 0.8140, 0.8909, 0.1517],
#          ...,
#          [0.5298, 0.3919, 0.6242,  ..., 0.1503, 0.9446, 0.7762],
#          [0.3007, 0.4251, 0.6003,  ..., 0.1441, 0.0233, 0.0407],
#          [0.3498, 0.3891, 0.0787,  ..., 0.1565, 0.0419, 0.7865]]])
</code></pre>
<p>Also, consider the <code>labels_offset</code>, showing the start and end of each label as shown below:</p>
<pre><code class="lang-python">labels_offset # torch.Size([2, 64])
# tensor([[ 1,  3,  5,  6,  7,  8, 11, 15, 16, 19, 21, 22, 23, 24, 28, 29, 30, 33,
#          34, 36, 37, 38, 39, 40, 44, 45, 47, 51,  0,  0,  0,  0,  0,  0,  0,  0,
#           0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
#           0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
#         [ 1,  5,  6,  8, 11, 12, 13, 14, 15, 16, 18, 22, 28, 29, 30,  0,  0,  0,
#           0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
#           0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
#           0,  0,  0,  0,  0,  0,  0,  0,  0,  0]])
</code></pre>
<p>Accordingly, in the first batch sample, we have the representation of the first label in the range <code>[1 3)</code> and the last label in the range <code>[47 51)</code>.</p>
<p>Then, is there an approach to compute each label’s representation as the average of the vectors in each <code>labels_offset</code> interval?</p>
CUDA issues during test-runs of distributed training loops within Pytest framework

username: yutanagano
cooked: <p>Hello,</p>
<p>I have a relatively complex training loop system that I want to make sure continues to behave as expected as I continue to add code to it and develop. That’s why I maintain unit-testing and integration-testing code alongside my Pytorch code (the test code is written using the Pytest framework).</p>
<p>Most of my testing code is comprised of simple unit tests, but the last bit is a set of integration tests that execute a mock-training run of sorts, and ensures that the pytorch model and training logs etc. are saved correctly. I have:</p>
<ul>
<li>One test that mock-runs a loop on the CPU</li>
<li>A second test that mock-runs a loop on the GPU</li>
<li>A third  that mock-runs a distributed loop using 2 GPUs (via DistributedDataParallel)</li>
</ul>
<p>When I run my test suite, all the tests pass except for that final one that tests whether the distributed loop works. The test fails with the first-spawned torch.multiprocessing process throwing the error:</p>
<pre><code class="lang-auto">E       RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable
</code></pre>
<p>However, if I run ONLY the test function which mock-runs a distributed training run, then it passes.</p>
<p>It’s only when I either run the whole testing suite, or even just one other test and the distributed test, where the distributed test fails.</p>
<p>Interestingly, running the distributed test twice in a row within the same Pytest invocation still runs fine. Both instances of the distributed test passes.</p>
<p>My question here is: is there something to do with how Pytorch manages GPU resources that I am not taking account of in my testing code? Like for example do I need to somehow ‘free’ GPU resources after each unit test which makes use of the GPUs? The unstable nature of my testing suite makes me think that I am probably not following the best practices for this kind of situation.</p>
<p>Thank you for your help in advance!</p>

username: Yanli_Zhao
cooked: <p>right, each unit test should be a separated ‘binary’ program and processes should be completely destroyed along with the GPU resources, please refer to the ‘<a href="https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/distributed/distributed_test.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/distributed_test.py at master · pytorch/pytorch · GitHub</a>’</p>

username: yutanagano
cooked: <p>Dear Yanli,</p>
<p>Sorry for the late response.<br>
Understood, I have restructured the tests so that each training loop is executed as its own process. It seems to have done the trick! Thank you.</p>
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>

username: A_the_kunal
cooked: <pre><code class="lang-auto">torch.manual_seed(101)
num_epochs=30
for epoch in tqdm(range(num_epochs)):
    for step,i in enumerate(train_loader):
        img,label_one_hot,label = i
        img = Variable(img).cuda()
        label_one_hot= Variable(label_one_hot.float()).cuda()
        pred = cnn_model(img)
        loss_value = loss_fn(pred,label_one_hot)
        optimizer.zero_grad()
        loss_value.backward()
        optimizer.step()
        print('epoch:', epoch+1, 'step:', step+1, 'loss:', loss_value.item())
</code></pre>
<p>This is my training loop</p>
<pre><code class="lang-auto">class Mydataset(Dataset):
    def __init__(self,path,is_train=True,transform=transform):
        self.path = path
        if is_train: self.img = os.listdir(self.path)
        self.transform = transform
    def __getitem__(self,idx):
        img_path = self.img[idx]
        files = os.listdir(self.path)
        for file in files:
            if (file.endswith(".jpg") and file[:-4] == length_of_captcha):
                img = cv2.imread(img_path,cv2.IMREAD_GRAYSCALE)
                kernel = cv2.getStructuringElement(cv2.MORPH_RECT,(5,12))
                img = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel)
                label = Path(self.path/img_path).name[:-4]
                label_ = []
            else:
                continue
            for i in label:
                label_ += encode(i)
            if self.transform is not None:
                img = self.transform(img)
            return img,np.array(label_),label
    def __len__(self):
        return (len(self.img))
</code></pre>
<p>Here I am trying for character recognition for 6 digit captcha image after some preprocessing. I am getting the error message in the title<br>
This is my dataset class and here I have passed successfully creating train_loader. But I am still encountering this and I don’t know what is going wrong. Can anyone help me?</p>

username: ptrblck
cooked: <p>Based on the error message I guess that some return values are empty.<br>
Could you add a condition into <code>__getitem__</code> before the <code>return</code> statement and check if <code>img</code>, <code>label_</code>, and <code>label</code> have valid values?</p>

username: MiLap
cooked: <p>Hello there, i am new to PyTorch and get the same error from my dataloader. I am not sure if i should open a new discussion or post it here?<br>
Anyway, <a class="mention" href="/u/ptrblck">@ptrblck</a> by check if the variables have valid values, you mean to print their shape?<br>
Many thanks for any help already in advance!</p>

username: ptrblck
cooked: <p>The error message claims that <code>NoneType</code>s are found in the <code>default_collate</code>, so you would need to check why some samples are loaded as <code>None</code>.</p>

username: MiLap
cooked: <p>Ok, i found the bug. Thanks for your help.</p>
To much time for loading data using data loader and Invalid for Input size error

username: Ab_Niazi
cooked: <p>I am loading custom data set from google drive and using google colab.</p>
<p><strong>first of all it is taking so much time to exectue the code and secondly it throws an error in training loop.</strong></p>
<p>code is below<br>
‘’’<br>
data_dir = ‘…/content/gdrive/My Drive/digits_train_set’</p>
<p>train_transforms = transforms.Compose([<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[0.5,],<br>
std=[0.5,] )<br>
])</p>
<p>data = datasets.ImageFolder(root=data_dir,  transform= train_transforms)</p>
<p>from torch.utils.data import DataLoader</p>
<p>batch_size = 100</p>
<p>data_loader = DataLoader(data, batch_size, shuffle=True)</p>
<p>for img_batch, label_batch in data_loader:<br>
print(‘first batch’)<br>
print(img_batch.shape)<br>
plt.imshow(img_batch[0][0], cmap=‘gray’)<br>
print(label_batch)<br>
break<br>
‘’’</p>
<p><strong>to display it is taking too much time.</strong></p>
<p>secondly</p>
<p>in the training loop it is taking so much time and then throws the error</p>
<p>the code of the training loop</p>
<p>‘’’</p>
<p>num_epochs = 10<br>
total_step = len(data_loader)<br>
d_losses, g_losses, real_scores, fake_scores = [], [], [], []</p>
<p>for epoch in range(num_epochs):<br>
for i, (images, _) in enumerate(data_loader):<br>
# Load a batch &amp; transform to vectors<br>
images = images.reshape(batch_size,-1).to(device)</p>
<pre><code>    # Train the discriminator and generator
    d_loss, real_score, fake_score = train_discriminator(images)
    g_loss, fake_images = train_generator()
    
    # Inspect the losses
    if (i+1) % 200 == 0:
        d_losses.append(d_loss.item())
        g_losses.append(g_loss.item())
        real_scores.append(real_score.mean().item())
        fake_scores.append(fake_score.mean().item())
        print('Epoch [{}/{}], Step [{}/{}], d_loss: {:.4f}, g_loss: {:.4f}, D(x): {:.2f}, D(G(z)): {:.2f}' 
              .format(epoch, num_epochs, i+1, total_step, d_loss.item(), g_loss.item(), 
                      real_score.mean().item(), fake_score.mean().item()))
    
# Sample and save images
save_fake_images(epoch+1)
</code></pre>
<p>‘’’</p>
<p><strong>The error is  <em>shape ‘[100, -1]’ is invalid for input of size 14112</em></strong></p>
<p>is that I am doing wrong with the data loader<br>
The first batch size is  torch.Size([100, 3, 28, 28])<br>
and my input is according to that, where I am missing?</p>

username: Ab_Niazi
cooked: <p>I change the following code and now it works fine</p>
<p><span class="hashtag">#images</span> = images.reshape(batch_size,-1).to(device)<br>
images = images.view(images.size(0), -1).to(device)</p>
Standard SOS and EOS for seq2seq 1D real/complex signal encoding?

username: Blupon
cooked: <p>Are there any standard / recommended SOS and EOS tokens for seq2seq encoding using RNN / LSTM / Transformers applied to real-valued and/or complex-valued 1D signals (with few samples, i.e. &lt;100 samples per input signal to be encoded) ? The encoding being generated for signal discrimination, in other words we are generating fixed-size 1D signal embeddings for classification.</p>
<p>I have a hard time finding good references for this. I know there are wav2vec (with convolutions) and wav2vec 2.0 (with convolutions and transformer), but I’m not sure such approaches are adapted for short signals never exceeding say a hundred samples. I also had a look at articles, such as <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=8688435&amp;tag=1" rel="noopener nofollow ugc">“LSTM-based auto-encoder model for ECG arrhythmias classification”</a>, without finding a proper explanation for the choice of SOS and EOS values. Perhaps I missed a typical case (like for a real-valued signal in [0,1], SOS could be 2 and EOS -1 ?).</p>
<p>I would definitely be also interested in a more elaborate encoding reference where the encoding network takes into account a sampling frequency diversity in the input signals, in addition to clearly stating which SOS and EOS are used and why.</p>
<p>Acronyms suggested in this <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">PyTorch tutorial</a>:<br>
SOS: start of sequence token<br>
EOS: end of sequence token</p>
<p><strong>EDIT (possible answer):</strong> It may be that SOS and EOS are not necessary for fixed-sized signal reconstruction by a seq2seq, i.e. EOS (and SOS ?) is necessary when the seq2seq is applied to translation, where the task is also about learning to predict the translated sentence size / choosing when to stop the sentence. In contrast, fixed-sized signal reconstruction discards learning to predict the output sequence size. I would still gladly receive indications on references developing such intuitions.</p>
<p>(I post in uncategorized since this seems to be neither an “audio” nor a “data” category question)</p>
Adding CUBLAS_WORKSPACE_CONFIG:4096:8 to Vscode launch.json file

username: papoo13
cooked: <p>This is mostly related to VsCode but since I am running a deep learning model in PyTorch and the issue is related to making PyTorch functions deterministic, I ask my question here. Hope you guys’ experiences can help me.</p>
<p>I need to add one of these two environemtal variables before running my code in vscode:</p>
<blockquote>
<p>CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8</p>
</blockquote>
<p>yet I don’t know exactly add this to the launch.json file of vscode program.<br>
I tried this:</p>
<blockquote>
<p>{<br>
“version”: “0.2.0”,<br>
“configurations”: [</p>
<pre><code>    {
        "name": "Python: Current File",
        "type": "python",
        "request": "launch",
        "program": "${file}",
        "console": "integratedTerminal",
        "args": ["${CUBLAS_WORKSPACE_CONFIG:4096:8}",
</code></pre>
</blockquote>
<p>to no avail. I appreciate your guidance here.</p>
Unable to reproduce PyTorch tutorial results

username: kbtorcher
cooked: <p>I installed cuda drivers + pytorch for my system. To ensure, everything is working as expected, I ran the <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">Transfer Learning tutorial</a> from the Pytorch website. However, I am unable to reproduce the results consistently. For some runs, I get the accuracy as per the tutorial. Other times it is off by 1%. These are my results:</p>
<pre><code class="lang-auto">Training complete in 0m 54s
Best val Acc: 0.934641

Training complete in 0m 46s
Best val Acc: 0.954248
</code></pre>
<p>I ain’t able to reproduce even while running the code on CPU.</p>
<pre><code class="lang-auto">Training complete in 3m 58s
Best val Acc: 0.941176

Training complete in 1m 51s
Best val Acc: 0.954248
</code></pre>
<p>On another CPU run, here’s what I got:</p>
<pre><code class="lang-auto">Training complete in 4m 1s
Best val Acc: 0.934641

Training complete in 1m 48s
Best val Acc: 0.954248
</code></pre>
In LibTorch, is torch::jit::Module::get_method documented somewhere?

username: Fei_Liu
cooked: <p>The <code>torch::jit::Module::get_method</code> is a very useful member function when I use the scripted mododule in cpp. However I didn’t find any proper docs in the official page, only this one:<br>
<a href="https://pytorch.org/cppdocs/api/program_listing_file_torch_csrc_jit_api_module.h.html?highlight=get_method" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/program_listing_file_torch_csrc_jit_api_module.h.html?highlight=get_method</a><br>
(which is pretty implicit. I searched “get_method” in the <a href="https://pytorch.org/cppdocs/" rel="noopener nofollow ugc">LibTorch cppdocs</a> top left search bar to located the page.)</p>
<p>Where can I find more proper docs on methods like this? Should I consider the official website as the “most golden” one for docs (other than reading the source code directly which I will probably not do <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"> ) ?</p>
How to write custom module with segmented function?

username: yangxi86
cooked: <p>I want to use pytorch as an simple function fitter, and the logic of function looks like:</p>
<pre><code class="lang-auto">if (x &lt; param_seg1) return param_a * x;
else if (x &lt; param_seg2) return param_b * x *x;
else return param_c * x * x * x;
</code></pre>
<p>and all these <code>param_xxx</code>s need to be learned.</p>
<p>As I want to utilize pytorch’s autograd feature, all arithmetic operations shall be taken with pytorch’s builtin. So how can I cope with the segment points? Is there anything like <code>step</code> function in GPU shader that works as replacement of <code>if</code> switches?</p>

username: ptrblck
cooked: <p>You should be able to directly execute the code without any modifications to the if conditions etc.<br>
PyTorch uses the “eager” mode by default, i.e. it’ll execute each line of code once it reaches it and is able to understand Python conditions, loops, etc.</p>
Do I need to use a distributed/modified optimizer with DistributedDataParallel

username: JamesDickens
cooked: <p>I went through this tutorial here:<br>
<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></p>
<p>My setup is a single multi-gpu machine.</p>
<p>I am wondering, does the optimizer need to be changed in anyway to account for the<br>
multiple processes?</p>
<p>Also should optimizer.step() only be called by the process at rank 0?</p>

username: ptrblck
cooked: <p>No, you don’t need to change the usage of the <code>optimizer</code> and each rank should call the <code>step()</code> function in DDP, not only rank0.</p>
Updating Optimizer Hyperparameters

username: J_Johnson
cooked: <p>Just want to check if I am doing this correctly because, when I print the optimizer.defaults, they are not updated.</p>
<pre><code class="lang-auto">dummy_model=nn.Linear(10, 2)

optimizer=torch.optim.SGD(dummy_model.parameters(), lr=0.01, momentum=0.9, nesterov=True)

#update the momentum to a new value:
optimizer.param_groups[0]['momentum']=0.8

print(optimizer.param_groups[0])

print(optimizer.defaults)
</code></pre>
<p>The first print statement shows the expected momentum at 0.8. The second still shows 0.9. So will this produce the desired effect of changing the momentum for training?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/j_johnson">@J_Johnson</a>,</p>
<p>If you want to change the momentum constant for all values, you need to apply it over all groups not just the first one. So,</p>
<pre><code class="lang-auto">for group in optimizer.param_groups:
  group['momentum']=0.8
</code></pre>

username: J_Johnson
cooked: <p>That’s an interesting point you’ve raised, which begs another question. I’ve never seen more than 1 group in optimizer.param_groups. Do you know of any example which would produce more than one group? Just curious.</p>

username: AlphaBetaGamma96
cooked: <p>Ah, ignore what I said (although that can be applied to more complicated models). The <code>.defaults</code> attribute is basically a <code>dict</code> of default values your network has when it’s initialized, there’s a bit more discussed in the comments of the source <code>torch.optim.Optimizer</code> class <a href="https://pytorch.org/docs/stable/_modules/torch/optim/optimizer.html#Optimizer" rel="noopener nofollow ugc">here</a>.</p>
<p>When you update via,</p>
<pre><code class="lang-auto">for group in optimizer.param_groups:
  group['momentum']=0.8
</code></pre>
<p>it will update and overwrite the default value.</p>
<p>I’ve always used <code>for group in optimizer.param_groups</code> over <code>optimizer.param_groups[0]</code> as that’s the same way it’s done within the base class, and covers optimizers with more than one group. In the documentation (shown <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options" rel="noopener nofollow ugc">here</a>) there’s an example with more than one group.</p>

username: J_Johnson
cooked: <p>Okay, so the param_groups, though, is the part I need to update in order to update optimizer parameters between training epochs.</p>
<p>By the way, there is also an:</p>
<p><code>optimizer.defaults['momentum']=0.8</code></p>
<p>which can be used to update the defaults. But perhaps the defaults are kept if you want to log what the starting optimizer values were, though, so that would be pointless to update.</p>
Add parameters to optim.Adam during training

username: Max_Unhold
cooked: <p>Hello,<br>
when I initialize a parameter from <a href="https://pytorch.org/docs/stable/optim.html" rel="noopener nofollow ugc">torch.optim — PyTorch 1.12 documentation</a>, i would do it like</p>
<pre><code class="lang-auto">optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
</code></pre>
<p>where the model has been defined beforehand.<br>
Let us assume that during training, I want to add some parameters to the training, e.g. add a module to <code>nn.Sequential</code>.</p>
<p>Can I add a parameter to optimizer that is considered in <code>optimizer.step()</code> or do I have to re-initialize the optimizer? In the last case, all information about the momentum would be lost.</p>

username: ptrblck
cooked: <p>Yes, you can use <a href="https://pytorch.org/docs/stable/generated/torch.optim.Optimizer.add_param_group.html#torch-optim-optimizer-add-param-group"><code>optimizer.add_param_group</code></a>.</p>
2.5D V-Net number of input and ouput channels

username: Petra_S
cooked: <p>Hi,</p>
<p>I am rewriting 3D V-Net to 2.5D V-Net to segment livers.<br>
It means I have 2 labels - background, livers.<br>
This specific 2.5D networks means that we use 5 slices as 5 channels.<br>
Input to the network has shape (1,5,300,300).<br>
But the output has a shape (1,2,300,300) where 2 represents number of classes.<br>
Why the output shape is not like (1,2,5,300,300)? I need different output for each channel since they are different right? Do you have any idea how it should be correctly?<br>
Thanks  for the answers.</p>
Turning an existing optimization package to a PyTorch Optimizer

username: SimplyAdi
cooked: <p>Hello, I wanted to convert torchswarm- an existing PSO library- for use in training a PyTorch neural network.</p>
<p>The PSO optimizer of torchswarm takes in inputs: objective function, dimensions, swarm size and maximum iterations.</p>
<p>I have made progress in creating a class that uses torch.optim.optimizer and transfer the neural net dimensions to the PSO optimizer of torch swarm.</p>
<p>However, I am stuck in creating the step function inside the optimizer class. Specifically, how do I code for updating the weights of the neural net using the output of the PSO optimizer.</p>
<p>Please help.</p>