akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gradients'dtype is not fp16 when using torch.cuda.amp

wangchengtao: <p>When I use torch.cuda.amp.autocast, the output is fp16, but the gradients are not fp16. Why?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
a = torch.randn([4, 5], requires_grad=True, device='cuda')
b = torch.randn([5, 4], requires_grad=True, device='cuda')
with torch.autocast(device_type='cuda', dtype=torch.float16):
    c = a @ b
c.backward(torch.ones_like(c))
print(c.dtype)
print(a.grad.dtype)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a336db9595817ca1abdc47e5833b0bafcb3ca20.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a336db9595817ca1abdc47e5833b0bafcb3ca20" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a336db9595817ca1abdc47e5833b0bafcb3ca20_2_690x221.png" alt="image" data-base62-sha1="61keOTswy8pqBPNXZxDeDCwU2R2" width="690" height="221" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a336db9595817ca1abdc47e5833b0bafcb3ca20_2_690x221.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a336db9595817ca1abdc47e5833b0bafcb3ca20_2_1035x331.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a336db9595817ca1abdc47e5833b0bafcb3ca20_2_1380x442.png 2x" data-dominant-color="1E2020"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1566×502 47.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

wangchengtao: <p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Could you help?</p>

ptrblck: <p>This is expected since the parameters will stay in <code>float32</code> and will not be replaced with lower-precision <code>dtype</code>s.</p>

wangchengtao: <p>Can you explain the internal mechanism? I guess that the loss is fp16, so the gradients produced are also fp16. However, the parameters are fp32, so pytorch will transform the gradients from fp16 to fp32. Right?</p>
Alternatives to torch.isin()

adv010: <p>Hi, I am looking for an alternative to torch.isin().I;m using the below code snippet to compare instances in 2 tensors (1D tensors but different lengths), and masking out non-common tensors to obtain 2 identical tensors. This works on my local machine, but fails on the slurm cluster at my university.</p>
<pre><code class="lang-auto">        valid_instances = np.intersect1d(instance_idx.cpu().numpy(),instance_idx_pair.cpu().numpy()) #
        valid_instances = torch.tensor(valid_instances,device=device)
       
        # intersect_mask, to remove instances which are present A but B and VICE VERSA
        intersect_mask = torch.isin(instance_idx,valid_instances)
        intersect_mask_pair = torch.isin(instance_idx_pair,valid_instances)
        
        instance_idx = instance_idx[intersect_mask]
        instance_idx_pair = instance_idx_pair[intersect_mask_pair]
</code></pre>
<p>I am guessing Pytorch version mismatch could be an issue. Is there an alternative operator to torch.isin()?</p>

adv010: <p>For the moment, I have reverted to using a for loop for this task</p>
<pre><code class="lang-auto">        intersect_mask = torch.tensor([idx in valid_instances for idx in instance_idx], device=device, dtype=torch.bool)
        intersect_mask_pair = torch.tensor([idx in valid_instances for idx in instance_idx_pair], device=device, dtype=torch.bool)
</code></pre>
<p>It works, but is too slow. Could someone suggest a better way / more elegant solution for the same?</p>

KFrank: <p>Hi Advait!</p>
<aside class="quote no-group" data-username="adv010" data-post="2" data-topic="190297" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adv010/48/45694_2.png" class="avatar"> adv010:</div>
<blockquote>
<p>Could someone suggest a better way / more elegant solution for the same?</p>
</blockquote>
</aside>
<p>You can avoid the explicit python (list-comprehension) loops at the cost<br>
of materializing what could be a large tensor.  (But it still uses an inefficient<br>
brute-force algorithm, so figuring out how to get <code>torch.isin()</code> working<br>
would really be the right way to go.)</p>
<p>I haven’t waded into the rest of your code, but here is a loop-free pytorch<br>
replacement for <code>isin()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.1.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt;
&gt;&gt;&gt; e = torch.randint (128, (64,))
&gt;&gt;&gt; t = torch.randint (128, (128,))
&gt;&gt;&gt;
&gt;&gt;&gt; isin = torch.isin (e, t)
&gt;&gt;&gt;
&gt;&gt;&gt; isinB = (e.unsqueeze (1) == t).sum (dim = 1).bool()
&gt;&gt;&gt;
&gt;&gt;&gt; torch.equal (isin, isinB)
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
How to make all floating point numbers have the same precision and on the same device

gengxingze: <p>for example</p>
<pre><code class="lang-auto">class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(in_features=200, out_features=1, bias=True)
        )
        self.net1 = nn.Parameter(torch.Tensor(1, 200))

    def forward(self, x):
        x.requires_grad_()
        aa = torch.randn(1, 200)
        m = feat * aa
        return m
net.double()
net.to("cuda")
</code></pre>
<p>Although the precision and device are set, the tensor created in the forward is still of type float and cpu.<br>
The designation must be displayed again in order to maintain type consistency through the “to(double)/to(“cuda”)”</p>

ptrblck: <p>You could use <code>torch.randn_like</code> and use the input or any parameter in the desired <code>dtype</code> or you could explicitly use e.g. <code>torch.randn(..., dtype=self.net1.weight.dtype)</code> (or the same with the input etc.).</p>

gengxingze: <p>Thank you very much！</p>
K-Means for Tensors

shivangi: <p>Is there some clean way to do K-Means clustering on Tensor data without converting it to numpy array.</p>
<p>I have a list of tensors and their corresponding labes and this is what I am doing.</p>
<p>def evaluateKMeansRaw(data, true_labels, n_clusters):</p>
<pre><code>kmeans = KMeans(n_clusters=n_clusters,n_init=20)
kmeans.fit(data)

acc = cluster_acc(true_labels, kmeans.labels_)
nmi = metrics.normalized_mutual_info_score(true_labels, kmeans.labels_)
return acc, nmi
</code></pre>
<p>But this doesn’t work on the output of a Conv Layer</p>

justusschock: <p>You cannot use scikit-learn on tensors as scikit-learn (and all of its methods) only work on numpy arrays.</p>
<p>However if you are not afraid to use custom implementations you could use something like <a href="https://discuss.pytorch.org/t/method-for-better-utilization-of-gpu-memory-for-kmeans-clustering/9586?u=justusschock">this</a></p>

shivangi: <p>Do we not have these algorithms as part of the framework as of now ?</p>

subhadarshi: <p>check out this <a href="https://github.com/subhadarship/kmeans_pytorch" rel="nofollow noopener">github repo</a>. can be installed in a breeze using pip:</p>
<p><code>pip install kmeans-pytorch</code></p>
<p>find documentation <a href="https://subhadarship.github.io/kmeans_pytorch" rel="nofollow noopener">here</a></p>

JosueCom: <p>I implemented NN, KNN and KMeans on a project I am working on only using PyTorch. You can find the implementation here with an example: <a href="https://gist.github.com/JosueCom/7e89afc7f30761022d7747a501260fe3" class="inline-onebox" rel="noopener nofollow ugc">Nearest Neighbor, K Nearest Neighbor and K Means (NN, KNN, KMeans) only using PyTorch · GitHub</a></p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch as th
&gt;&gt;&gt; from clustering import KNN
&gt;&gt;&gt; data = th.Tensor([[1, 1], [0.88, 0.90], [-1, -1], [-1, -0.88]])
&gt;&gt;&gt; labels = th.LongTensor([3, 3, 5, 5])
&gt;&gt;&gt; test = th.Tensor([[-0.5, -0.5], [0.88, 0.88]])
&gt;&gt;&gt; knn = KNN(data, labels)
&gt;&gt;&gt; knn(test)
tensor([5, 3])
</code></pre>

JosueCom: <p>This is an <a href="https://gist.github.com/wzjoriv/7e89afc7f30761022d7747a501260fe3" rel="noopener nofollow ugc">updated link</a></p>
NA labels in EMNIST dataset

abhraimrb: <p>Hi , I  am running the below code for loading EMNIST letters data. I am getting N/A labels as a extra class after running ‘dataset.classes’. Why these N/A labels are coming?</p>
<p>torch version-2.1.0<br>
tochvision version- 0.16.0</p>
<p>import torch<br>
import torchvision<br>
from torchvision.datasets import EMNIST<br>
dataset = EMNIST(root=‘data/’, split=‘letters’, download=True)<br>
dataset.classes</p>

jonathanjsmith: <p>In the source code the classes for EMNIST are defined as:</p>
<pre><code class="lang-auto">classes_split_dict = {
        "byclass": sorted(list(_all_classes)),
        "bymerge": sorted(list(_all_classes - _merged_classes)),
        "balanced": sorted(list(_all_classes - _merged_classes)),
        "letters": ["N/A"] + list(string.ascii_lowercase),
        "digits": list(string.digits),
        "mnist": list(string.digits),
 }
</code></pre>
<p>This raises the question of why the letters need “N/A” class. I thought maybe some targets did not fall in the range of ‘a-z’</p>
<p>So I checked the targets.</p>
<pre><code class="lang-auto">import torch
import torchvision
from torchvision.datasets import EMNIST
import collections

dataset = EMNIST(root='data', split='letters', transform=transforms.ToTensor(), download=True)
data_classes = collections.defaultdict(int)
for i in range(len(dataset)):
  data_classes[chr(ord('a') - 1 + dataset.targets[i].item())] += 1
print([(x, data_classes[x]) for x in sorted(data_classes.keys())])
</code></pre>
<p>The output was 4800 images for each letter, so each image contained a valid target (with no N/A). I am not sure of the purpose of the N/A since all images fall within a valid target class. It’s possible that it can be removed from the classes. However, it shouldn’t affect the usability of the dataset.</p>
PyTorch inference prioritize using /usr/local/cuda in PATH or cudatoolkit?

runa1996821: <p>I wanted to try to align the precision of Pytorch inference between the two versions, but I found that adding CUDA paths to PATH and LD_LIBRARY_PATH affected the results of inference. So I did some comparative experiments, and these are my CUDA environment：</p>
<p>NVIDAI Driver: NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2<br>
conda env a : torch1.9.0+cu111<br>
conda env b : torch1.13.0+cu117<br>
“SET_CUDA_ENV” means execute following command:<br>
<code>export PATH=/usr/local/cuda-11.2</code><br>
<code>export LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64:$LD_LIBRARY_PATH</code><br>
“NO_CUDA_ENV” means do nothing about PATH and LD_LIBRARY_PATH.</p>
<p>experiment1:<br>
[ torch 1.9.0+cu111 &amp; SET_CUDA_ENV ] vs [ torch1.9.0+cu111 &amp; NO_CUDA_ENV ]  I got same result<br>
experiment2:<br>
[ torch 1.13.0+cu117 &amp; SET_CUDA_ENV ] vs [ torch1.13.0+cu117 &amp; NO_CUDA_ENV ] I got different results on the first torch.nn.linear layer.<br>
experiment3:<br>
[ torch 1.9.0+cu111 &amp; SET_CUDA_ENV ] vs [ torch1.13.0+cu117 &amp; SET_CUDA_ENV ]  I got different results on the first torch.nn.linear layer.<br>
experiment4:<br>
[ torch 1.19.0+cu111 &amp; NO_CUDA_ENV ] vs [ torch1.13.0+cu117 &amp; NO_CUDA_ENV ]  I got different results on the first torch.nn.linear layer.</p>
<p>Based on the above experiments, I can’t know which cuda libraries pytorch uses for inference.If pytorch had prioritized /usr/local/cuda11.2 in PATH and LD_LIBRARY_PATH, experiment 3 should have yielded the same result. If cudatoolkit is preferred, then experiment 2 should got the same result. These experiments show that there may be other factors that influence the model’s inference.<br>
I’ve seen these discussions <a href="https://discuss.pytorch.org/t/how-to-check-which-cuda-version-my-pytorch-is-using/116622">How to check which cuda version my pytorch is using</a>, <a href="https://discuss.pytorch.org/t/how-does-pytorch-pick-cuda-versions/137941">How does PyTorch pick cuda versions?</a></p>
<p>Hope to get some explanation, thanks!</p>

ptrblck: <p>The PyTorch binaries ship with their own CUDA dependencies and should not use any system-wide libs. If you want to use your system-wide CUDA toolkit you could build PyTorch from source.</p>