akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Using a discriminator for generating similar outputs

jyy20422: <p>Hello All,</p>
<p>I am currently trying to train a 2 encoders/generators so that they generate similar outputs. Each generator generates 2 different outputs, so in total I have 4 different generated outputs that I want to constrain to be similar. I had the idea to use a discriminator to constrain the generators in an adversarial training scheme (GAN). I wanted train the adversarial process so that the generators are trained to generate 4 different outputs that the discriminator fails to differentiate. Since a discriminator is essentially a classifier, I thought I could just implement a 4-class classifier with a softmax output layer where each of the 4 outputs’ true label is [0, 0, 0, 1], [0, 0, 1, 0], [0, 1, 0, 0], [1, 0, 0, 0]. So the goal of the discriminator would be to correctly classify the correct label given one of the generated outputs. On the other hand, the generator would be trained so that the discriminator’s output is [0.25, 0.25, 0.25, 0.25], i.e. a random guess. I have implemented this using the nn.CrossEntropyLoss. However, the model would never converge and the loss would not decrease. Does anyone have any advice that they could provide me? Should I not be using the CrossEntropyLoss and use a BCEWithLogitsLoss? If so, what would the “fake” label be? I was thinking maybe [0,0,0,0], but then that is not necessarily a “random” guess no?</p>
What does `overload_name` mean in `native_functions.yaml`?

Bob_Liao: <p>As what I asked, I see there are some rules for defining functions in <code>native_functions.yaml</code>:</p>
<aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native" target="_blank" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native at main · pytorch/pytorch</a></h3>

  <p><a href="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native" target="_blank" rel="noopener nofollow ugc">main/aten/src/ATen/native</a></p>

  <p><span class="label1">Tensors and Dynamic neural networks in Python with strong GPU acceleration</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">- func: func_name[.overload_name](ArgType arg0[=default], ArgType arg1[=default], ...) -&gt; Return
</code></pre>
<p>I read this, but it didn’t explain the <code>overload_name</code>’s meaning.</p>
<p>For example:</p>
<pre><code class="lang-auto">- func: bernoulli_.float(Tensor(a!) self, float p=0.5, *, Generator? generator=None) -&gt; Tensor(a!)
</code></pre>
<p>I think it means overloading implementation for <code>t.bernoulli()</code> wehn <code>t</code>’s <code>dtype</code> is <code>float</code>. But for:</p>
<pre><code class="lang-auto">- func: bernoulli_.Tensor(Tensor(a!) self, Tensor p, *, Generator? generator=None) -&gt; Tensor(a!)
</code></pre>
<p>What does this mean? I can’t understand. Glad for your help or docs.</p>

Bob_Liao: <p>Oh, I got this,</p>
<p><code>.Tensor</code> means <code>p</code> is passed as a tensor.</p>
RuntimeError: vmap: It looks like you're calling .item() on a Tensor

Ritesh_Khurana: <p>Code:</p>
<pre><code class="lang-auto">def foo(a, ind, len):
    ind2 = ind + len
    return a[ind:ind2]

torch.vmap(foo, in_dims=(0, 0, None))(a, c, 2)
</code></pre>
<p>RuntimeError: vmap: It looks like you’re calling .item() on a Tensor. We don’t support vmap over calling .item() on a Tensor, please try to rewrite what you’re doing with other operations. If error is occurring somewhere inside PyTorch internals, please file a bug report.</p>
What is the most efficient way to collect samples in RL like PPO?

martin_xiao: <p>One of the main problems in reinforcement learning is that it is too inefficient to collect samples, so the GPU usually needs to wait for the CPU to collect samples. The obvious way is to create multiple agents through multiple processes to collect samples in parallel in multiple environments. Therefore, it is necessary to design a multi-process system, each process has its own actor with same parameters, same agent and same environment.</p>
<p>The question is how to use pytorch to create a such system?</p>

vmoens: <p>Great question <a class="mention" href="/u/martin_xiao">@martin_xiao</a> !<br>
Have a look at <a href="https://github.com/facebookresearch/rl/" rel="noopener nofollow ugc">torchrl</a>: we provide a set of multi-processing utils to run environments in parallel (see our <a href="https://github.com/facebookresearch/rl/tree/main/torchrl/collectors" rel="noopener nofollow ugc">data collectors</a> and <a href="https://github.com/facebookresearch/rl/blob/main/torchrl/envs/vec_env.py" rel="noopener nofollow ugc">parallel envs</a>).<br>
They are quite efficient and should suit your purpose. We also provide a PPO implementation by the way.<br>
Don’t hesitate to reach out if you have any further question!</p>

martin_xiao: <p>Hi Vincent,</p>
<p>torchrl is a splendid work!! I will look into it!</p>
<p>Thanks!!</p>

roeslib: <p>Can I use torchrl in windows? or does it only work in linux?</p>

vmoens: <p>We build and test it on windows so yes: it should!<br>
Happy to help solve any issue you might have.</p>
Embedding Categorical Variables

sdg91: <p>Hi, I am creating a LSTM model where categorical features need to be embedded before using it in the LSTM. Below is my code for LSTM. I am creating a custom dataset to pass to the model.</p>
<pre><code># Create a dictionary to store the embedding dimensions
embedding_dims = dict(zip(categorical_features, embedding_sizes))


class LSTMModel(nn.Module):
    def __init__(self, embedding_dims, hidden_dim, output_dim):
        super(LSTMModel, self).__init__()
        self.hidden_dim = hidden_dim
        self.embeddings = nn.ModuleDict({
            feat: nn.Embedding(num_categories, embedding_size)
            for feat, num_categories, embedding_size in zip(categorical_features, embedding_sizes, embedding_dims.values())
        })
        self.lstm = nn.LSTM(sum(embedding_dims.values()), hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        embedded = [self.embeddings[feat](x[:, :, i]) for i, feat in enumerate(categorical_features)]
        embedded = torch.cat(embedded, dim=2)
        lstm_out, _ = self.lstm(embedded)
        output = self.fc(lstm_out[:, -1, :])
        return output


   class CustomDataset(Dataset):
      def __init__(self, df):
          self.data = df.astype('category').values
  
      def __getitem__(self, index):
          x = torch.tensor(self.data[index:index+8, :-1])  # Sequence of 8 steps (input)
          y = torch.tensor(self.data[index+8, -1], dtype=torch.float32)  # Target value for the next step
          return x, y
  
      def __len__(self):
        return len(self.data) - 8
</code></pre>
<p>It shows the datatype error<br>
TypeError: can’t convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool. How to process the data show that I can feed it in above model architecture.</p>

srishti-git1110: <p>Hey, looks like you are trying to convert an <code>np.ndarray</code> into <code>torch.tensor</code> and the data type of <code>np.ndarray</code> is inappropriate for use in tensors as is specified by the error message.<br>
Only numerical types are supported. You could explicitly try to specify a correct data type for the <code>np.ndarray</code> before converting it to tensor, like so -</p>
<pre><code class="lang-python">import torch
import numpy as np

arr = np.array([1.0, 2.0], dtype=object)
arr = arr.astype(float, copy=False) # But for this line, the same error as yours is produced
x = torch.tensor(arr)
print(x) # tensor([1., 2.], dtype=torch.float64)
</code></pre>
Expected 4-dimensional input for 4-dimensional weight [192, 768, 1, 1], but got 2-dimensional input of size [50, 1000] instead

piperino11: <p>I’m try to modify Inception v3 pre trained in pytorch to have a multi-output. ( 4 output precisely).</p>
<p>I get this error: <strong>Expected 4-dimensional input for 4-dimensional weight [192, 768, 1, 1], but got 2-dimensional input of size [50, 1000] instead</strong></p>
<p>My input shape is : torch.Size([50, 3, 299, 299])</p>
<p>This is the code for my model,</p>
<pre><code class="lang-auto">class CNN1(nn.Module):
    def __init__(self, pretrained):
        super(CNN1, self).__init__()
        if pretrained is True:
            self.model = models.inception_v3(pretrained=True)    
        modules = list(self.model.children())[:-1]      # delete the last fc layer.
        self.features = nn.Sequential(*modules)
        self.fc0 = nn.Linear(2048, 10)   #digit 0
        self.fc1 = nn.Linear(2048, 10)  #digit 1
        self.fc2 = nn.Linear(2048, 10)    #digit 2
        self.fc3 = nn.Linear(2048, 10)   #digit 3  
    def forward(self, x):
        bs, _, _, _ = x.shape
        x = self.features(x)
        x = F.adaptive_avg_pool2d(x, 1).reshape(bs, -1)

        label0 = self.fc0(x)
        label1 = self.fc1(x)
        label2= self.fc2(x) 
        label3= self.fc3(x)
          
        return {'label0': label0, 'label1': label1,'label2':label2, 'label3': label3}
</code></pre>
<p>and this is a piece of iteration:</p>
<pre><code class="lang-auto">        for batch_idx, sample_batched in enumerate(train_dataloader):
            # importing data and moving to GPU
            image,label0, label1, label2, label3 = sample_batched['image'].to(device),\
                                             sample_batched['label0'].to(device),\
                                              sample_batched['label1'].to(device),\
                                               sample_batched['label2'].to(device) ,\
                                                       sample_batched['label3'].to(device)  

            
            # zero the parameter gradients
            optimizer.zero_grad()
            output=model(image.float())
</code></pre>
<p>anyone have a suggestion?</p>

my3bikaht: <p>if you have input (50,3,299,299) and something in between gets (50, 1000) but expects 4d input, it probably means you are trying to pass 1d ouput to image-processing layer (channels, h, w)</p>
<p>(50, 1000) looks like an output of some classifier with batch size 50 and num_classes = 1000</p>

Vasudha_Venkatesan: <p>Hi! I have a similar error, did you find any solution for this error? Thanks:)</p>
It takes a long time for torch to send back after specifying the gradient of the tensor

m13651927162: <p>I use pytorch to specify the gradient of the output tensor and then return the gradient, using this code: im_val.backward(grad_tmp.detach().clone()). im_val and grad_tmp are a matrix. im_val is obtained by this code: fdict_val, _, im_val, mask = generator.gen_fixed(im1.detach().clone(),init_logo). Then I just need to get the gradient returned to init_logo, it seems that I can’t get it with this code: grads_on_logo, = torch.autograd.grad(im_val, [init_logo]). Because torch.autograd.grad can only be passed back as a number of loss, I would like to ask whether it is possible to use an API of torch to directly obtain the gradient on init_logo without obtaining the gradient of other tensors. Another problem is that it takes a long time for me to use im_val.backward(grad_tmp.detach().clone()), it may take at least ten minutes, but the code of the tensorflow version may be returned in a minute at most, I don’t know why.</p>
How to get CUDA Kernels?

abderrahim_elhazmi: <p>Hello everyone,<br>
I am doing some research on Deep Learning performance on GPU. I want to get the CUDA Kernels that are executed on GPU. So I added these lines to my training file :</p>
<p>os.environ[“TORCH_DISTRIBUTED_DEBUG”] = “DETAIL”<br>
os.environ[“TORCH_CPP_LOG_LEVEL” ] = “INFO”</p>
<p>but I get nothing in log !<br>
Do you have any suggestions?<br>
Thanks !</p>

Vinayaka_Hegde: <p>Hi, <a class="mention" href="/u/abderrahim_elhazmi">@abderrahim_elhazmi</a> , I don’t have suggestions regarding your question, but I’m also looking for the same currently - mainly to get the kernels executed on GPU.</p>
<p>Could you please let me know the source from where did you found the two lines ? And does setting the two env variables give us logs on the Kernels being executed on GPU ?</p>
<p>Thanks a lot !</p>
<aside class="quote no-group" data-username="abderrahim_elhazmi" data-post="1" data-topic="180441">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abderrahim_elhazmi/48/60696_2.png" class="avatar"> abderrahim_elhazmi:</div>
<blockquote>
<p>os.environ[“TORCH_DISTRIBUTED_DEBUG”] = “DETAIL”<br>
os.environ[“TORCH_CPP_LOG_LEVEL” ] = “INFO”</p>
</blockquote>
</aside>

ptrblck: <p>You could use a profiler such as Nsight Systems, which will show the called CUDA kernels.</p>

abderrahim_elhazmi: <p>Hey <a class="mention" href="/u/vinayaka_hegde">@Vinayaka_Hegde</a>, I found them in this discussion<br>
<a href="https://discuss.pytorch.org/t/performance-impact-of-torch-cpp-log-level-info-and-torch-distributed-debug-detail/180199">Performance Impact of TORCH_CPP_LOG_LEVEL=INFO and TORCH_DISTRIBUTED_DEBUG=DETAIL - C++ - PyTorch Forums</a></p>
Does `torch.func.functionalize` support `fx.GraphModule` from `dynamo.export` w/ aten graph?

BowenBao: <p>Background: I wanted to remove mutation from my dynamo exported fx.graph. Part of the workflow looks like this</p>
<pre><code class="lang-python">gm, _ = torch._dynamo.export(model, dummy_input, aten_graph=True)
gm = proxy_tensor.make_fx(func.functionalize(gm))(dummy_input)
</code></pre>
<p>It worked for many simple cases, but I ran into the below error with shufflenet from torchvision.</p>
<pre><code class="lang-auto">  File "&lt;eval_with_key&gt;.4", line 15, in forward
  File "/home/bowbao/pytorch/torch/_ops.py", line 398, in __call__
    return self._op(*args, **kwargs or {})
RuntimeError: false INTERNAL ASSERT FAILED at "/home/bowbao/pytorch/build/aten/src/ATen/RegisterFunctionalization_2.cpp":7718, please report a bug to PyTorch. mutating a non-functional tensor with a functional tensor is not allowed. Please ensure that all of your inputs are wrapped inside of a functionalize() call.
</code></pre>
<p>The full repro script:</p>
<pre><code class="lang-python">import torchvision
import torch
import torch._dynamo
from torch import func
from torch.fx.experimental import proxy_tensor

model = torchvision.models.shufflenet_v2_x0_5(pretrained=False)
dummy_input = torch.randn(1, 3, 224, 224, requires_grad=True)

gm, _ = torch._dynamo.export(model, dummy_input, aten_graph=True)
gm = proxy_tensor.make_fx(func.functionalize(gm))(dummy_input)
</code></pre>

bdhirsh: <p>Dug into this a little. Some context:</p>
<p>(1) that model is mutating some of its buffers / params as part of the model forward</p>
<p>(2) <code>functorch.functionalize()</code> has a more limited contract: it will remove mutations on graph inputs and intermediates, but it will <strong>not</strong> remove any mutations done to captured variables or global state. When you’re functionalizing a <code>model.forward()</code> call, buffers/parameters count as non-local state (they weren’t lifted to be inputs of the function that we’re functionalizing over)</p>
<p>(3) Some time soon, we’re going to add an API to aot autograd that will probably do what you want - a way to take a function / model and return a functionalized graph, that also handles other stuff for you like flattening input pytrees, and lifting module state into graph inputs.</p>
<p>Side note: export will likely perform functionalization automatically soon (this doesn’t happen today though).</p>

gilfree: <p>From what I saw, you can make it work with current pytorch by passing a aten decomposition table to make_fx so that the make_fx will work on decomposed graph:</p>
<pre><code class="lang-python">from torch._decomp import core_aten_decompositions

def functionalized_export(func, *args, **kwargs):
    graph, _= torch._dynamo.export(func, *args, aten_graph=True, **kwargs)
    graph = make_fx(
        functionalize(interpret(graph)),
        decomposition_table=core_aten_decompositions()
    )(*args, **kwargs)
    return graph
</code></pre>
Multi-gpu model parallel training in PyTorch

iamexperimentingnow: <p>Hi, can someone recommend me a book or youtube tutorial channel or blogs to learn multi-gpu distributed model training or model parallel training using pytorch.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Brasilino: <p>Hi,</p>
<p>Have you looked at <a href="https://pytorch.org/tutorials/distributed/home.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed and Parallel Training Tutorials — PyTorch Tutorials 2.0.1+cu117 documentation</a> ?<br>
You’ll find pretty much anything over there…</p>

iamexperimentingnow: <p>thanks for this. I noticed this as well. I thought there would be some books to learn about it. Thanks though <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Functional -> Sequential, get weights and load them into the Functional

ibro45: <p>Hi,</p>
<p>I haven’t found a thread on the same topic, sorry if there is one already!</p>
<p>So, I am converting a model into a nn.Sequential model. Once trained, I would like to load those weights into the pre-Sequential (functional) model. Is there a way to map the Sequential model’s keys back to the original model?</p>
<p>Thank you very much!</p>

ptrblck: <p>You could manually load the <code>state_dict</code>s into the custom module and the actual approach would depend on the model architectures, the naming of the layers, etc.</p>

ibro45: <p>Alright, thank you! I was hoping there might be a way to figure out the functional-&gt;sequential mapping of the layer names automatically, but manually it is <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20"></p>
RuntimeError: Stop_waiting response is expected. RPC

Sanket_Desai1: <p>Worker 1:<br>
import torch<br>
import torch.distributed.rpc as rpc<br>
rpc.init_rpc(“worker0”, rank=0, world_size=2,<br>
rpc_backend_options=rpc.TensorPipeRpcBackendOptions(<br>
init_method=‘tcp://0.tcp.ngrok.io:18324’))<br>
while True:<br>
rref1 = rpc.remote(“worker1”, torch.add, args=(torch.ones(2), 3),timeout=0)<br>
rref2 = rpc.remote(“worker1”, torch.add, args=(torch.ones(2), 1))<br>
x = rref1.to_here() + rref2.to_here()</p>
<p>rpc.shutdown()</p>
<p>Worker 2:<br>
import torch.distributed.rpc as rpc<br>
rpc.init_rpc(“worker1”, rank=1, world_size=2,rpc_backend_options=rpc.TensorPipeRpcBackendOptions(<br>
init_method=‘tcp://0.tcp.ngrok.io:18324’))<br>
rpc.shutdown()</p>
<p>I Have been trying to  run these two codes on two different systems(completely different and only connected via internet).<br>
I just don’t understand what I am doing wrong as I am getting this error.</p>
<p>RuntimeError: Stop_waiting response is expected</p>
<p>Also I have a few questions regarding RPC:</p>
<ol>
<li>Can I have a system where I can send every single model blocks(Upto 96Transformer Blocks) on different GPU on multiple cloud service? Is it going to be slow?</li>
<li>If I can do the above how should I even begin?</li>
</ol>

H-Huang: <p>I haven’t tried to repro this, but my intuition for the error (RuntimeError: Stop_waiting response is expected) is that the “worker1” has finished it’s execution and shutdown() such that “worker0” can no longer communicate with it. You will need to make “worker1” long running.</p>
<blockquote>
<ol>
<li>Can I have a system where I can send every single model blocks(Upto 96Transformer Blocks) on different GPU on multiple cloud service? Is it going to be slow?</li>
</ol>
</blockquote>
<p>Is there a reason you need to send the entire model? Is this for some specific type of architecture like parameter server?</p>
How to check PyTorch2 model is compiled

zhihao_zhu: <p>Hi, how do I check my model has been successfully compiled in Pytorch 2, after calling torch.compile(). Basically, is there any properties in the model that I can check that it’s a compiled model?</p>
<p>Thanks</p>

marksaroufim: <p>There’s a test utility that’s very helpful for this called <code>CompileCounter</code> <a href="https://github.com/pytorch/pytorch/blob/main/test/dynamo/test_compile.py#L25" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_compile.py at main · pytorch/pytorch · GitHub</a></p>
<p>But otherwise the compilation happens after the first inference and another easy way to sanity check is to make sure kernels were generated if you add <code>TORCH_COMPILE_DEBUG</code> to True as an environemnt variable</p>

zhihao_zhu: <p>Thanks for your reply! For the second solution, how can I check the kernels are generated?</p>

marksaroufim: <p>The way I usually do it if I set <code>TORCH_COMPILE_DEBUG</code> then I’ll see the Triton or C++ kernels generated in a <code>tmp</code> folder that’ll be printed to your console if the debug flag is set</p>

zhihao_zhu: <p>Hi, got it, many thanks!</p>
Set CUDA_VISIBLE_DEVICE, Multiple MIGs, Single Job

J_W: <p>I am attempting to use a package that employs pytorch and I keep getting errors when asking it to select GPUs 0 and 1, saying there are not that many GPUs.</p>
<p><code>CUDA call failed lazily at initialization with error: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1670525552411/work/aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch.</code></p>
<p>The script tries to set GPUs via the following line of code, where gpu_list is ‘0,1’<br>
<code> os.environ['CUDA_VISIBLE_DEVICES'] = gpu_list --&gt; gpu_list = '0,1'</code></p>
<p>I have noted that the cluster I am using uses MIGs. How could I set CUDA_VISIBLE_DEVICES to multiple MIGs for a single script? I noted the following thread, but it applies to using MIGs on parallel jobs rather than together: <a href="https://discuss.pytorch.org/t/access-gpu-partitions-in-mig/142272" class="inline-onebox">Access GPU partitions in MIG</a></p>

ptrblck: <p>You cannot use multiple MIG instances in the same process and need to select one via <code>CUDA_VISIBLE_DEVICES</code>.</p>

J_W: <p>Is it possible to select the two GPUs rather than two MIGs, or would that require removing the multi-instancing?</p>

ptrblck: <p>I think you would need to disable MIG to select two (full) GPUs.</p>

vkalidas: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , Will the Multiple MIGs be supported in future pytorch versions? Are there any other alternatives with H100 GPU?</p>

ptrblck: <p>I’m not aware of relaxing the limitation and note it’s a CUDA limitation, nor a PyTorch one.</p>
Zipped csv file cannot be parsed using torchdata's CSVParser datapipe

seunggs: <p>Hi,</p>
<p>I’m trying to parse a single csv file that is zipped and stored in aws s3, but getting the following error:</p>
<pre><code class="lang-auto">"Exception when executing new-line character seen in unquoted field - do you need to open the file in 
universal-newline mode?\nThis exception is thrown by __iter__ of CSVParserIterDataPipe(fmtparams={'delimiter': ','}, source_datapipe=ZipArchiveLoaderIterDataPipe)"
</code></pre>
<p>Here’s my code:</p>
<pre><code class="lang-auto">dp = IterableWrapper([source]).list_files_by_fsspec()
dp = dp.open_files_by_fsspec(mode="rb")
dp = dp.load_from_zip()
dp = dp.parse_csv(delimiter=",")

df = pd.DataFrame()
for _, row in dp:
    print(row)
    df.append(row)
</code></pre>
<p>It looks like this error is due to how zip file was created (I downloaded the zip file from Kaggle), but I’m not sure what the solution is. Trying to change the mode to “rU” throws <code>Unimplemented</code> error.</p>
<p>Any help would be greatly appreciated - thank you!</p>
Unable to get cuda 11.3 with torch 1.9.0 working

AMPW: <p>I need pytorch 1.9.0 and I also want to use my 3080 ti. I’m using anaconda in a new eniviroment, I installed the cuda toolkit, installed cuda 11.3 (didn’t changed the nvidia driver with cuda 12.1), tried serveral versions of pytorch 1.9.0 but I always get cuda version none if I run “torch.version.cuda”. Any idea how to get it solved? Thx for any help, have a nice day.</p>

ptrblck: <p>The PyTorch 1.9.0 binaries were build with CUDA 10.2 and 11.1, whereas your 3080TI would need 11.x.<br>
You could install the pip wheel from <a href="https://download.pytorch.org/whl/torch_stable.html">here</a>.</p>

AMPW: <p>well, I tried now again using pip install, I tried it multiple times earlier, it never worked but now it’s working and I get at least cuda 11.1</p>
How to set labels for images in a dataset?

Pytorch_developer: <p><strong>I have some code:</strong></p>
<p>from torchvision.datasets import ImageFolder<br>
import torchvision.transforms as T<br>
from torch.utils.data import DataLoader</p>
<p>path = “/Users/edenbrown/Downloads/kagglecatsanddogs_3367a”<br>
transform = T.Compose([T.Resize((150, 200)),T.ToTensor()])</p>
<p>dataset = ImageFolder(root=path, transform=transform)</p>
<p>dataloader = DataLoader(dataset, batch_size=10, shuffle=True)</p>
<p>for batch_number, (images, labels) in enumerate(dataloader):<br>
print(batch_number, labels)<br>
print(images)<br>
break</p>
<p><strong>The output gives me the batch numbers, labels and tensors of the images in the batches which is correct. However, I have noticed that the labels are all 0 which is not what I want. I would like my labels to be 0 for dogs, 1 for cats or the other way around. Therefore, is there are way to set labels for images in a dataset?</strong></p>

Matias_Vasquez: <p>Can you post the full path to one dog image and to one cat image?</p>

Pytorch_developer: <p><strong>Cat path:</strong> /Users/edenbrown/Downloads/kagglecatsanddogs_3367a/PetImages/Cat<br>
<strong>Dog path:</strong>/Users/edenbrown/Downloads/kagglecatsanddogs_3367a/PetImages/Dog</p>

Matias_Vasquez: <p>OK,</p>
<p>Then you need to change the path you use for the dataset to</p>
<pre><code class="lang-python">path ="/Users/edenbrown/Downloads/kagglecatsanddogs_3367a/PetImages"
</code></pre>
<p>Let me know if it now works<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji only-emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

Pytorch_developer: <p>But how will I be able to set labels for dogs and cats just by changing the path for the dataset.</p>

Matias_Vasquez: <p>With the old path, there is only one folder (PetImages) and all of the images inside this folder correspond to this “class”</p>
<p>If you change the path, there will be 2 folders (dogs, cats). Now ImageFolder knows there are two classes. One folder is one class, the other folder is another class.</p>
<p>The labels will be either 0 or 1.</p>
<p>If you want to change this to a string label, you can create your own class that inherits from ImageFolder, delegate all major functionality to the parent class and only change the <strong>getitem</strong>() behavior to return “dog” or “cat” instead of 1 and 0.</p>
<p>But it’s late here now, so if you want me to show you how this is done, I can do it tomorrow</p>
<p>Or maybe someone else can answer for me <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

Pytorch_developer: <p>Thank you this worked</p>

Vivek_Yogi: <p>But suppose if our dataset have cat and dog images in single folder ,and additional json file for each image discription than how to label each images.</p>
How to get the MIG UUID of the current device in python?

Praveen_Starry: <p>I want to be able to get the MIG UUID of the visible device in python when using PyTorch. Is there a way to get this info? From Nvidia documentation I saw that a process sees only 1 MIG instance, so I just want to be able get the UUID for it.</p>

ptrblck: <p>You could call the internal <a href="https://github.com/pytorch/pytorch/blob/a630328695589fd1412edc0952fe141ef28cdc60/torch/cuda/__init__.py#L574-L603"><code>torch.cuda._raw_device_uuid_nvml()</code></a> method or re-implement a custom one if needed. Note that internal methods can easily break and do not guarantee to have a stable interface.</p>
Multitask model based on YoloV5

aspiranteaphd: <p>Hi, I want to create a multitask deep learning model, based on the architecture of yolov5. My model should have a classification head, a segmentation head, and multiple detection heads. Any idea how I should create this network? Can change the YAML configuration file, or create the network from scratch, considering that the YoloV5 architecture is not completely sequential?<br>
Is it also possible to train the different heads separately and then transfer the weights to the aggregated model?</p>
Jetson Nano AMP varied inference time

wjf37: <p>I have been testing AMP inference on the Jetson Nano with various model architectures, on the CIFAR-10 test set for inference, batch size of 32, and have seen some varied results for the inference time. ResNet runs about 10-15% slower compared to float32, VGG 30% faster, while MobileNetV2 stays around the same inference time.</p>
<p>I have read that Jetson Nanos should not have much speedup due to the lack of Tensor Cores, but the difference in results is unexpected, especially the slowdown on ResNet.<br>
I’ve tried using the profiler with a smaller subset due to lack of memory but the output does not seem to provide much information.</p>
<p>All the architectures have been tested in the same way, with the models and weights coming from <a href="https://github.com/chenyaofo/pytorch-cifar-models/tree/master" rel="noopener nofollow ugc">here</a>.<br>
The code for testing:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import DataLoader, Subset
import time
import torchvision.transforms as transforms
from torchvision.datasets import CIFAR10

DEVICE = "cuda:0"

def load_data():
    testtransform = transforms.Compose(
    [transforms.ToTensor(),
        transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.201)),
    ])
    subset = list(range(0,1000))
    testset = CIFAR10(".", train=False, download=True, transform=testtransform)
    testset = Subset(testset, subset)
    testloader = DataLoader(testset, batch_size=32)
    return  testloader

def test(net, testloader):
    correct, total= 0, 0
    with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU,torch.profiler.ProfilerActivity.CUDA], profile_memory=True, with_flops = True, use_cuda=True) as p:
        with torch.cuda.amp.autocast_mode.autocast(dtype=torch.float16):
            with torch.no_grad():
                for data in testloader:              
                    images,labels = data[0].to(DEVICE), data[1].to(DEVICE)
                    outputs = net(images)
                    _, predicted = torch.max(outputs.data, 1)
                    total += labels.size(0)
                    correct += (predicted == labels).sum().item()
    print(p.key_averages().table(sort_by="cuda_time_total"))   
    accuracy = correct / total
    return accuracy

net = torch.hub.load(repo_or_dir="chenyaofo/pytorch-cifar-models", model="cifar10_resnet20", pretrained= True)
net.eval()
net = net.to(DEVICE)

testloader = load_data()

start = time.time()
accuracy = test(net, testloader)
end = time.time()
print("Accuracy:",accuracy, "\nTime taken:", end-start)
</code></pre>
<p>The tests have been carried out on Python 3.8, PyTorch 1.10, torchvision 0.11, CUDA 10.2.</p>
Maskrcnn_resnet50_fpn normalization

cosmosaa: <p>It looks like the images are not normalized when training maskrcnn_resnet50_fpn. See <a href="https://github.com/pytorch/vision/blob/main/references/detection/presets.py#L7" rel="noopener nofollow ugc">https://github.com/pytorch/vision/blob/main/references/detection/presets.py#L7</a> where <code>hflip</code> is the default argument for the training recipe defined here <a href="https://github.com/pytorch/vision/tree/main/references/detection#mask-r-cnn" rel="noopener nofollow ugc">https://github.com/pytorch/vision/tree/main/references/detection#mask-r-cnn</a></p>
<p>Is it intentional? Why is normalization not applied?</p>
<p>Edit: I also see it mentioned <a href="https://pytorch.org/vision/main/auto_examples/plot_visualization_utils.html#visualizing-bounding-boxes" rel="noopener nofollow ugc">here</a></p>
<blockquote>
<p>Note that these models don’t require the images to be normalized, so we don’t need to use the normalized batch</p>
</blockquote>
<p>but still not sure why.</p>
Local_rank / local-rank

iksooman1: <p>I’ve looked into the torch.distributed of both torch1.13 (<a href="https://pytorch.org/docs/1.13/distributed.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.13 documentation</a>) and torch 2.0 (<a href="https://pytorch.org/docs/2.0/distributed.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.0 documentation</a>). I noticed that the variable “local_rank” has been modified to “local-rank”. Is this an intentional change or an error?</p>

agu: <p>This looks like it was a result of a better-engineering change: <a href="https://github.com/pytorch/pytorch/pull/94505" class="inline-onebox" rel="noopener nofollow ugc">[BE] Prefer dash over underscore in command-line options by XuehaiPan · Pull Request #94505 · pytorch/pytorch · GitHub</a></p>
java.lang.UnsatisfiedLinkError: dlopen failed: library "libpytorch_jni.so" not found

babysudal: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5846367123279ca8538f8ec7b9fa7f738e9ccae.png" data-download-href="https://discuss.pytorch.org/uploads/default/d5846367123279ca8538f8ec7b9fa7f738e9ccae" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5846367123279ca8538f8ec7b9fa7f738e9ccae.png" alt="image" data-base62-sha1="usRlbVno5UP27l4VeM7YdZ7V5ue" width="690" height="381" data-dominant-color="383133"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">983×544 18.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I can’t know why this error happens</p>
CUBE tensor get 6 face tensor then convert into (1,6xN)

Yong_Guang_Li: <p>Is there a convenient way to get the values of the six sides of the cube tensor, such as the tensor of NxNxN, get the tensor of NxN with 6 sides, and form the 1d tensor of N values from the largest value of each row in NxN?</p>

ptrblck: <p>Could you write a (slow) reference code which we could use for a proper solution to compare against, please?</p>

Yong_Guang_Li: <p>Certainly! Here’s a reference code( from chatgpt) that provides a slower, but straightforward solution for extracting the values of the six sides of a cube tensor and forming a 1D tensor with the largest value from each row in the extracted NxN tensors:</p>
<pre><code class="lang-auto">import torch

# Assume you have an NxNxN cube tensor named cube_tensor
N = cube_tensor.size(0)  # Get the size of the cube tensor

# Extract each side (NxN tensor) using slicing
front_side = cube_tensor[:, :, 0]
back_side = cube_tensor[:, :, -1]
top_side = cube_tensor[:, 0, :]
bottom_side = cube_tensor[:, -1, :]
left_side = cube_tensor[0, :, :]
right_side = cube_tensor[-1, :, :]

# Form a 1D tensor with the largest value from each row in each side
front_largest_per_row = torch.max(front_side, dim=1).values
back_largest_per_row = torch.max(back_side, dim=1).values
top_largest_per_row = torch.max(top_side, dim=1).values
bottom_largest_per_row = torch.max(bottom_side, dim=1).values
left_largest_per_row = torch.max(left_side, dim=1).values
right_largest_per_row = torch.max(right_side, dim=1).values

# Concatenate the largest values from each side into a single 1D tensor
largest_values = torch.cat([front_largest_per_row, back_largest_per_row,
                            top_largest_per_row, bottom_largest_per_row,
                            left_largest_per_row, right_largest_per_row])

</code></pre>

Yong_Guang_Li: <p>Is there a more concise way to implement？ <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Change nn.Parameter type to nn.Module type

core-not-dumped: <p>I want to replace the parameters inside a model with a custom nn.Module layer that I created. Below is a very simplified example of the code I desire.</p>
<p>Code:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

class change_to_layer(nn.Module):
  def __init__(self):
    super().__init__()
    self.w = nn.Parameter(torch.randn(100, 100))
  
  def __mul__(self, other):
    return self.forward(other)
  
  def __rmul__(self, other):
    return self.forward(other)

  def forward(self, x):
    return x @ self.w


class simple_model(nn.Module):
  def __init__(self):
    super().__init__()
    self.fc1 = nn.Linear(100, 100)
    self.scale = nn.Parameter(torch.ones(1))
    self.fc2 = nn.Linear(100, 100)

  def forward(self, x):
    x = self.fc1(x)
    x = self.scale * x
    x = self.fc2(x)
    print(x)


model = simple_model()

model.scale = change_to_layer()  # change nn.Parameter to nn.Module (error occurs)

input = torch.randn(100)
print(model(input))
</code></pre>
<p>However, I encounter the following error.</p>
<p>Error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-15-1a4295f999b1&gt; in &lt;cell line: 33&gt;()
     31 model = simple_model()
     32 
---&gt; 33 model.scale = change_to_layer()
     34 
     35 input = torch.randn(100)

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in __setattr__(self, name, value)
   1633         elif params is not None and name in params:
   1634             if value is not None:
-&gt; 1635                 raise TypeError("cannot assign '{}' as parameter '{}' "
   1636                                 "(torch.nn.Parameter or None expected)"
   1637                                 .format(torch.typename(value), name))

TypeError: cannot assign '__main__.change_to_layer' as parameter 'scale' (torch.nn.Parameter or None expected)
</code></pre>
<p>How can I change the type of a class variable?<br>
Or directly change nn.Parameter to nn.Module?</p>

ptrblck: <p>You could delete the parameter first via <code>del model.scale</code> before assigning the new module to this attribute.</p>
Tensor's shape error when pass tensor[...,i] into thread i and replace the elements in it

GIS-PuppetMaster: <p>I’m writing a cpp extension to sample different columns of a table, the sampling process on each column is independent. So I hope to speed it up with C++ multithreading. My code is like this:</p>
<pre><code class="lang-auto">//python
mysampler.sample(tuples, new_tuples,new_preds,self.columns_size, self.has_nones, num_samples)

//c++ interface
void
sample(const torch::Tensor&amp; tuples, torch::Tensor&amp; new_tuples, torch::Tensor&amp; new_preds, const std::vector&lt;int&gt;&amp; columns_size, std::vector&lt;bool&gt;&amp; has_nones, int num_samples){
    unsigned long long col_num = columns_size.size();
    std::vector&lt;std::thread&gt; thread_list;
    for (int i=0;i&lt;col_num; ++i){
        torch::Tensor new_tuple_i = new_tuples.index({"...", i});
        torch::Tensor new_pred_i = new_preds.index({"...", Slice(i*5,(i+1)*5)});
        std::cout&lt;&lt;"col:"&lt;&lt;i&lt;&lt;"new_pred_i.shape:"&lt;&lt;new_pred_i.sizes()&lt;&lt;std::endl;
        // shape is correct here
        // I didn't do anything in the thread, just print the shape of new_pred_i in the thread, and the shape is always smaller than it should be (except for the first two threads)
        std::thread t(sample_i, i, tuples.index({"...", i}), std::ref(new_tuple_i), std::ref(new_pred_i), columns_size[i], 0, num_samples, has_nones[i], num_samples, -1, nullptr);
        thread_list.push_back(std::move(t));
    }

void
sample_i(int i, const torch::Tensor&amp; tuples, torch::Tensor&amp; new_tuples, torch::Tensor&amp; new_preds, int column_size, int start_idx, int end_idx, bool has_none, int num_samples, int first_pred, const torch::Tensor* first_mask) {
    try {
        std::cout&lt;&lt;"col:"&lt;&lt;i&lt;&lt;"new_preds.shape:"&lt;&lt;new_preds.sizes()&lt;&lt;std::endl;
        //...other stuff
       // I use things like torch::index_put_(new_preds.index({mask}), samples);
    }
   catch (std::exception &amp;e){
    std::court&lt;&lt;e.what()&lt;&lt;std::endl;
    }
}
</code></pre>
<p>I got output like this:</p>
<pre><code class="lang-auto">col:0new_preds.shape:[2048, 2, 5]          // correct
col:1new_preds.shape:[2048, 2, 5]          // correct
col:2new_preds.shape:[5]                        // incorrect
...all incorrect
</code></pre>
<p>The <code>tuples</code> is the tensor that I want to sample on, and the <code>new_tuples</code> and <code>new_preds</code> are the tensors where I put my result in.<br>
Their shape are: <code>tuples[batch, col_num], new_tuples[batch, 2, col_num], new_preds[batch, 2, col_num*5]</code>. So I index them on <code>dim=-1</code> and pass the result (as a reference) into each column’s thread.<br>
However, only the first threads receive the correct new_tuples and new_preds, the others have the wrong shape such as [539,], [4,], [].</p>
<p>I guess that these tensors are generated from on common tensor and share a common data storage, so maybe there are thread safety problems. However, adding mutex lock to every operation to these tensors is not working.</p>
Getting output of neural network in equation form

manasi_ps: <p>Hi.<br>
I am trying to get the output of the neural network layers (last second layer) in equation form; basically wanted to observe what type of basis functions the network has created in the process. I can get to show the output of the last second layer in the form of a plot; but was hoping to get it in mathematical form/equations.</p>
<p>Any leads will be appreciated. Thank you</p>

J_Johnson: <p>You might find it helpful to work on some basic neural networks in spreadsheets to develop an intuition for what is going on behind the scenes.</p>
<div class="youtube-onebox lazy-video-container" data-video-id="kCL065_0zTY" data-video-title="Machine Learning MNIST using a Neural Network in Excel" data-video-start-time="" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=kCL065_0zTY" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/kCL065_0zTY/maxresdefault.jpg" title="Machine Learning MNIST using a Neural Network in Excel" width="690" height="388">
  </a>
</div>

<p>Pytorch docs also can be helpful for understanding the function behind a given layer type. Take, for instance, the commonly used Linear layer:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cca7c7538133b47bc030c112c8e7703ac1170c1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9cca7c7538133b47bc030c112c8e7703ac1170c1" title="IMG_20230522_090450"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cca7c7538133b47bc030c112c8e7703ac1170c1_2_674x500.jpeg" alt="IMG_20230522_090450" data-base62-sha1="mn2jdCrNfMZ4EF8vxdTdKs958rL" width="674" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cca7c7538133b47bc030c112c8e7703ac1170c1_2_674x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9cca7c7538133b47bc030c112c8e7703ac1170c1_2_1011x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cca7c7538133b47bc030c112c8e7703ac1170c1.jpeg 2x" data-dominant-color="F4F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20230522_090450</span><span class="informations">1078×799 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here we see an input vector, <code>x</code>, is matrix multiplied by a weight matrix, <code>A</code> (that is transposed, hence the superscript <code>T</code>), and then a vector of biases added, <code>b</code>.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Linear.html</a></p>
Proper way to train an autoregressive RNN

Messiah: <p>Hi,</p>
<p>I implemented an RNN that steps through each element of a sequence and decides with some probability whether to use the ground-truth target or the output of the previous model prediction as the current input to predict the next timestep. In order to allow the model to learn from its own predictive errors when using the models output as input, I would like to backpropogate through the output-to-input connection induced by that choice by preserving it in the computation graph.</p>
<p>Is this the proper way to account for long-term predictive errors induced by an autoregressive model? When I apply this strategy, the performance is worse compared to when I do not backpropogate through the output-to-input connection and treat the previous model output as essentially a constant by <code>.detach()</code>ing it at each timestep. Should I also be optimizing the model output used as input tensor by including it in the optimizer instantiation?</p>
<p>Any insights are appreciated.</p>
Inference time on CPU varying massively with training weight choice

Wologman: <p>I’m trying to classify 128 x 313 images using EfficientNetV2.  (tf_efficientnetv2_s_in21k from timm).  Created from Mel Spectrograms.</p>
<p>I can’t understand why my inference speed is varying by a factor of up to 20, depending on which set of model weights I use.  Could somebody please explain what might cause this?  The relevant details I can think of sumarised below:</p>
<ul>
<li>
<p>In all cases they are weights I have trained with another notebook, then loaded from checkpoint.</p>
</li>
<li>
<p>Trained, and inference with float32 normalised onto [0, 255]</p>
</li>
<li>
<p>Trained on GPU, mixed precision but inference on CPU.</p>
</li>
<li>
<p>I don’t think there are memory issues</p>
</li>
<li>
<p>The inference batch size doesn’t seem to make much difference</p>
</li>
<li>
<p>Earlier I was getting inference completed in 9 seconds per 120 image files, now it takes 380 seconds, changing nothing but the checkpoint weights.</p>
</li>
<li>
<p>Using PyTorch Lightning, and not using a dataloader for inference.</p>
</li>
</ul>

ptrblck: <p>Try to use <code>torch.set_flush_denormal(True)</code> and check if this changes the performance (assuming you are using a x86 CPU supporting SSE3 instructions).</p>

Wologman: <p>This indeed was the solution,   thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>    I initially tried applying it to the inference notebook and only got a modest improvement.  It was setting it on the training process that did the trick.  That didn’t occur to me at first.</p>