akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to concatenate to a Tensor with a 0 dimension?

samarth-robo: <p>In Numpy I can do:</p>
<pre><code class="lang-auto">np.hstack((np.zeros((3, 0)), np.zeros((3, 3)))
</code></pre>
<p>and it would give me a 3x3 zero matrix.<br>
But in pytorch,</p>
<pre><code class="lang-auto">torch.cat((torch.zeros(3, 0), torch.zeros(3, 3)), dim=1)
</code></pre>
<p>gives me a run time error <code>RuntimeError: dim out of range - got 1 but the tensor is only 1D</code>. Because <code>torch.zeros(3, 0)</code> is actually a 3-element Tensor (as opposed to Numpy, where it is empty).</p>
<p>Now, <a class="mention" href="/u/smth">@smth</a> has said before that there are no 0 dimensional Tensors in pytorch (<a href="https://discuss.pytorch.org/t/for-loop-with-a-2d-matrix-of-size-0/3599/2?u=samarth-robo">For-loop with a 2D matrix of size 0</a>) but does anyone know of a solution to this problem, where for example the 0 size of the dim is being calculated in a loop?</p>
<p>For example if I have</p>
<pre><code class="lang-auto">for i in xrange(5):
  print torch.cat((torch.zeros(3, i), torch.zeros(3, 3)), dim=1)
</code></pre>
<p>the first iteration would give an error…</p>

Jingcai_Guo: <p>Hi, have you found the solution?</p>

samarth-robo: <p>Yes - apparently now (in version 0.3.0) you can create 0-dimensional tensors. For example,<br>
<code>torch.zeros(0, 0)</code> will give <code>[torch.FloatTensor with no dimension]</code>.</p>
<p>So now you can do <code>torch.cat((torch.zeros(0, 0), torch.zeros(3, 3)))</code>.</p>

richard: <p>You can also do <code>torch.Tensor()</code>.</p>

linerocks: <p>I do something like this to concatenate zero dimensional tensor to another tensor</p>
<pre><code class="lang-auto">A = torch.tensor([0]) # [0]
B = torch.tensor([1, 2, 3, 4]) # [1, 2, 3, 4]
C = torch.cat((A.view(1), B)) # [0, 1, 2, 3, 4]
</code></pre>

Dalin68: <p>A = torch.tensor(0)<br>
B = torch.tensor([1,2,3])<br>
C = torch.cat((A.reshape(1), B))</p>

milan_kalkenings: <p>this works best for me. thanks!</p>

Giorgio: <p>doesn’ work on gpu though… what would you suggest to use the tensor on the gpu?</p>

Giorgio: <p>This doesn’t work anymore in the latest pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"> any alternative? thanks</p>

cijerezg: <p>If you do torch.empty(10,0), then you can concatenate it to torch.rand(10,24)</p>

Giorgio: <p>Thanks I’ll try that</p>

juvekaradheesh: <p><a class="mention" href="/u/giorgio">@Giorgio</a> It should work with <code>torch.cat((torch.zeros(0, 3), torch.zeros(3, 3)))</code> All the other dimension except the concatenation dimension should be same, and as you can see <code>torch.zeros(0, 3)</code> and <code>torch.zeros(0, 0)</code> are same</p>
Sharing CUDA Tensors

Ali_Youssef: <p>Hello everyone!</p>
<p>So I have been getting this error and I tried researching more about what it means but I’m having a hard time understanding it.</p>
<p>So I have 2 data loaders, one for training and one for validation, each has num_workers =4.</p>
<p>Every 1000 iterations, I call the validation function and after the validation loader is over, I get the following error, although it does not terminate the code.</p>
<p>[W C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]</p>
<p>Setting num_workers to 0 for both data loaders seem to solve the issue but I don’t really understand why am I getting it with num_workers&gt;0.</p>
<p>Thank you!</p>
<p><strong>Edit: Even with just the training loader, setting num_workers to anything above 0 gives the same error after each epoch</strong></p>
Could not run 'aten::_slow_conv2d_forward' with arguments from the 'QuantizedCPU' backend

soudabeh_mousavi: <p>I was trying I was trying to quantize my model I mainly used model structure from here:<a href="https://gist.github.com/martinferianc/d6090fffb4c95efed6f1152d5fde079d" class="inline-onebox" rel="noopener nofollow ugc">Quantisation example in PyTorch · GitHub</a><br>
I copied a custom myConv 2d layer from the same Conv2d and added the original torch.nn code in the source.<br>
I used Eager mode post training static quantization. After convert(), I got the error when I tried to do inference.</p>
<p>NotImplementedError: Could not run ‘aten::_slow_conv2d_forward’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::_slow_conv2d_forward’ is only available for these backends: [CPU, CUDA, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMeta, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].</p>
<p>Could some help me please? Thanks!</p>

andrewor: <p>Hi Soudabeh, this is a common error a lot of our users have run into. Please see if this is helpful <a href="https://pytorch.org/docs/stable/quantization.html#common-errors" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.0 documentation</a>.</p>
<p>Best,<br>
-Andrew</p>
In C++, torch::nn::Sequential initializes its arguments from right to left

PatrickV: <p>I just found that <strong>torch::nn::Sequential</strong> does not evaluate its arguments in the same order as Python.<br>
Here is how I can show it. I create my own module <strong>myLinear</strong> that prints its name during initialization.</p>
<pre><code class="lang-auto">struct _myLinear : torch::nn::Module
{
    torch::nn::Linear  m_linear  { nullptr };
   
    _myLinear (const char *name, int size)
    {
        m_linear = register_module( name, torch::nn::Linear( size, size ) );
        std::cout &lt;&lt; name &lt;&lt; std::endl;
    }

    torch::Tensor forward (torch::Tensor x) { return m_linear( x ); }
};

TORCH_MODULE_IMPL( myLinear, _myLinear );
</code></pre>
<p>Now, if I create my sequential module</p>
<pre><code class="lang-auto">auto seq = torch::nn::Sequential(
                      myLinear( "l1", 10 ),
                      myLinear( "l2", 10 ) );
</code></pre>
<p>It will print</p>
<pre><code class="lang-auto">l2
l1
</code></pre>
<p>This clearly shows the arguments are evaluated from right to left !!!<br>
If we don’t know this, it could be a hidden source of bugs.<br>
Also, in my case, it is a problem to validate my C++ code, trying to get the same result as in Python.</p>
<p>To solve this, I now initialize the sequential modules with <strong>push_back</strong></p>
<pre><code class="lang-auto">auto seq = torch::nn::Sequential();
seq-&gt;push_back( myLinear( "l1", 10 ) );
seq-&gt;push_back( myLinear( "l2", 10 ) );
</code></pre>
<p>This should be clearly written in some libtorch documentation. Or is it and I did not find it?</p>
`RendezvousClosedError` when a node restarts and join the group

f_g: <p>Hi,</p>
<p>I use torch elastic to run jobs with multiple workers. I have been seeing cases where a worker restarts and joins back into the group, but quickly encounters <code>RendezvousClosedError</code> which causes it to fail. The rest of the group is still running no problem, but then because this one node failed, it causes the job to fail.</p>
<p>I use torchrun like: <code>torch.distributed.run --nproc_per_node=1 --node_rank=0 --nnodes=8:8 --master_addr=10.8.68.146 --master_port=2222 --rdzv_endpoint=10.8.68.146:2222 --rdzv_id=0 --rdzv_backend=c10d --rdzv_conf=join_timeout=3600,read_timeout=3600</code></p>
<p>Would appreciate if there’s any pointers to the setup. Thanks!</p>

H-Huang: <p>I assume since <code>nnodes=8:8</code> that means that min-max are both 8 nodes, so it tolerates no failures. If one of the node fails, then the whole group must restart. Shouldn’t the number of restarts be &gt;0 so that it tolerates some failures?</p>
Using pytorch with bazel

OliverFM: <p>I have tried to install torch in a bazel project using gazelle. I have run into some serious build issues. First, I needed to patch torch because of circular dependencies, however I am now stuck with linking issues.</p>
<p>I have created a repo with a minimal reproduction: <a href="https://github.com/OliverFM/pytorch_with_gazelle" class="inline-onebox" rel="noopener nofollow ugc">GitHub - OliverFM/pytorch_with_gazelle</a></p>
<p>To install torch and dependencies, run: <code>make generate_build_files</code> and then run <code>bazel run //test_project:test_project_bin</code>. The issue that I am getting is a linking error:</p>
<pre><code class="lang-auto">❯ bazel run //test_project:test_project_bin
INFO: Analyzed target //test_project:test_project_bin (1 packages loaded, 4 targets configured).
INFO: Found 1 target...
Target //test_project:test_project_bin up-to-date:
  bazel-bin/test_project/test_project_bin
INFO: Elapsed time: 0.757s, Critical Path: 0.13s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/test_project/test_project_bin
Traceback (most recent call last):
  File "/home/oliver/.cache/bazel/_bazel_oliver/a5974369252d7c7f580df32b3fcee8ce/execroot/build_file_generation_example/bazel-out/k8-fastbuild/bin/test_project/test_project_bin.runfiles/build_file_generation_example/test_project/__main__.py", line 3, in &lt;module&gt;
    import triton
  File "/home/oliver/.cache/bazel/_bazel_oliver/a5974369252d7c7f580df32b3fcee8ce/execroot/build_file_generation_example/bazel-out/k8-fastbuild/bin/test_project/test_project_bin.runfiles/pip_triton/site-packages/triton/__init__.py", line 9, in &lt;module&gt;
    import torch  # noqa: F401
  File "/home/oliver/.cache/bazel/_bazel_oliver/a5974369252d7c7f580df32b3fcee8ce/execroot/build_file_generation_example/bazel-out/k8-fastbuild/bin/test_project/test_project_bin.runfiles/pip_torch/site-packages/torch/__init__.py", line 229, in &lt;module&gt;
    from torch._C import *  # noqa: F403
ImportError: libcupti.so.11.7: cannot open shared object file: No such file or directory
</code></pre>
<p>Any suggestions on debugging this?</p>
Where to find mentor/expert for collaboration?

timur: <p>Hi folks, I’m trying to master PyTorch by myself, but as a self-learner I’m pretty slow at this. I’m using several courses on Udemy/YouTube + GPT if I don’t understand something, but overall I need a mentor.</p>
<p>I’m willing to share/pay in return. It’s hard to find quality teacher online.</p>
<p>Do you have any resources or maybe you know someone?</p>
How to compute loss between ground truth bounding box and bounding box computed from segmentation mask

Aayushktyagi: <p>I am trying to compute the loss between ground truth bounding box and bounding box which is generated from predicted segmentation mask. My function to compute the bounding box loss is as following:</p>
<pre><code class="lang-python">import torch.nn as nn
import torchvision.ops as ops


def get_bounding_boxes_from_masks(segmentation_masks, device):
    bbox_tensor_list = []
    for i in range(segmentation_masks.shape[0]):
            if torch.sum(segmentation_masks[i]) == 0:
                bounding_boxes = torch.tensor([[0.0, 0.0, 1.0, 1.0]], dtype=torch.float32).to(device)
            else:
                bounding_boxes = ops.masks_to_boxes(segmentation_masks[i])
            bbox_tensor_list.append(bounding_boxes)
    concatenated_bbox = torch.cat(bbox_tensor_list, dim = 0)
    return concatenated_bbox
</code></pre>
<p>Once I have the bounding boxes I compute the loss between ground truth bounding boxes (bboxes) and bounding boxes from segmentation mask (mask_bboxes). My segmentation mask contains single object only.</p>
<pre><code class="lang-python">mse_loss = nn.MSELoss()
for step, (image, gt, boxes) in enumerate(tqdm(train_dataloader)):
      bboxes = Variable(bboxes, requires_grad=False)
      
      # mask_predictions are coming from segmentation model 
      mask_predictions = model(image, boxes)
      
      mask_predictions = torch.sigmoid(mask_predictions) 
      binary_masks = mask_predictions &gt; 0.5
      mask_bboxes = get_bounding_boxes_from_masks(binary_masks, device=device)
      mask_bboxes = Variable(mask_bboxes, requires_grad=True)
      loss = mse_loss(mask_bboxes.to(device), bboxes.to(device))
      print(loss.item(), loss.grad())
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
      epoch_loss += loss.item()
</code></pre>
<p>I am getting loss.grad as “None” and loss is constant after each epoch (not even decimal change). I am not able to understand what’s happening here. Can anyone help?</p>
<p>Let me know if any other information is required.</p>
<p>Thanks</p>

nwn: <p>Hello Aayushktyagi,<br>
I suppose your code is breaking the gradient function. In the first if-statement you are creating a new tensor which seems to be detached from the segmentation mask, therefore, from the “whole” gradient function tracking that is required to update your model’s parameters.<br>
To be sure, use a debugger and follow the gradient flow.</p>
<p>Best,<br>
Luis</p>

Aayushktyagi: <p>Hey <a class="mention" href="/u/nwn">@nwn</a> I tried removing the if-statement but I am still getting the same issue i.e. loss is constant and loss.grad is None.<br>
Is there anything else that can be tried?</p>
<p>Thanks<br>
Aayush</p>

nwn: <p>I would start a debugger an set break point(s) after your forwards pass, then go line by line through your code and monitor the gradient functions/ check those for consistency.</p>
Any PyTorch professionals mind taking part in an academic survey?

cnorval: <p>Hi there,</p>
<p>We’re doing some academic research about algorithmic systems at the University of Cambridge, and we’re looking to reach out to professionals that work with ML/AI systems. Given that this will include some PyTorch devs, we thought we’d try asking here!</p>
<p>If you’re a professional working with PyTorch or other types of ML/AI systems, and happen to have a spare ~20 minutes, would you mind answering some questions? It’d be a massive help!</p>
<p>Here’s the link: <a href="https://cambridge.eu.qualtrics.com/jfe/form/SV_3n6RuowNogZKG34" rel="noopener nofollow ugc">https://cambridge.eu.qualtrics.com/jfe/form/SV_3n6RuowNogZKG34</a></p>
<p>PS: I hope this is ok to post! Feel free to delete if not <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Cheers,<br>
Chris</p>
Writing a function in pytorch that reads all the images in a folder

amy2: <p>I want to use pytorch to write a simple function that reads images in a folder and then visualize them,this is the function I used :</p>
<blockquote>
<p>import matplotlib.pyplot as plt<br>
import os<br>
import torch<br>
image_path=“files”<br>
filename = [name for name in os.listdir(image_path)]<br>
print(filename)<br>
batch_size = len(filename)<br>
batch = torch.zeros(batch_size, 3, 240, 320, dtype=torch.uint8)<br>
for i, file in enumerate(filename):</p>
</blockquote>
<pre><code>    batch[i] = torchvision.io.read_image(os.path.join(image_path, file))
    fig = plt.figure(figsize=(8, 2))
    for i in range(batch.shape[0]) 
        ax = fig.add_subplot(1, 3, i+1)
        ax.imshow(batch[i].permute(1, 2, 0))
</code></pre>
<p>I got an error “ValueError: num must be an integer with 1 &lt;= num &lt;= 3, not 4”<br>
I suppose it’s because it only shows 3 images ,<br>
But isn’t batch[0] suppose to have the value of the batch_size in my case 252 as I have a 252 20*240 RGB images ?<br>
and when I try to print the i I got 0 why is that can anyone help me ?</p>
Best way to validate multiple networks in parallel?

setsailforfail: <p>I am trying to find the best hyperparameters for a simple CNN. To achieve this, I want to, for example, evaluate 100 hyperparameter configurations on a multiple budgets, for example on 5, 10 and 15 epochs.</p>
<p>What would be the best way to achieve this? That is, how I can most efficiently run multiple trainings in parallel? So far I have been using something like this:</p>
<pre><code class="lang-auto">   # Define the queue, which is passed to the train function and the final validation score is inserted
    queue = Queue()

    # samples n_configs from the configuration space
    samples = configspace.sample_configuration(size=n_evals)

    # maps a process to the required budget
    mapping = {}

    # define the budgets on which the model should be trained on, e.g. epochs
    budgets = [5, 10, 15]

    # number of configs we want to evaluate
    n_evals = 100

    # number of workes we want to use in parallel
    n_workers = 3

    # evaluate the configs 
    for budget in budgets:
        processes = []
        configs_processed = 0
        while configs_processed &lt; n_evals - n_workers:
            for _ in range(n_workers):
                p = Process(target=train_funciton, args=(samples[configs_processed], budget, queue))
                # the mapping is required, so we know what processes was associated with which budget
                mapping[p] = budget
                configs_processed += 1
                p.start()
                processes.append(p)

            for processs in processes:
                processs.join()
                val_error = queue.get()
                results[mapping[processs]].append(val_error)
</code></pre>
<p>I am pretty sure there must be a better way, and I hope someone can show me how it can be done better.</p>
Pytorch/Pytorch_lightning DDP

Bhavay_Malhotra: <p>Hi everyone, I’m using PyTorch_lightning DDPPlugin module for training my model on multiple GPUs.<br>
I used this code to initialise the plugin.<br>
plugs = DDPPlugin(find_unused_parameters=True).</p>
<p>I first set the find_unused_parameters as False but it gave an error and asked to turn it True. When I turned it True, it now gives an error stating -<br>
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the <code>forward</code> function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple <code>checkpoint</code> functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.</p>
<p>My PyTorch version is 1.10.0 and PyTorch_lightning version is 1.5. I cannot change my library versions as I’m working on someone’s else code and it might break the pipeline. Is this issue solved for this version of PyTorch? I saw some posts here that its still in progress and cannot find a solution online.</p>

H-Huang: <p>Did you try calling the <code>_set_static_graph()</code> method as mentioned in the error message?</p>
<p>The <code>find_unused_parameters=True</code> is model dependent and should be set when there are some parts of the model where not all of the parameters participate in the forward pass. I assume that this is the case for your model?</p>

Bhavay_Malhotra: <p>Hi <a class="mention" href="/u/h-huang">@H-Huang</a> , I solved the problem. Thanks</p>
How to find the cause of function runtime instability

BearBiscuit05: <p>I now have a simple feature extraction function. The process of the function involves transferring results calculated on the GPU to the CPU. Subsequently, the computed results need to be combined with an additional ID to form a collection of IDs (approximately 256k IDs). Then, I will extract corresponding data from the “feat” data. However, during my testing, I found that the computation time of this function is highly unstable. In my printout, the computation time varies from 0.005s to 0.026s. I would like to understand the reasons behind this issue and explore possible optimizations to stabilize this computation time.</p>
<pre><code class="lang-auto">def featMerge(self,nodeids):    
        toCPUTime = time.time()  
        nodeids = nodeids.to(device='cpu')
        print("to CPU time {}s".format(time.time()-toCPUTime))
        catTime = time.time()
        temp_merge_id[1:] = nodeids
        print("cat time {}s".format(time.time()-catTime))
        featTime = time.time()
        test = self.feats[temp_merge_id]       
        print("feat merge {}s".format(time.time()-featTime))
        print("all merge {}s".format(time.time()-toCPUTime))
        return test
</code></pre>

ptrblck: <p>CUDA operations are executed asynchronously while the D2H transfer is blocking. Without explicit synchronizations blocking calls will accumulate CUDA execution times and will report invalid timings.</p>

BearBiscuit05: <p>Thanks a lot for the reply, it helped me a lot. In addition, I would like to ask, is there a better way to extract the features corresponding to the specified index from feat? In my project, this is a running bottleneck</p>
<pre><code class="lang-auto">test = self.feats[temp_merge_id]
</code></pre>

ptrblck: <p>Indexing is the right operation to get the feature. Did you properly synchronize the code and narrowed down the bottleneck to this op? If so, how slow is it compared to other operations?</p>

BearBiscuit05: <p>Yes, after analysis, it’s true that the bottleneck I want to compute is indeed at this point. The time required for my other operations is approximately 0.02 seconds, while the feature extraction step takes about 0.03 seconds of my time (finding the positions required among 1.2 million indices for 256,000 locations, and these locations might have duplicates). Compared to other existing implementations, my self-built solution is indeed much slower in terms of performance. I’m not sure if there are areas that can be optimized.</p>
Torch.nn.functional.interpolate() slow on MPS device (sometimes)

iwasserman: <p>I’m developing an inference pipeline on MPS, and the last step is a small upscale from a shape of <code>(1, 25, 497, 497)</code> to <code>(1, 25, 512, 512)</code>. To do so, I’m using <code>torch.nn.functional.interpolate()</code>. I’ve noticed that this operation takes over 3 seconds on an MPS device (M1 Pro), whereas it takes less than 1/10 of a second on a CUDA device (T4). While a difference in performance is expected across the board between these two GPUs, this difference seems wildly disproportionate.</p>
<p>I’ve also noticed that if I just create an arbitrary tensor of size <code>(1, 25, 497, 497)</code> and perform the same interpolation outside of the network or in a different Jupyter cell, the interpolation only takes 1/10 of a second on MPS.</p>
<p>Does anyone have any idea what’s causing this performance issue or what steps I could take to identify the issue?</p>

felix-lindgren: <p><a class="mention" href="/u/iwasserman">@iwasserman</a> I was trying to run an existing model, however it was very slow when using MPS and I also found that <code>torch.nn.functional.interpolate()</code> was the culprit, similar to what you’re describing. It was only during model inference and not when only running the interpolation by itself, same as for you.</p>
<p>Did you manage to find the issue?</p>

iwasserman: <p>No, sorry. I never solved this and have just accepted it as a limitation of the MPS implementation.</p>
Simple question about CIFAR10 dataset in torchvision.datasets module

seberino: <p>I’m following the PyTorch CNN example that uses the CIFAR10 dataset here:</p>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 2.0.1+cu117 documentation</a></p>
<p>The shape of the CIFAR10 training data seems really odd.</p>
<p>Here is the size of the images: torch.Size([4, 3, 32, 32])<br>
Here is the size of the labels: torch.Size([4])</p>
<p>I understand the images are 32x32 pixels with 3 channels.  What I don’t understand is the 4 in the images dimensions.  Why are the input image sizes torch.Size([4, 3, 32, 32]) instead of<br>
torch.Size([3, 32, 32])??</p>
<p>Lastly, why isn’t the label simply the index of the correct class?  Why is it 4 class numbers?</p>
<p>Thanks.</p>
<p>P.S.  You can print the CIFAR10 data sizes with this code:</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms

trans = [torchvision.transforms.ToTensor(),
         torchvision.transforms.Normalize(3 * [0.5], 3 * [0.5])]
trans = torchvision.transforms.Compose(trans)
train = torchvision.datasets.CIFAR10(root      = "training_data",
                                     download  = True,
                                     train     = True,
                                     transform = trans)
loader_train = torch.utils.data.DataLoader(train,
                                           batch_size = 4,
                                           shuffle    = True)

for e in loader_train:
           print(len(e), e[0].shape, e[1].shape)
</code></pre>

nwn: <p>Hello Seberino,<br>
this is nothing specific to CIFAR10 and would look very similar if you had used another dataset. If you use a DataLoader usually the batch dimension is the first (0-th) dimension that apperars. In your case you picked a batch size of 4, in other words, your dataloader loads 4 images at once.</p>

seberino: <p>nwn</p>
<p>Oh thanks!  That totally makes sense!  So the first input size dimension is the<br>
<strong>batch size</strong>.</p>
<p>And why aren’t the labels just single numbers?  It appears the labels<br>
are the “top 4” answers?  That seems odd.</p>
<p>cs</p>

nwn: <p>Yes, exactly, it is the batch size. I suppose that it returns 4 labels which would match your batch size, note that you load 4 images at once which means there should be an equally large number of labels, i.e. 4. If you want to be sure please the print message of your code.</p>

seberino: <p>Thanks again.  <strong>All</strong> my confusion was because of batches!<br>
You cleared everything up!</p>
When i run code i found error

hizen: <pre><code class="lang-auto">from torch.utils.data import Dataset
from sklearn.datasets import fetch_openml

# Load data from https://www.openml.org/d/554
X, y = fetch_openml('mnist_784', version=1, return_X_y=True)
print(X.shape)
</code></pre>
<pre><code class="lang-auto">class SimpleDataset(Dataset):
        
    def __init__(self, X, y):
        super(SimpleDataset, self).__init__()
        self.X = X
        self.y = y
    
    def __getitem__(self, index):
        #This "work" could have gone in the constructor, but you should get into 
        inputs = torch.tensor(self.X[index,:], dtype=torch.float32)
        targets = torch.tensor(int(self.y[index]), dtype=torch.int64)
        return inputs, targets 

    def __len__(self):
        return self.X.shape[0]
#Now we can make a PyTorch dataset 
dataset = SimpleDataset(X, y)
</code></pre>
<pre><code class="lang-auto">print("Length: ", len(dataset))
example, label = dataset[0]
print("Features: ", example.shape) #Will return 784
print("Label of index 0: ", label)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9fcfde4b5c340e18e620c052ffe366fcbe2a155.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9fcfde4b5c340e18e620c052ffe366fcbe2a155" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9fcfde4b5c340e18e620c052ffe366fcbe2a155.png" alt="Capture" data-base62-sha1="zFuVc8kwK34e6eQBzvCyadNTIXz" width="690" height="402" data-dominant-color="F7F6F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1065×621 30.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The screenshot shows an indexing error raised by <code>pandas</code> so check where a <code>pandas.DataFrame</code> is used and how it’s being indexed to narrow down the root cause of the wrong indexing value.</p>

hizen: <p>This is my notebook link please help me solve my error <a href="https://colab.research.google.com/drive/1tzYQlIktxWVlJiXwSHdaJhJhNdoY2du0?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1tzYQlIktxWVlJiXwSHdaJhJhNdoY2du0?usp=sharing</a>.</p>

olecram: <p>Try indexing using iloc:</p>
<p>inputs = torch.tensor(self.X.iloc[index,:], dtype=torch.float32)<br>
targets = torch.tensor(int(self.y.iloc[index]), dtype=torch.int64)</p>
What will happen if I set requires_grad = False?

1160300929: <p>suppose three Linear layers A,B,C,and I set B’s requires_grad to False,can C’s loss bp to A?</p>

ptrblck: <p>Yes, intermediate parameters, which don’t require gradients, do not stop the backpropagation, if some earlier parameters need gradients.</p>

1160300929: <p><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=9" title=":flushed:" class="emoji" alt=":flushed:">But if the intermediate parameter don’t require gradient,how do the earlier parameters gets loss bp from the later parameters?I don’t understand the process</p>

1160300929: <p>Does requires_grad  mean compute grads but do not flush them?</p>

ptrblck: <p>The parameter gradients, which are not needed, won’t be computed (their <code>.grad</code> attribute won’t be updated), but the gradient calculation will continue, if it’s needed for earlier layers.</p>

ndvbd: <p>I think you can set requires_grad=False only to leaf nodes, so is the question valid?</p>

ptrblck: <p>Yes, this 4 year old question was valid. The parameters of an intermediate modules were frozen and the question was if this would allow Autograd to properly backpropagate to the first module, which it does.</p>
Dataloader bottleneck even if data is loaded on RAM

donde: <p>Hi there,</p>
<p>I’m running some training jobs on a subset of 100 classes from ImageNet. I wrote a custom dataset class to load all data into RAM at the beginning of training to decrease training time, however the training time per epoch is similar to using an ImageFolder class (which as I understand it lazily loads images from disk).</p>
<p>Profiling with cProfile seems to have most time spent in a method “aquire” of “_thread.lock” objects. Since training data is already loaded in memory, why does it seem like the dataloader is still a bottleneck?</p>
<p>Is there anything I’m missing here?</p>
<p>Thanks in advance.</p>
System RAM crashes while training RNN

hanoch: <p>after a few minutes of training, system ram used inflates and it crashes.<br>
please see colab file here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1O6IBFZw8BRRFtytFpxDzdcfvlilDhYNA?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/0d019cccc9902b1938454fb955bec25a/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1O6IBFZw8BRRFtytFpxDzdcfvlilDhYNA?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1O6IBFZw8BRRFtytFpxDzdcfvlilDhYNA?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>why is this happening?</p>
RuntimeError: derivative for aten::_histogramdd_from_bin_cts is not implemented

naomishmish: <p>Hi,</p>
<p>I’m trying to write loss function that maximize the mutual information between 2 images (the image that was created from a model and the target image).<br>
the calculation of mutual information contains 2d histogram, and I get the error that there is no implementation for the derivative of that function.<br>
is there anything that I can do to fix it?</p>
<p>this is the code for calculation the mutual information loss:</p>
<p>class MaxMutualLoss(th.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MaxMutualLoss, self).<strong>init</strong>()</p>
<pre><code>def forward(self, inputs, targets):
    th2d = th.stack((targets.ravel(),inputs.ravel()), -1)
    hist_2d, _, = th.histogramdd(th2d.cpu(), bins=20)
    pxy = hist_2d / float(th.sum(hist_2d))
    px = th.sum(pxy, axis=1) # marginal for x over y
    py = th.sum(pxy, axis=0) # marginal for y over x
    px_py = px[:, None] * py[None, :] # Broadcast to multiply marginals
    # Now we can do the calculation using the pxy, px_py 2D arrays
    nzs = pxy &gt; 0 # Only non-zero pxy values contribute to the sum
    return -1 * th.sum(pxy[nzs] * th.log(pxy[nzs] / px_py[nzs]))
</code></pre>
<p>the error that I’m getting when applying loss.backward():<br>
RuntimeError: derivative for aten::_histogramdd_from_bin_cts is not implemented</p>
<p>Thank you!<br>
Naomi</p>
Question about `Warning unrecognized nn.Module`

AprilDomLin: <p>I tried to use the method <code>autograd.grad</code> to calculate the sum of gradients of outputs. But I encountered the warnings that are like <code>Warning unrecognized nn.Module : LayerNorm</code>, <code>Warning unrecognized nn.Module : Identity</code> and  <code>Warning unrecognized nn.Module : GELU</code>.</p>
<p>Could anyone help me with this problem? Or show me some ideas on how to solve it.</p>
<p>The version of <code>pytorch</code> is 1.12.0.</p>

ptrblck: <p>Are you seeing the same warning in the latest stable or nightly PyTorch release?</p>

AprilDomLin: <p>Unfortunately， it raises the same warning in the <code>2.0.1+cu118</code>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fdaffbdab9f54c4c833f67cc7c9e6065a8c5c668.png" data-download-href="https://discuss.pytorch.org/uploads/default/fdaffbdab9f54c4c833f67cc7c9e6065a8c5c668" title="1691905334964"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fdaffbdab9f54c4c833f67cc7c9e6065a8c5c668.png" alt="1691905334964" data-base62-sha1="AcdQiL1irBLaEclYCJiwGe4e53a" width="690" height="451" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1691905334964</span><span class="informations">716×468 31.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AprilDomLin: <p>Is there any being method which could solve this warning? I will get a wrong explaing becasue of uncorrect gradient. <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>
Issue with ImageNet training set

44_Pashikanti_Pranit: <p>Hello guys,<br>
I am trying to run resnet18 on Imagenet and here is the code<br>
and i am getting a validation accuracy over 82.09% but the expected accuracy is 69.76%.<br>
Is there a issue in my code?</p>
<pre><code class="lang-auto">import deeplake
import torch
import torchvision
from torchvision import transforms
from tqdm import tqdm

# Load pre-trained ResNet-18 model
model = torchvision.models.resnet18(pretrained=True)  # Load the pre-trained weights
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()  # Set the model to evaluation mode

correct = 0
total = 0

# Iterate over the test_loader for validation
with torch.no_grad():
    for x, y in tqdm(test_loader):
        x = x.to(device)
        y = y.to(device)

        y_pred = model(x)
        predicted_labels = y_pred.argmax(dim=1)

        correct += (predicted_labels == y).sum().item()
        total += y.size(0)  # Increment the total count by batch size

accuracy = correct / total
print("Validation Accuracy: {:.2%}".format(accuracy))
</code></pre>
Assignment to split tensor causes memory leak

DominikP: <p>I want to train a network for smoothing a point cloud.<br>
It is applied independently for each point and has the surrounding points as input.<br>
The function below performs the actual smoothing by splitting the points (<em>x</em>) and the neighborhoods (<em>y</em>) into batches and then calling the model.</p>
<pre><code class="lang-auto">def do_smoothing(x, y, model, batch_size):
    model.eval()
    torch.no_grad()   # Make sure that no gradient memory is allocated
    print('Used Memory: {:3f}'.format(torch.cuda.memory_allocated('cuda') * 1e-9 ))
    xbatches = list(torch.split(x, batch_size))
    ybatches = list(torch.split(y, batch_size))
    assert(len(xbatches) == len(ybatches))
    print('Used Memory: {:3f}'.format(torch.cuda.memory_allocated('cuda') * 1e-9 ))
    for i in range(len(xbatches)):
        print("Batch {} [Memory: {:3f}]".format(i + 1, torch.cuda.memory_allocated('cuda') * 1e-9))
        tmp = self.forward(_xbatches[i])
        xbatches[i] += tmp
        del tmp
        print("Batch {} [Memory: {:3f}]".format(i + 1, torch.cuda.memory_allocated('cuda') * 1e-9))
        print("------------------------------------")
    return x
</code></pre>
<p>Unfortunately, the memory consumption is very high leading to an out of memory error. The console output is:</p>
<pre><code class="lang-auto">Used Memory: 0.256431
Used Memory: 0.256431
Prior batch 1 [Memory: 0.256431]
Post batch 1 [Memory: 2.036074]
------------------------------------
Prior batch 2 [Memory: 2.036074]
 -&gt; Out of memory error
</code></pre>
<p>If I remove the line <code>xbatches[i] += tmp</code> the allocated memory is not changing (as expected).<br>
If I also remove the line <code>del tmp</code> on the other hand the code once again allocates huge amounts of GPU memory.</p>
<p>I assume that <a href="https://pytorch.org/docs/stable/torch.html#torch.split" rel="nofollow noopener">torch.split</a> creates views onto the tensor and therefore the update should not use additional memory.<br>
Did I miss anything or is this unintended behavior?</p>
<p>I am using pytorch 1.4.0 with CUDA 10.1 on a Windows 10 machine.</p>
<p>Thanks in advance for any tips.</p>

ptrblck: <p>Could you try to use <code>torch.no_grad()</code> in a <code>with</code> statement:</p>
<pre><code class="lang-python">model.eval()
with torch.no_grad():
    xbatches = ...
</code></pre>
<p>If I’m not mistaken, your current approach shouldn’t change the gradient behavior.</p>

DominikP: <p>Thank you very much, this indeed fixes the problem.</p>

Riccorl: <p>I know it is an old question, but I have the same problem. The proposed solution for me doesn’t work because I need to split tensors during training. Is there a way to split a tensor without getting OOM errors?</p>

nuoyi_ma: <p>Have you found a solution?</p>
Unable to export pytorch model with dynamically changing **kernel** shape to ONNX

Adrien88: <p>I have a PyTorch model that performs correlation between the <strong>dynamically changing shapes</strong> of template and search images. For example:</p>
<p>The pytorch model code:</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()

    def forward(self, template, search):
        out = torch.nn.functional.conv2d(search, template)

        return out
</code></pre>
<p>The onnx export code:</p>
<pre><code class="lang-python">dummy_inputs = (dummy_template, dummy_search)
input_names = ["template", "search"]
output_names = ["outputs"]

dynamic_axes = {
    "template": {
        2: "height",
        3: "width"
    },
    "search": {
        2: "height",
        3: "width"
    }
}

torch.onnx.export(model,
                  args=dummy_inputs,
                  f=onnx_path,
                  input_names=input_names,
                  output_names=output_names,
                  dynamic_axes=dynamic_axes,
                  opset_version=11,
                  export_params=True)
</code></pre>
<p>I got this error <code>RuntimeError: Unsupported: ONNX export of convolution for kernel of unknown shape.</code></p>
<p>Cause I use <code>torch.nn.functional.conv2d</code> for the correlation operation. But the input shape of template is dynamically changing, so I got the error above.</p>
<p>Currently, I have tried to implement the correlation manually.</p>
<pre><code class="lang-python">def corr(input: torch.Tensor, kernel: torch.Tensor) -&gt; torch.Tensor:
    channel = input.shape[1]
    in_h, in_w = input.shape[-2:]
    kh, kw = kernel.shape[-2:]
    output_width = (in_w - kw) + 1
    output_height = (in_h - kh) + 1

    output = torch.zeros(1, channel, output_height, output_width).to(input.device)

    for c in range(channel):
        for h in range(in_h - kh + 1):
            for w in range(in_w - kw + 1):
                input_window = input[0, c, h:h+kh, w:w+kw]
                kernel_window = kernel[0, c, :, :]
                corr = torch.sum(input_window * kernel_window)
                output[0, c, h, w] = corr

    return output
</code></pre>
<p>However, the results using this method is incorrect, possibly because ONNX doesn’t support the <strong>slice</strong> method (<code>input[0, c, h:h+kh, w:w+kw]</code>)</p>
<p>I want to know <strong>any</strong> possible solution.</p>
Message passing using sparse tensors

alexj: <p>Hi Everyone,</p>
<p>I am trying to implement a Message Passing NN without pytorch-geometric. I find it quite easy to do using normal tensors:<br>
torch.<strong>version</strong></p>
<pre><code class="lang-auto">a = [[1., 2.], [3., 4.]]
b = [[5., 6.], [7., 8.], [9., 10.]]
ta = torch.tensor(a)
tb = torch.tensor(b).t()
torch.matmul(ta, tb)

&gt; tensor([[17., 23., 29.],
&gt;        [39., 53., 67.]])

tba = torch.mul(ta.view((2, 2, 1), 3, 0), tb.view((1, 2, 3), 3, 0))
tba.shape

&gt; torch.Size([2, 2, 3])

tba.sum(1)

&gt; tensor([[17., 23., 29.],
&gt;        [39., 53., 67.]])
</code></pre>
<p>This way, I can replace <code>sum</code> with any other symmetric function (say <code>max</code>)</p>
<p>However, there is no way to handle a real graph’s adjaxcency matrix in a dense tensor. I cannot find a way to do the same with sparse tensors. Help?</p>
Best Practice for sampling from in-memory data

nwn: <p>Hello there,<br>
I am searching for a best practice regarding the sampling of in-memory data.<br>
Let’s assume there exists a tensor with shape [10, 5000, 1, 224, 224]  where 10 is the number of models, 5000 the number of samples and the last three dimensions are the dimensions per sample. The task is to sample from this tensor so that a cartain number of samples are presented e.g. to a model ensemble. A standard custom random batch sampler would look something like that:</p>
<pre><code class="lang-auto">class CustomSampler:

  def __init__(self, data, labels, batch_size):
          ...
  
  def get_index(self, ...):
          return torch.randint(0, n_samples, batch_size)

   def __next__(self):
          batch_idx = self.get_index()
          img = self.data[:, batch_idx, :]
          labels = self.labels[:, batch_idx]
          return (img, labels)
</code></pre>
<p>Note that data and labels are created by going through a big preprocessing pipeline resulting in the tensors already having been moved to the GPU.</p>
<p><strong>Is there a best/ better practice than the above mentioned approach to sample from data that was already loaded in memory and exists as one large tensor?</strong></p>
<p>Looking forward to you answers.</p>
<p>Best,<br>
Luis</p>