akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to manage cache invalidation

Bilel_Omrani: <p>When part of a datapipe is cached on disk with the <code>OnDiskCacheHolder</code> datapipe, how to manage cache invalidation?<br>
More specifically, say I have multiple files stored in a remote object storage. These files are fed into a datapipe using <code>FSSpecFileLister</code>, and cached locally using <code>.on_disk_cache</code>. I want to invalidate the cache and re-compute the datapipe when one or more remote files are changed, probably based on their hash.<br>
Is there a way to achieve something similar?</p>

nivek: <p>While <code>OnDiskCacheHolder</code> accepts an <code>extra_check_fn</code> to carry out extra validation given the file paths, I don’t think we have automatic cache invalidation at the moment. It may requires changes to our existing cache holder implementation or a separate DataPipe. Feel free to open an issue on GitHub or a PR with what you end up implementing. We are happy to discuss further.</p>

Bilel_Omrani: <p>Thank you, I submitted an issue <a href="https://github.com/pytorch/data/issues/1111" rel="noopener nofollow ugc">here</a></p>
How to make the CPU block instead of spin on cudaMemcpyAsync/cudaStreamSynchronize

Puneeth_Naik: <p>I am studying a pytorch model. I am running it on a CPU/GPU heterogeneous system. So whenever there is a dynamic control flow; i.e., an if statement whose predicate depends on the value of a tensor on the GPU), the CPU synchronized with the GPU waiting for the data value to be computed by the GPU(This happens if the GPU is the bottleneck i.e., the CPU is ahead of the GPU and the instruction pointer(on the CPU) reaches the predicate of the if statement before the kernel that calculates the tensor on which the predicate is dependent on finishes on the GPU). Upon profiling, I can see that the CPU spins during the entire synchronization duration. CUDA supports the CPU to block during the synchronization duration by using the <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__DEVICE.html#group__CUDART__DEVICE_1g69e73c7dda3fc05306ae7c811a690fac" rel="noopener nofollow ugc">cudaSetDeviceFlags API</a>. Also, some issues[<a href="https://forums.developer.nvidia.com/t/synchronization-spins-cpu-under-linux-tesla-driver-bug/36972/2" rel="noopener nofollow ugc">1</a>] on NVIDIA developer forum report that this behaviour is present on linux machines. Since I use linux, I can verify this to be true in my case. I have asked a related query on NVIDIA developer forums too[<a href="https://forums.developer.nvidia.com/t/cpu-spins-while-waiting-for-gpu-to-finish-computation/241672" rel="noopener nofollow ugc">link</a>].</p>
<p><strong>Question</strong></p>
<ul>
<li>Does pytorch API allow us to use cudaSetDeviceFlags API? If yes, please direct me on the steps as I cannot find the relevant documentation. If no, are there problems with the CPU blocking instead of spinning during synchronization?</li>
</ul>
<p>Please feel free to point out anything I missed. Thanks.</p>

ptrblck: <p>I don’t think it’s currently possible to change this flag in PyTorch and an older feature request was discussed <a href="https://github.com/pytorch/pytorch/issues/28224">here</a>. However, I don’t know the status of it.</p>

Puneeth_Naik: <p>Thank you for the related issue and the response. No worries, I will keep a track of the issue. I am marking this as the solution to close the topic and will track the issue for a future update.<br>
Thanks</p>
DataParallel vs single GPU training

tecker: <p>Same seed  (details below), same machine. DP: DataParallel</p>
<p>Single GPU: batch size 32, learning rate 0.01<br>
4 GPUs DP: batch size 32, learning rate 0.0025</p>
<p>Shall these two settings have the same training result?<br>
I think they should have the same result, but the experimental results on CIFAR10 dataset show similar but different training losses and accuracies.</p>
<p>By the way, how about the DistributedDataTraining (DDP) process? 4 GPUs DDP: batch size 32, learning rate 0.01? Will this lead to the same result?<br>
Thank you very much in advance!</p>
<pre><code class="lang-auto">random.seed(seed)
np.random.seed(seed)
os.environ['PHTHONHASHSEED'] = str(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
            torch.cuda.manual_seed(seed)
            torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministric =True
</code></pre>

Jamie_Donnelly: <p>Training a model in parallel like this is supposedly mathematically equivalent to the serial implementation - meaning you should obtain the same model at the end of training whether you used <code>DataParallel</code> or not.</p>
<p>So by training the <code>DataParallel</code> model with a learning rate of <code>0.0025</code> you would be getting the same final model as if you trained on a single GPU with that learning rate. Set the same hyperparameters for a parallel implementation as you would want for a serial implementation.</p>
<p>Manually configuring some of the backends like you have will probably help for consistency between approaches since often different low-level code might get executed unaware to the user.</p>
How do you check the shape of tensors?

setsailforfail: <p>Hi!</p>
<p>I am very curious about your approaches of checking shapes of tensors. I am not sure if this is even a normal thing to do, but I often run into errors due to missmatches of the shapes of tensors. Thus, I often (very beginner like, I know) use print statements to check the size of a tensor and make changes accordingly. Debugging takes way more time for me to get the actual shape, so I am curious what is considered best practice here? How do you check the shape of tensors? Or is this something only beginners do in the first place.</p>

marksaroufim: <p>Na it’s not just a beginner thing, I inspect tensor shapes all the time</p>
<p><code>import</code> <code>torch </code></p>
<p><code>v </code> <code>=</code> <code>torch.tensor([[</code> <code>1</code> <code>,</code> <code>0</code> <code>],[</code> <code>0</code> <code>,</code> <code>1</code> <code>]]) </code></p>
<p><code>x </code> <code>=</code> <code>list</code> <code>(v.shape) </code></p>
<p><code>print</code> <code>(x)</code></p>
<p>Granted there are some tools that might help you visualize all the shapes in one go like TensorSensor</p>
Memory leak (no_grad) when accumulating dataloader outputs

PoseAI: <p>Hi, we had a large RAM memory leak during gpu inference and narrowed down the culprit.  Not critical for us as we found a quick workaround but I figured I would post this here in case it could help someone else or lead to a better fix.</p>
<p>Adding a sample snippet from our code:</p>
<pre><code>    with torch.no_grad():
        for batch_num, data in enumerate(dataloader):
            out = self.forward(data)[0]
            store_out.append(out.detach().cpu().numpy())

            # these would leak
            # gt_batch = data['gt'].detach()
            # gt_batch = data['gt'].detach().cpu().numpy()

            # this doesn't leak
            gt_batch = data['gt'].detach().cpu().numpy().copy()

            store_gt.append(gt_batch)
</code></pre>
<ul>
<li>Model in eval mode and torch.no_grad</li>
<li>iterating through dataloader, we store prediction and some loader data for each batch in a list (to be torch.cat after loop)<br>
=&gt;memory quickly overflowed to crash levels.<br>
Accumulating predictions alone was fine (i.e. no data directly from dataloader).<br>
Accumulating data from the dataloader caused leak.<br>
We tried .detach() =&gt; leak<br>
We tried .detach().cpu().numpy() =&gt; leak<br>
we tried .detach().cpu().numpy().copy() =&gt; no leak, hurrah!</li>
</ul>
<p>So something in the dataloader output seems to keep graph alive.<br>
Hope that helps someone somewhere</p>
Improving code readability of torchvision.model.detection

umong: <p>I am a student who is learning about Faster R-CNN family and trying to understand the implementation provided in the <code>torchvision.model.detection</code> module. However, I find the code difficult to read and understand. Some blocks of code can be simplified more. Example:</p>
<pre><code class="lang-python"># https://github.com/pytorch/vision/blob/2b25d67925df9741ba2a75a07bc3046302969e87/torchvision/models/detection/_utils.py#L162
box_sum = 0
for val in boxes_per_image:
    box_sum += val
</code></pre>
<p>Therefore, I propose to improve the code’s readability to make it more accessible to beginners like myself. Here are some possible improvements that I could make:</p>
<ul>
<li>Add more comments and docstrings to explain the purpose and functionality of each function and module</li>
<li>Simplify and reorganize complex code blocks.</li>
</ul>
<p>I believe that these improvements will make the code more accessible to beginners and improve the overall quality of the module.</p>
<p>Please let me know if you have any feedback or suggestions. I would be happy to collaborate with other contributors to make these changes.</p>
<p>Thank you for your consideration.</p>

ptrblck: <p>Thanks for suggesting improvements to the code base!<br>
I guess you could create a feature request on GitHub discussing potential changes and improvements, but let me also add <a class="mention" href="/u/pmeier">@pmeier</a> as the code owner so that we can chime in.</p>

pmeier: <p>There is an issue now: <a href="https://github.com/pytorch/vision/issues/7458" class="inline-onebox" rel="noopener nofollow ugc">Improving code readability of torchvision.models.detection · Issue #7458 · pytorch/vision · GitHub</a>. Let’s keep the discussion there.</p>
Pytorch detectes only one GPU in Jetson AGX Orin

axnet: <p>I’m working to develop a face detection in Jetson AGX Orin using PyTorch but when running</p>
<blockquote>
<blockquote>
<p>import torch<br>
print(torch.cuda.device_count())<br>
only one GPU is detected.<br>
The FPS using the GPU approximately matches with the CPU detection.</p>
</blockquote>
</blockquote>

ptrblck: <p>Jetson Orins use a single GPU as seen in <a href="https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/">these technical specs</a> so I’m unsure what your expectations are.</p>

axnet: <p>I’m using Jetson AGX Orin not the nano orin</p>

ptrblck: <p>The link I’ve posted shows the technical specs for different Orins:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25421e9ce5c5abbc6723be363b2d1bba94c79aeb.png" data-download-href="https://discuss.pytorch.org/uploads/default/25421e9ce5c5abbc6723be363b2d1bba94c79aeb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25421e9ce5c5abbc6723be363b2d1bba94c79aeb_2_690x147.png" alt="image" data-base62-sha1="5jBkgOXnSGkXlHBOoKjauH9jGd5" width="690" height="147" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25421e9ce5c5abbc6723be363b2d1bba94c79aeb_2_690x147.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25421e9ce5c5abbc6723be363b2d1bba94c79aeb_2_1035x220.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25421e9ce5c5abbc6723be363b2d1bba94c79aeb_2_1380x294.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1565×335 53.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

axnet: <p>sorry I have very recently started to use an edge device, does that means that it has only one GPU? what about the number of cores?<br>
so if Pytorch detects one gpu then it’s utilizing the full potential of the GPU?</p>

ptrblck: <p>Yes, it has one GPU with multiple cores. Also yes, PyTorch will launch CUDA kernels to fully utilize the device.</p>

axnet: <p>Now it’s clear thank you</p>
Conv1d time-series expected normalization

davids: <p>I am using custom audio data for model training. The time-series data is between ~-1.5 and ~+1.5 with the mean being around 0. The model I am using consists of Conv1d layers with ReLU activation. After the ReLU, I am left with only positive side of the signal. Is this OK? Or should the input signal be normalized in a way that it consists of only positive values? I feel like I am losing half of the information now.</p>

J_Johnson: <p>The <a href="https://en.m.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="noopener nofollow ugc">Rectified Linear Activation function</a> (aka ReLU), like most activation functions, sets a cut off to reset model thru-data to zero that falls below a certain threshold. Think of it as a way for a model to make clear cut decisions, based on dynamic variables.</p>
<p>However, it probably is not ideal to place an activation prior to a learnable layer as you will lose certain information that may  be needed by the model.</p>

davids: <p>So does that mean that the input data should be shifted to include only positive values or not? I understand what ReLU does (thanks for the explanation, anyways!), but I am not sure if this means that the input data should be positive-only by default. Previously, I was working with images (and ReLU), where this question was not relevant as all pixel values are positive anyways.</p>

J_Johnson: <p>Whether you normalize the values between -1 to 1 or between 0 and 1 makes no difference, because of the nature of a convolution operation(plus a possible bias).</p>
<p>For a really simple example, suppose we have a sequence with the following values <code>[-1.0, -1.0, 1.0]</code> and the Conv1d kernel at channel 0 is <code>[-1.0, -1.0, -1.0]</code>. Multiply those elementwise and sum and you get <code>[1.0]</code>. Now suppose the kernel at channel 1 is <code>[1.0, 1.0, 1.0]</code>. The same operation gives <code>[-1.0]</code>. So before any activation, an appropriately trained kernel will decide whether that information is important to pass along or not, for that given channel. However, another channel might pass that information along as it may be trained to identify a different feature. But even if you had a sequence between 0 and 1, the kernel can have negative elements and change the output to be outside of that range.</p>
<p>So, as long as your activation is after a learnable layer(s), then it should be fine with either initial normalization, and as long as you stay consistent.</p>

davids: <p>I see, thanks for the insight! I am using activation as a part of a convblock sequence: Conv1d, BatchNorm1d, ReLU, which is very standard I believe, so the order is fine. Do you maybe know if there’s a list of what filters are used in Conv1d? Or are they just random? Could not find anything online.</p>

J_Johnson: <p>The filters are a learnable component. That’s why it doesn’t matter if numbers out of the activation layer are positive, because the next filter can make them negative again, if necessary.</p>
Fixed value output – mo improbing / moving but just sometimes

MikeTensor: <p>I have a model which resulted in very good results.<br>
I saved all the parameters (regressor, vgg16-ft_extractor, regressor, optimizer) after initialisation.</p>
<p>If I ran the model for training again, or change some Hyperparamters in <strong>most</strong> of the cases a similar result is the outcome (as expected).<br>
(I initialize the model with the parameters of the the successful run; as described above).</p>
<p>But sometimes it doesnt do nothing. The (bad) training/val results of the epochs dont move at all.<br>
The estimation for the (lets say distance) is always fixed (one value for all predictions in every epoch).<br>
– The strange thing is, that the same *.py script and initialized parameters mostly works – without any problems…</p>
<p>Any idea why the network is behaving like this?<br>
Is there something like a “cuda”-cache which I have to reset manually?</p>
<p>Example:</p>
<pre><code class="lang-auto">Target Distance   : tensor([[ 3.3200],
        [17.7000],
        [28.5000],
        [43.3200]], device='cuda:0')
Predicted Distance: tensor([[27.3686],
        [27.3686],
        [27.3686],
        [27.3686]], device='cuda:0')
</code></pre>

eqy: <p>It could be possible that there is some nondeterminism in your setup and that some initializations perform poorly compared to others. To rule this out, could you try setting a manual seed and consulting<br>
<a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 2.0 documentation</a> ?</p>
Releasing cuda memory after each run

simines: <p>I am doing hyperparameter tuning using Hyperopt and 2 gpus. Initially the gpu RAM used is 758 MB which is less than the threshold that I have defined, but after doing one more training the RAM used increase to 1796. To solve this issue I tried using torch.cuda.empty_cache() after each training, but it seems that it is not working. Also, I tried deleting some of the tensors after training but the number of such variables are lot.<br>
Does anyone know how I can solve this problem?</p>

eqy: <p>How are you measuring the memory usage before and after each run? If you are measuring memory usage before all kernels have been executed once, then lazy module loading would increase memory usage as kernels are loaded for the first time and clearing the cache wouldn’t free this memory if the process is kept alive. Similarly, memory is also used for workspaces (e.g., cuBLAS) or “plans” (e.g., cuFFT) that would be kept around as workspaces and plans are likely to be reused. It is generally not advisable to manually free this memory but they can be with functions such as</p>
<pre><code class="lang-auto">torch._C._cuda_clearCublasWorkspaces()
</code></pre>
<p>and</p>
<p><a href="https://pytorch.org/docs/stable/backends.html#torch.backends.cuda.clear" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/backends.html#torch.backends.cuda.clear</a></p>
<p>If you are observing that the memory usage is still increasing with each run after everything has fully “warmed up,” then that would be an unexpected bug.</p>

simines: <p>Thank you for your reply.<br>
For each run, I am using the following:<br>
torch.cuda.init()<br>
res = {‘gpu’: torch.cuda.utilization(device)}<br>
torch_cuda_mem = torch.cuda.mem_get_info(device)<br>
mem = {<br>
‘used’: torch_cuda_mem[-1] - torch_cuda_mem[0],<br>
‘total’: torch_cuda_mem[-1]<br>
}</p>

eqy: <p>Right, so I would check that memory usage stabilizes after warmup (running all the computation/a single iteration once) if the initial measurement is being done before everything has been run once.</p>

simines: <p>So you mean initially when I run the first training the memory used is 758, and then I empty cache and do another training and the memory increased slightly and then stabilizes for the remaining training jobs?</p>

eqy: <p>Yes, unless you mean the first measurement was done <em>following</em> the first training run.</p>

simines: <p>Thank you. No, it was done before each training run.</p>
Data augmentation methods used in pre-trained models provided by torchvision for image classification?

a2108870: <p>Hi, I’m interested in using the pre-trained models provided by torchvision for image classification. I was wondering if you could tell me what data augmentation methods are used in these models? Specifically, do they include any color-based random augmentations? Thank you!</p>

ptrblck: <p>All available weights from pretrained <code>torchvision.models</code> are available <a href="https://pytorch.org/vision/stable/models.html#table-of-all-available-classification-weights">here</a> including their training <a href="https://github.com/pytorch/vision/tree/main/references/classification#image-classification-reference-training-scripts">recipe</a>. You could check the commands used to train each model and check how the <a href="https://github.com/pytorch/vision/blob/2b25d67925df9741ba2a75a07bc3046302969e87/references/classification/train.py"><code>train.py</code></a> script is called using these arguments.</p>

a2108870: <p>Thank you for providing an answer to my question on the post.  Your response has been very helpful to me.</p>
Saving the parameters during the update in each epoch

Mooky1100: <p>Hello, I am trying to save the update of parameters during training for each epoch, and I have been using the following code below.</p>
<pre><code class="lang-auto">
startTime = time.time()
for t in range(n_steps):
    # Compute prediction
    loss = - model.compute_elbo_loop(x_train_1, y_train_1,x_train_2, y_train_2)
    # Compute loss
    # Zero gradinets
    optimizer.zero_grad()
    # Compute gradients
    loss.backward()
    #a = list(model.parameters())[0].clone() 
    #print(list(model.parameters())[0].grad,a)
    
    optimizer.step()
    #b = list(model.parameters())[0].clone() 
    #print(b)
    if t % 10 == 9:
        loss_array[int((t + 1) / 10 - 1)] = loss.item()
        time_array[int((t + 1) / 10 - 1)] = time.time() - startTime 
        
        for name, param in model.named_parameters():
            print(name,param)
            param_dictionary[name] = param_dictionary[name] + [param.cpu().detach().numpy()]
        
        # store updated parameters in the dictionary
        params_dict[f't{t}_update{0}'] = model.state_dict().copy()
    
    if t % 10 == 9:
        print(f"Loss: {loss.item()}, Step [{t}/{n_steps}]")
        print(model.ModelString())

endTime = time.time()
</code></pre>
<p>I have checked that in each epoch, the parameters are being updated. But when I check the ‘param_dictionary’, all the values are changed to the updated ones. For example, if the current updated value for certain parameter is 3.8, then all the values saved in the dictionary are 3.8. I have used the same dictionary code for other projects, and it worked fine. Is there any specific reason why this is happening in here? am I missing something?</p>

srishti-git1110: <p>Hi,<br>
This is because <code>.detach()</code> and <code>.numpy()</code> return tensors that share the same storage with the original one.<br>
In order to make a copy, you can use <code>.clone()</code>, so replace your code with the following and it should work.<br>
<code>param.cpu().detach().clone().numpy()</code></p>

Mooky1100: <p>Thank you, Srishti Gureja !</p>
<p>I just checked the code with ‘clone()’, and it saves all the results!! Thank you so much! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
I need help with loading dataset from other place than cache

dubljanin96: <p>Hi all,<br>
I’m trying to run Alexnet but I don’t have enough space on C so I downloaded on D the ImageNet dataset.<br>
How to point to it?</p>
<h1>
<a name="device-configuration-1" class="anchor" href="#device-configuration-1"></a>Device configuration</h1>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)</p>
<p>model = torch.hub.load(‘pytorch/vision:v0.10.0’, ‘alexnet’, pretrained=True)<br>
model.eval().to(device)</p>
<p>from datasets import load_dataset<br>
ds = load_dataset(“imagenet-1k”)<br>
test_ds = ds[“test”]</p>
<p>dataloader = DataLoader(<br>
test_ds,<br>
batch_size=64, # may need to reduce this depending on your GPU<br>
num_workers=0, # may need to reduce this depending on your num of CPUs and RAM<br>
shuffle=False,<br>
drop_last=False,<br>
pin_memory=True<br>
)</p>
<p>with torch.no_grad():<br>
correct = 0<br>
total = 0<br>
for images, labels in dataloader:<br>
images = images.to(device)<br>
labels = labels.to(device)<br>
outputs = model(images)<br>
_, predicted = torch.max(outputs.data, 1)<br>
total += labels.size(0)<br>
correct += (predicted == labels).sum().item()<br>
del images, labels, outputs</p>
<pre><code>    print('Accuracy of the network on the {} test images: {} %'.format(total, 100 * correct / total))
</code></pre>

ptrblck: <p>I don’t know where this line of code is coming from:</p>
<pre><code class="lang-python">from datasets import load_dataset
</code></pre>
<p>but if you use <code>torchvision.datasets.ImageNet</code> you can use the <code>root</code> argument to point to the root folder containing the class subfolders with the images.</p>
How to convert list of loss tensor to numpy array

uqhah: <p>Hi my loss is a list of tensors as follows:<br>
<code>[tensor(0.0153, device='cuda:0', grad_fn=&lt;DivBackward0&gt;), tensor(0.0020, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)]</code><br>
how do I convert this to a NumPy array?<br>
Thank you so much!</p>

J_Johnson: <pre><code class="lang-auto">np_loss_list = np.array([x.item() for x in pt_loss_list])
</code></pre>
Data Loader for multiple classes with train / test subfolders

bornexmachina: <p>Struggling with PyTorch My folders are organized as such:</p>
<pre><code class="lang-auto">imgs
|
|_classA
  |_train
     |_img1.png
     |_img2.png
  |_test
     |_img20.png
     |_img21.png
|
|
|_classB
  |_train
  |_test
...
</code></pre>
<p>And I would like for each class (!) load data from train, train some network, evaluate on test.</p>
<p>RTFM got me so far:</p>
<ol>
<li>ImageLoader for loading of my .png</li>
<li>Custom Dataset, e.g. to keep track which class my image is from</li>
<li>DataLoader to get batches of my data and hand it over to my model</li>
</ol>
<p>Now most of the tutorials have the structure</p>
<pre><code class="lang-auto">train
|
|
test
</code></pre>
<p>and the classes are within the <code>train / test</code> folders.</p>
<p>Now I am not sure, how to get started, due to <code>train / test</code> being sub-folders in my structure.</p>

ptrblck: <p>As you have already explained using the native <code>ImageFolder</code> dataset won’t work since your folder structure is different. You could of course rearrage it (via symlinks if you don’t want to move the actual data), but in case that’s not possible you could write a custom <code>Dataset</code> and check the <a href="https://github.com/pytorch/vision/blob/2b25d67925df9741ba2a75a07bc3046302969e87/torchvision/datasets/folder.py#L48-L104"><code>make_dataset</code></a> method which is traversing the folders to collect the sample paths and the corresponding targets.</p>
Real time music generation

ebgd: <p>Hi<br>
Is it possible for an AI (cGAN ?)</p>
<ol>
<li>to receive a user’s voice as input, with some conditioning variables</li>
<li>and generate a musical atmosphere</li>
<li>an dmost importantly : in real time<br>
If yes, what would be the best tool to do that ?<br>
Thanks</li>
</ol>
Subclassing torch.autograd.Function to write a custom loss function and maintain gradients

HeleneAthene: <p>Hello,<br>
I am trying to use gradient descent to learn the affine matrix values necessary to minimize a cost function.</p>
<p>In short, I have n pairs of images, and a tensor of size [n, 3, 3], one identity matrix for each image. For each pair of images, I use the corresponding tensors to create an affine transformation of these images. I then calculate a full-normalized covariance matrix of the two images. Importantly, I cannot use pytorch for this step. The cost is the difference between the location of the largest values in the ncov matrix and the center of the ncov array, which would be a perfect overlay. This difference (loss) is returned as a single scalar value.</p>
<p>Because I can’t use pytorch to calculate the ncov, I break the computational graph and cannot calculate the gradients. I gather that I can make a subclass of <code>torch.autograd.Function</code> (<a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">described here</a>), however, it is very unclear to me specifically how to write the <code>backward()</code> method to ensure that the new <code>Function</code> works properly with the autograd engine.</p>
<p>It seems like the values that should be returned from the <code>backward()</code> method depend heavily on which non-torch computations are used. Am I overthinking this? If not, is there a good reference simplifying how forward/backward should be structured for unconventional loss functions like this?</p>

soulitzer: <p>There’s no easy way to do this unfortunately. You’ll need to manually find the derivative of the non-torch operations you used.</p>
<p>You’re allowed to write the backward in non-torch operations, and you can use torch.autograd.gradcheck to help you figure out if your gradients are correct or not.</p>

HeleneAthene: <aside class="quote no-group" data-username="soulitzer" data-post="2" data-topic="175589">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>You’ll need to manually find the derivative of the non-torch operations you used.</p>
</blockquote>
</aside>
<p>Ok good to know, thank you! Are there any resources for manually finding the derivatives of the non-torch operations I use?</p>

soulitzer: <p>I don’t have any specific recommended ones in mind, but from googling “backpropagation derivation” I found <a href="https://medium.com/binaryandmore/beginners-guide-to-deriving-and-implementing-backpropagation-e3c1a5a1e536" class="inline-onebox" rel="noopener nofollow ugc">A beginner’s guide to deriving and implementing backpropagation | by Pranav Budhwant | binaryandmore | Medium</a>. It would probably be good to look at a couple tutorials and see what explanation works best for you.</p>
<p>If you have any specific questions, happy to answer them here as well.</p>

HeleneAthene: <p>ok I will see what I can do, thank you!</p>
Torch.onnx.export tensor shape gets changed

SergeyLev: <p>While exporting model input tensor gets changed from Tensor(1, 1, 32, 151) to Tensor: (tensor(1), tensor(1), tensor(32), tensor(151)). How can I stop this behavior?</p>
<p>Example code is simple:</p>
<pre><code class="lang-auto">dummy_input = torch.randn([1, 1, 32, 151], device="cuda")
onnx_reader_path = 'onnx_model.onnx'
torch.onnx.export(Model.model, dummy_input, onnx_reader_path, ...)
</code></pre>

tkupek: <p>I have the exact same issue with a Yolov7 model export.<br>
It’s happening somewhere in the <code>graph, out = torch._C._create_graph_by_tracing(</code> function. The input is still as expected before the call, but in the first call of <code>wrapper</code>, the <code>in_vars</code> are already unflattened.</p>
<p>I assume this could be a Pytorch 2.0 thing, what version are you using?<br>
Any help is appreciated.</p>
TypeError: __init__() got an unexpected keyword argument 'rotation'

rrrrst: <p>from keras.preprocessing.image import ImageDataGenerator</p>
<p>train_datagen=ImageDataGenerator(rescale=1./255,<br>
rotation=40,<br>
wigth_shift_range=0.2,<br>
height_shift_range=0.2,<br>
shear_range=0.2,<br>
zoom_range=0.2,<br>
horizontal_flip=True,<br>
fill_mode=‘nearest’<br>
)<br>
test_datagen=ImageDataGenerator(rescale=1./255)<br>
train_generator=train_datagen.flow_from_directory(train_dir,<br>
target_size=(150,150),<br>
class_mode=‘categorical’,<br>
batch_size=batch_size)<br>
test_generator=test_datagen.flow_from_directory(train_dir,<br>
target_size=(150,150),<br>
class_mode=‘categorical’,<br>
batch_size=batch_size,<br>
shuffle=False)</p>

ptrblck: <p>Based on the posted code you are using <code>Keras</code> so I would recommend asking in the TensorFlow discussion board as you will find the experts there.<br>
With that being said the error points to an invalid argument called <code>rotation</code> so check the docs of this class and make sure valid arguments are passed to it.</p>
pyTorch2.0 Docker Container

MikeTensor: <p>Hello everyone.<br>
I am looking for …</p>
<ol>
<li>a pyTorch2.0 DockerContainer (which is accessible via SSH) and which has python and all regular dependecies included.</li>
<li>the same as 1) but –if possible– with jupyLab.</li>
</ol>
<p>Any idea where I can get such containers from?<br>
Seems like there is not sooo much around since pyTorch2.0 is really new.</p>
<p>Found this one – but the size seems to be a bit to small for a whole package (OS, python-env, pytorch, …).<br>
<a href="https://hub.docker.com/layers/pytorch/pytorch/2.0.0-cuda11.7-cudnn8-runtime/images/sha256-fabb5a665a05b8ee0ac76f0d943acc40039e13536e11a44d3dc47625a266e759?context=explore" class="onebox" target="_blank" rel="noopener nofollow ugc">https://hub.docker.com/layers/pytorch/pytorch/2.0.0-cuda11.7-cudnn8-runtime/images/sha256-fabb5a665a05b8ee0ac76f0d943acc40039e13536e11a44d3dc47625a266e759?context=explore</a></p>

marksaroufim: <p>Images will get published here <a href="https://hub.docker.com/r/pytorch/pytorch/tags" class="inline-onebox" rel="noopener nofollow ugc">Docker</a> and nightlies here <a href="https://github.com/orgs/pytorch/packages/container/pytorch-nightly/versions?filters%5Bversion_type%5D=tagged" rel="noopener nofollow ugc">https://github.com/orgs/pytorch/packages/container/pytorch-nightly/versions?filters[version_type]=tagged</a></p>
<p>To create one with Jupyter lab you can just FROM our package and create your own dockerfile</p>
DDP Error on multi node CPU training

atang96: <p>Hi,</p>
<p>I followed this tutorial <a href="https://leimao.github.io/blog/PyTorch-Distributed-Training/" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Distributed Training - Lei Mao's Log Book</a><br>
and modified some of the code to accommodate CPU training since the nodes don’t have GPU. My code is using gloo and I changed the device to CPU. However, I keep running into the following error at the end of the training:</p>
<p>ERROR:torch.distributed.elastic.agent.server.api:Error waiting on exit barrier. Elapsed: 0.060767173767089844 seconds<br>
Traceback (most recent call last):<br>
File “/usr/local/lib/python3.6/dist-packages/torch/distributed/elastic/agent/server/api.py”, line 904, in _exit_barrier<br>
barrier_timeout=self._exit_barrier_timeout,<br>
File “/usr/local/lib/python3.6/dist-packages/torch/distributed/elastic/utils/store.py”, line 67, in barrier<br>
synchronize(store, data, rank, world_size, key_prefix, barrier_timeout)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/distributed/elastic/utils/store.py”, line 53, in synchronize<br>
agent_data = get_all(store, key_prefix, world_size)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/distributed/elastic/utils/store.py”, line 31, in get_all<br>
data = store.get(f"{prefix}{idx}")<br>
RuntimeError: Connection reset by peer</p>
<p>I suspect it is caused by the None device id and prefix of CPUs Is there a way I can do DDP on multiple nodes using only CPUs?</p>
<p>Thanks!</p>

wanchaol: <p>cc <a class="mention" href="/u/h-huang">@H-Huang</a> <a class="mention" href="/u/rvarm1">@rvarm1</a> looks like multiple people hit similar issues about connection reset by peer</p>

f_g: <p>The same happens to me on multi node GPU training as well, it doesn’t seem to happen all the time, but when it happens it would make the node fail and not able to restart on its own.</p>
Traceable model with gaussian fit in preprocessing

Avillalpando: <p>Hi,</p>
<p>We are trying to export a model for inference on an iPad. To do this we are using the torch.jit.trace() function to create a Torch Script module that can be later compiled in C++.<br>
The model itself is rather simple, just a sequence of fully connected layers, and is not giving any issue by itself.</p>
<p>The problem is that we need to perform a preprocessing step to transform a raw signal into a sum of gaussian functions, from which we extract the values that will be the inputs for the layers of the model.<br>
This gaussian fit is easy to do with access to SciPy library, for instance, but in order to have full compatibility with Torch Script, we require that only Pytorch is used.<br>
As an alternative, we used the idea posted here:<br>
<a href="https://discuss.pytorch.org/t/using-pytorch-optimizers-for-nonlinear-least-squares-curve-fitting/123635" class="inline-onebox">Using PyTorch optimizers for nonlinear least squares curve fitting</a>,<br>
where an optimizer is used to find the best fitting curve. This gave us satisfactory results, but when we trace the model, the use of a closure function and the call to backward() won’t return a traced model. We have tried to use a different optimizer that does not require a closure function, such as Adam, but still we need to perform the backward() pass.</p>
<p><code> 'iterations=5000</code><br>
<code>'for _ in range(iterations):</code><br>
<code>    </code> <code>   '' self.my</code> <code>_fit.optimizer.zero_grad()</code><br>
<code>  '' output =</code> <code>self.my</code> <code>_fit(</code> <code>self.my</code> <code>_fit.x, </code> <code>self.my</code> <code>_fit.init_params)# returns gauss function with this x axis and set of params</code><br>
<code> '' loss = F.mse_loss(output,</code> <code>self.my</code> <code>_fit.y)</code><br>
<code> ''  loss.backward()</code><br>
<code> </code> <code>'' self.my</code> <code>_fit.optimizer.step()</code></p>
<p>The error we get on the loss.backward() line is the following:</p>
<p><code>Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient</code></p>
<p>Do you have any suggestion regarding how to perform this gaussian fit in a model that can be traced and exported to production?</p>
<p>Thank you very much,<br>
kind regards.</p>
How to convert a list of tensors to a Pytorch tensor?

shrbrh: <p>Hi, I have a list of tensors of size 4 that I want to convert into a Pytorch tensor. I used<br>
<code> y = torch.tensor(x)</code> where x is the list.<br>
But I am getting the following error:<br>
<code>ValueError: only one element tensors can be converted to Python scalars</code><br>
How do I resolve this?</p>

Joschka: <p>Depending on what exactly you want, you’ll most likely want to use either <a href="https://pytorch.org/docs/stable/generated/torch.stack.html?highlight=stack#torch.stack" rel="noopener nofollow ugc">stack</a> (concatenation along a new dimension) or <a href="https://pytorch.org/docs/stable/generated/torch.cat.html?highlight=cat" rel="noopener nofollow ugc">cat</a> (concatenation along an existing dimension).</p>
<p>Here’s a quick example:</p>
<pre><code class="lang-python">import torch

x = torch.tensor([1,2,3,4])
y = torch.tensor([5,6,7,8])
list_of_tensors = [x,y]

# will produce a tensor of shape (2,4)
stacked_0 = torch.stack(list_of_tensors, dim=0)

# will produce a tensor of shape (4,2)
stacked_1 = torch.stack(list_of_tensors, dim=1)

# will produce a tensor of shape (8,)
concatenated = torch.cat(list_of_tensors, dim=0)

</code></pre>

shrbrh: <p>I tried applying <code>torch.stack()</code> but I am getting the following error now:<br>
<code>RuntimeError: stack expects each tensor to be equal size, but got [1, 64, 128, 128] at entry 0 and [1, 128, 64, 64] at entry 1 </code><br>
Can you please help me resolve this?</p>

Joschka: <p>Stacking tensors along a new dimension requires them to be equally sized in all other existing dimensions. The best / “correct” way to make sure this prerequisite is satisifed heavily depends on your specific use-case, i.e., how you obtain these tensors in the first place, what they represent and so on</p>

shrbrh: <p>The tensors in the list are actually style features of an image extracted from 4 different layers of VGG-19 network, hence the differences in size. I actually need to concatenate these style features with a tensor of content features for which I need to convert the list into a tensor first, but I am unable to do so.</p>

Joschka: <p>If this is all supposed to be happening as part of a new model forward process you should consider adding additional layers which take care of resizing the inputs such that they match. For upsampling the spatial dimensions you could use <a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html" rel="noopener nofollow ugc">ConvTranspose2d</a>, or the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html?highlight=upsample#torch.nn.Upsample" rel="noopener nofollow ugc">Upsample</a> layer which performs interpolation rather than a learned transformation. In addition to that you would need to make sure that the number of channels (size of the first dimentions) also match, e.g. by applying respective linear transformations or setting the <code>in_channels</code> and <code>out_channels</code> value of <code>ConvTranspose2d</code> to whatever you need.</p>
How can compiled model generates a torch.fx.GraphModule

cokespace2: <p>Compile a model using <code>optimized = optimize("inductor")(model)</code>, the <code>optimized</code> is of type <code>&lt;class 'torch._dynamo.eval_frame.OptimizedModule'&gt;</code>, how to convert it to <code>torch.fx.GraphModule</code>?</p>

marksaroufim: <p>You shouldn’t need to convert an OptimizedModule to a GraphModule directly if your primary goal is getting speedups, all you need to do is run an inference</p>
<p>However if you’re trying to modify the graph module in some way you can set a breakpoint here <a href="https://github.com/pytorch/pytorch/blob/master/torch/_inductor/compile_fx.py#L142" class="inline-onebox" rel="noopener nofollow ugc">pytorch/compile_fx.py at master · pytorch/pytorch · GitHub</a> and change the graphmodule that’s passed in</p>

cokespace2: <p>Thanks Mark!<br>
But my goal is not just speedup inference but also trying to replace module like what I did in TorchFX, and according to official doc,</p>
<blockquote>
<p><strong>TorchDynamo</strong> generates FX Graphs from Python bytecode.</p>
</blockquote>
<p>so I think maybe I can get a FX Graph from TorchDynamo, is there any API that allows me to do so?</p>

marksaroufim: <p>This notebook might help <a href="https://colab.research.google.com/drive/1Zh-Uo3TcTH8yYJF-LLo5rjlHVMtqvMdf#scrollTo=zaMCj87J_JLU" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

cokespace2: <p>Thanks! That’s very helpful!</p>
How to build torch as a pip wheel?

nlgranger: <p>I recently submitted a pull request, managed to build and test it locally by copy-pasting some commands from a blog online, but I’d like to test my modifications more thoroughly on another machine. Unfortunately I am unable to build a wheel that is portable and complete (libc++ ABI version mismatch, cuda libraries are not pulled or found, etc.).</p>
<p>There are many different instructions online, none seem up to date or complete. The pytorch/builder repo has dozens of different scripts to build pytorch.</p>
<p>I’d like instructions on:</p>
<ul>
<li>how to generate a clean build environment (in a container without the need for GPU preferably)</li>
<li>which environment variables and commands I should run to build a portable wheel package similar to the official release (cuda 11.7).</li>
</ul>
The shape of mean and variance in torch.nn.functional.instance_norm does not make sense

hello-fri-end: <p>Consider the following code:</p>
<pre><code class="lang-auto">x = torch.rand((2, 3, 4, 5)) #N,C,H,W
x1_running_mean = torch.zeros((3))
x1_running_var = torch.ones((3))
x2_running_mean = torch.zeros((2, 3))
x2_running_var = torch.ones((2, 3))
res1 = torch.nn.functional.instance_norm(x, running_mean = x1_running_mean, running_var = x1_running_var, use_input_stats = False)
res2 = torch.nn.functional.instance_norm(x, running_mean = x2_running_mean, running_var = x2_running_var, use_input_stats = False)
</code></pre>
<p>res1 runs without any errors while res2 fails with the error saying that the function expected a tensor of size (3).</p>
<p>For instance norm, the mean, and variance are calculated for each N,C i.e across all the spatial dimensions and therefore the size of the mean and variance should be (N,C) and not (C) . Does pytorch handle instance norm differently?</p>

ptrblck: <aside class="quote no-group" data-username="hello-fri-end" data-post="1" data-topic="175634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hello-fri-end/48/58647_2.png" class="avatar"> hello-fri-end:</div>
<blockquote>
<p>and therefore the size of the mean and variance should be (N,C) and not (C)</p>
</blockquote>
</aside>
<p>Wouldn’t this add a dependency on the batch size which would then crash is the number of samples changes during training or eval?<br>
Using the module I see these shapes which match your working example:</p>
<pre data-code-wrap="python"><code class="lang-python">m = nn.InstanceNorm2d(100, affine=True, track_running_stats=True)
input = torch.randn(20, 100, 35, 45)
output = m(input)

m.weight.shape
# torch.Size([100])
m.bias.shape
# torch.Size([100])
m.running_mean.shape
# torch.Size([100])
m.running_var.shape
# torch.Size([100])
</code></pre>

hello-fri-end: <p>You are right about the dependency on the batch size, I hadn’t realized this. I found the following numpy implementation in pytorch tests which made me understand how the mean and variance is calculated across N &amp; C channels:</p>
<pre><code class="lang-auto">

def ref_nchw(x, scale, bias):

    x = x.reshape(batch_size * input_channels, size_a * size_b)
    y = (x - x.mean(1)[:, np.newaxis])
    y /= np.sqrt(x.var(1) + epsilon)[:, np.newaxis]
    y = y.reshape(batch_size, input_channels, size_a, size_b)
    y = y * scale.reshape(1, input_channels, 1, 1)
    y = y + bias.reshape(1, input_channels, 1, 1)



</code></pre>
Calibration curve of a multiclass logistic regression

klark: <p>Hello,</p>
<p>I’m trying to plot a calibration curve for my logistic regression model. I have 8 classes. Is there any written algorithm in pytorch or in other python package to do that. The scikit-learn machine learning library seems to work for only binary models.</p>

tataganesh: <p>Assuming that the calibration curve is the same as a Reliability Diagram, you can use the implementation <a href="https://github.com/torrvision/focal_calibration/blob/475422a6754e20d25216158a874749c33e826e21/Metrics/plots.py#L59" rel="noopener nofollow ugc">here</a>. If this doesn’t work, searching for other implementations of Reliability Diagram should be helpful.</p>

klark: <p>Thank you. It worked.</p>
Torchrun using GPU memory on rank 0 for all the processes

smujjiga: <p>For all the processes spanned by torchrun (DDP) a small amount of GPU on rank 0 is used for all the processes.</p>
<p>Test code:</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
import torch.nn as nn
from torch.nn.parallel import DistributedDataParallel as DDP

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net = nn.Linear(10, 5)

dist.init_process_group("nccl")
rank = dist.get_rank()
print(f"Start running basic DDP example on rank {rank}.")
# One way to fix it
# import os
# os.environ["CUDA_VISIBLE_DEVICES"] = str(rank)
# torch.cuda.set_device(rank)

device_id = rank % torch.cuda.device_count()
model = ToyModel().to(device_id)
ddp_model = DDP(model, device_ids=[device_id])
    
if dist.get_rank() == 1:
    import pdb; pdb.set_trace()
    
dist.barrier()
</code></pre>
<p>Command used: <code>OMP_NUM_THREADS=4 torchrun --standalone --nproc_per_node=3 t.py</code><br>
Nvidia-smi:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/652cd4b28141f5fc6ee09ee6726f2464a337c5db.png" alt="image" data-base62-sha1="er2gZToWB01qEq9Apnk79Mmn8lZ" width="681" height="94"></p>
<p>There are similar issue posted:</p>
<ol>
<li><a href="https://github.com/pytorch/pytorch/issues/23138" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel: resume training from a checkpoint results in additional processes on GPU 0 · Issue #23138 · pytorch/pytorch · GitHub</a></li>
<li><a href="https://discuss.pytorch.org/t/ddp-taking-up-too-much-memory-on-rank-0/62443" class="inline-onebox">DDP taking up too much memory on rank 0</a></li>
</ol>
<p>But all of them are because of some where in the code the tensors get allocated to <code>cuda</code> which default to rank 0. But it is not the case with the above bare minimal snippet.</p>