akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use LR scheduler in DDP and amp

Dylan_Wang: <p>Hi,</p>
<p>I am wondering is there any tutorials or examples about the correct usage of learning rate scheduler when training with DDP/FSDP? For example, if the LR scheduler is <code>OneCycleLR</code>, how should I define <code>total number of steps in the cycle</code>, i.e., <code>total_steps</code> or (<code>steps_per_epoch</code> and <code>epochs</code>) arguments of the scheduler?<br>
The reason I am asking is that this scheduler updates LR based on the steps but in DDP the steps for current process is different from effective steps (i.e., in non-DDP, the step is 256, but in DDP with 2 GPU, the step is 128 for each process).<br>
Also, where should I call <code>scheduler.step()</code> if I am also using <code>amp</code>? Assume my code is:</p>
<pre><code class="lang-auto">steps_per_epoch = ???
n_epoch = 10
scheduler = (optimizer, max_lr=0.01, steps_per_epoch=steps_per_epoch, epochs=n_epoch)
for it, (img, labels) in enumerate(train_loader):
        # measure data loading time
        data_time.update(time.time() - end)
        bs = img.size(0)

        img = img.cuda(non_blocking=True)
        labels = labels.cuda(non_blocking=True)

        # compute output and loss
        with autocast(enabled=args.amp, dtype=args.amp_dtype):
            outputs = model(img)
            loss = criteria(outputs, labels)

        optimizer.zero_grad()
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()
        # scheduler.step() is this position correct?
</code></pre>
<p>Thank you!</p>

suraj.pt: <p>Yes, with DDP since the effective batch size scales by N the <code>steps_per_epoch</code> would effectively be <code>steps_per_epoch/N</code> and that is likely what you want to use.</p>
<p>I’m assuming you’re scaling LR proportional to N as well? A thumb rule I commonly see is <code>lr*sqrt(N)</code> but afaik the best scaling is an open question</p>

Dylan_Wang: <p>Thank you for the reply!</p>
<p>Are you suggesting that the <code>steps_per_epoch</code> should be set to <code>128</code> in the example I gave for 2 GPU? In that case, since each process have gradients in terms of all batch (i.e., 256), would not setting <code>steps_per_epoch = 128</code> be wrong when the optimiser tries to update the LR?</p>

suraj.pt: <p>Yeah, in your example if a single GPU setup takes 256 steps to run through the entire dataset, then <code>steps_per_epoch</code> should be 128 for 2 GPUs.</p>
<p>At <code>steps_per_epoch=128</code> your distributed model would have seen the entire dataset. This is the same as setting <code>steps_per_epoch=256</code> in a non-distributed setting</p>
Pass parameter in forward pass

bornexmachina: <p>My helper class looks like this:</p>
<pre><code class="lang-auto">class GradientReversalFunction(Function):
    """
    Gradient Reversal Layer from:
    Unsupervised Domain Adaptation by Backpropagation (Ganin &amp; Lempitsky, 2015)

    Forward pass is the identity function. In the backward pass,
    the upstream gradients are multiplied by -lambda (i.e. gradient is reversed)
    """

    @staticmethod
    def forward(ctx, x, lambda_):
        ctx.lambda_ = lambda_
        return x.clone()

    @staticmethod
    def backward(ctx, grads):
        lambda_ = ctx.lambda_
        lambda_ = grads.new_tensor(lambda_)
        dx = -lambda_ * grads
        return dx, None


class GradientReversal(torch.nn.Module):
    def __init__(self, lambda_=1):
        super(GradientReversal, self).__init__()
        self.lambda_ = lambda_

    def forward(self, x):
        return GradientReversalFunction.apply(x)
</code></pre>
<p>and in my model I use</p>
<pre><code class="lang-auto">        self.discriminator = nn.Sequential(
            GradientReversal(),
            nn.Linear(40, 20),
            nn.ReLU(),
            nn.Linear(20, 10),
            nn.ReLU(),
            nn.Linear(10, 1),
        )
</code></pre>
<p>and in the forward pass I call</p>
<p><code>self.discriminator(x).squeeze()</code></p>
<p>Now instead of a fixed <code>lambda_=1</code> I would like to call my forward function with varying values.</p>
<p>I have tried to update the forward call of the <code>GradientReversal</code> etc. but I only get errors.</p>

bornexmachina: <p>Our Lord and Saviour ChatGPT actually pointed it out to me.</p>
<p>It does not suffice to update the forward method of the GradientReversal() to</p>
<pre><code>def forward(self, x, lambda_):
    return GradientReversalFunction.apply(x, lambda_)
</code></pre>
<p>but I also have to explicitly grab the first entry of my discriminator</p>
<pre><code>x_grad_rev = self.discriminator[0](x, alpha)
domain_predictions = self.discriminator[1:](x_grad_rev).squeeze()
</code></pre>

InnovArul: <aside class="quote no-group" data-username="bornexmachina" data-post="1" data-topic="180223">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bornexmachina/48/58727_2.png" class="avatar"> bornexmachina:</div>
<blockquote>
<p>but I only get errors.</p>
</blockquote>
</aside>
<p>What is the specific error message that you get when trying to add <code>lambda_</code>?</p>
What's wrong with our loss and PyTorch?

Filippo_Portera: <p>Given the samples $\vec{x_i} \in \mathbb{R}^d, i \in [1,…,l]$ where $l$ is the number of training samples, $d$ is the number of input features, the related target values $y_i \in \mathbb{R}$, and the $l \times l$ matrix defined below:</p>
<p>$$ S_{i,j} =  e^{-\gamma_S ||\vec{x}_i - \vec{x}_j||^2} = e^{-\gamma_S (\vec{x}_i’ \vec{x}_i -2 \vec{x}_i’ \vec{x}_j + \vec{x}_j’\vec{x}_j)}<br>
$$</p>
<p>where $i \in [1,…,l], j \in [1,…,l]$, and $\gamma_S$ is another hyper-parameter, we would like to use with PyTorch the following custom loss for a regression task:</p>
<p>$$\sum_{i=1}^l \sum_{j=1}^l \sqrt{|p_i-y_i|}\sqrt{|p_j-y_j|} S_{i,j}$$<br>
where $p_i$ is the $i$-th estimation.</p>
<p>Our loss is implemented with this code:</p>
<pre><code>def ourLoss(out, lab):
    global stra, sc
    abserr = torch.abs(out - lab).flatten().float()
    serr = torch.sqrt(abserr)
    bm = stra[sc : sc + out.shape[0], sc : sc + out.shape[0]].float()
    loss = torch.dot(serr, torch.matmul(bm, serr))
    return loss
</code></pre>
<p>where ‘stra’ is $S$, sc is a counter used for batch evaluations, and then the Adam optimizer returns a nan loss value…</p>
Why is the loss not decreasing with Pytorch Geometric GATv2Conv (and GATconv)?

Ken_Semanov: <p>Loss begins at 219.7  or so and never budges.</p>
<p>There is nothing wrong with my training data or my training loop. I can literally substitute my AttnAggregator module below for a <code>SAGEconv</code> version, and training proceeds just fine.  If I substitute <code>GATconv()</code> layers  for the <code>GATv2conv()</code> layers, this frozen loss also occurs.  When I print out the  final value of <code>out</code> from the bottom for <code>forward()</code>  and it gives perfectly coherent numbers.</p>
<p>What am I missing?</p>
<pre><code class="lang-auto">
    out tensor([0.1702, 0.1489, 0.1702, 0.1702, 0.1702, 0.1702, 0.1667, 0.1667, 0.1667,
        0.1667, 0.1667, 0.1667], grad_fn=&lt;CatBackward0&gt;)




	epoch 126 , loss 219.74465942382812
	epoch 127 , loss 219.7310791015625
	epoch 128 , loss 219.74839782714844
	epoch 129 , loss 219.73077392578125
	epoch 130 , loss 219.73785400390625
	epoch 131 , loss 219.73355102539062
	epoch 132 , loss 219.74407958984375
	epoch 133 , loss 219.75140380859375
	epoch 134 , loss 219.79591369628906
	epoch 135 , loss 219.70147705078125
	epoch 136 , loss 219.73855590820312
	epoch 137 , loss 219.72088623046875
	epoch 138 , loss 219.7546844482422
	epoch 139 , loss 219.77212524414062
	training time elapsed mins 0.7
	over 139 epochs


</code></pre>
<p>= =</p>
<pre><code class="lang-auto">import torch.nn as nn
	import torch_geometric as pyg
	from torch import flatten
	from torch import concat
	class AttnAggregator( nn.Module ):
		 def __init__(self ):
			 super(AttnAggregator, self).__init__()      
			 self.dropout = 0.004          
			 self.trainPhase = True
			 self.obj_goal_split = None         
			 self.conv1 = pyg.nn.GATv2Conv( 5, 64, heads=8, dropout=0.0  )
			 self.conv2 = pyg.nn.GATv2Conv( 64*self.in_heads, 64, concat=False,heads=1, dropout=0.01  )      
			 self.SOFTo = nn.Softmax(dim=0)
			 self.SOFTg = nn.Softmax(dim=0)

		 def forward(self, x, edgeIndex):        
			 assert( not (self.obj_goal_split is None) )          
			 out = self.conv1(x,edgeIndex)         
			 out = self.conv2(out,edgeIndex)
			 out = flatten(out)
			 sp = self.obj_goal_split        
			 out = concat( (self.SOFTo(out[sp[0]:sp[1]]), self.SOFTg(out[sp[2]:     ])),0)
			 return (  out  )        

		 def set_phase(self, isTrainPhase ):
			 self.trainPhase = isTrainPhase
			 
		 def set_object_goal_split(self, split ):
			 self.obj_goal_split = split
</code></pre>
<p>===</p>
<pre><code class="lang-auto">### Adam optimizer ###
params = list( GNNmodel.parameters() )
optimizer = torch.optim.Adam( params, lr=0.005, weight_decay=5.0e-4 )
optimizer.zero_grad()
</code></pre>
Linker error while trying to load ptl model in android native

Harshavardhana: <p>Hello Everyone,<br>
I am new to the world of pytorch. I am trying to infer a sample pytorch model (converted to ptl) on mobile.</p>
<p>I have referred the link <a href="https://pytorch.org/tutorials/recipes/android_native_app_with_custom_op.html" class="inline-onebox" rel="noopener nofollow ugc">Making Native Android Application that uses PyTorch prebuilt libraries — PyTorch Tutorials 1.11.0+cu102 documentation</a> for building my application.</p>
<p>Instead of using a snapshot of pytorch android (‘org.pytorch:pytorch_android:1.6.0-SNAPSHOT’), I am using latest lite version ('org.pytorch:pytorch_android_lite:1.12.0). I have also modified CmakeLists.txt to search for libpytorch_jni_lite.so instead of pytorch_jni</p>
<p>While building the code, I am getting linker error to the function torch::jit::load(). How do i fix this problem</p>
<p>FAILED: …/…/…/…/build/intermediates/cmake/debug/obj/arm64-v8a/libpytorch_nativeapp.so<br>
: &amp;&amp; /home/harshavardhana/ProjectTools/AndroidSdk/ndk/21.0.6113669/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ --target=aarch64-none-linux-android28 --gcc-toolchain=/home/harshavardhana/ProjectTools/AndroidSdk/ndk/21.0.6113669/toolchains/llvm/prebuilt/linux-x86_64 --sysroot=/home/harshavardhana/ProjectTools/AndroidSdk/ndk/21.0.6113669/toolchains/llvm/prebuilt/linux-x86_64/sysroot -fPIC -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security   -O0 -fno-limit-debug-info  -Wl,–exclude-libs,libgcc_real.a -Wl,–exclude-libs,libatomic.a -static-libstdc++ -Wl,–build-id -Wl,–fatal-warnings -Wl,–no-undefined -Qunused-arguments -shared -Wl,-soname,libpytorch_nativeapp.so -o …/…/…/…/build/intermediates/cmake/debug/obj/arm64-v8a/libpytorch_nativeapp.so CMakeFiles/pytorch_nativeapp.dir/src/main/cpp/pytorch_native.cpp.o  …/…/…/…/build/pytorch_android_lite-1.12.aar/jni/arm64-v8a/libpytorch_jni_lite.so -llog -latomic -lm &amp;&amp; :<br>
CMakeFiles/pytorch_nativeapp.dir/src/main/cpp/pytorch_native.cpp.o: In function <code>Java_com_example_pytorchnativedemo_PytorchNative_load_1model': /home/harshavardhana/Project/PytorchNativeDemo/app/.cxx/cmake/debug/arm64-v8a/../../../../src/main/cpp/pytorch_native.cpp:40: undefined reference to </code>torch::jit::load(std::__ndk1::basic_string&lt;char, std::__ndk1::char_traits, std::__ndk1::allocator &gt; const&amp;, c10::optional<a>c10::Device</a>)’<br>
clang++: error: linker command failed with exit code 1 (use -v to see invocation)<br>
ninja: build stopped: subcommand failed.</p>

SQSong: <p>I have the same problem. Anyone help?</p>
Incompatible CPU Errors for lib-torch on a Linux Server

ThatGuyJamal: <p>So I’ve built my PyTorch code on my local machine and hooked it up to an API so the model can process data over the web. I used the cpp+ API to access the model library. Everything was working fine, however, I ran into some problems.</p>
<p>So on my VM, it was giving some errors. These are displayed in this <a href="https://github.com/guillaume-be/rust-bert/issues/370" rel="noopener nofollow ugc">GitHub issue</a> for the rust to PyTorch bindings. Im using the CPU version only, no GPU code.</p>
<p>After a few days of testing on different systems and thinking, I’ve concluded that PyTorch is not supported on Arm Processors. My local machine is AMD and it works well.</p>
<pre><code class="lang-auto">Architecture:           aarch64
CPU op-mode(s):       32-bit, 64-bit
Byte Order:           Little Endian
CPU(s):                 3
Vendor ID:              ARM
  Model name:           Neoverse-N1
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 3
    Socket(s):          1
    Stepping:           r3p1
    BogoMIPS:           50.00
</code></pre>
<p>Above, is my current VM host machine CPU information, and as seen from the <a href="https://github.com/guillaume-be/rust-bert/issues/370" rel="noopener nofollow ugc">GitHub issue</a> images, there is a problem with the torch_cpu.so .</p>
<p>If anyone knows any more information on this, please let me know. Ive been thinking of using docker to package my image for vms now but I still don’t think that will fix this CPU problem, might have to move to a different VM.</p>

ptrblck: <p>We are building and publishing docker containers for ARM-based server CPUs (tested on a Neoverse N1, too) <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch">here</a> so your build issues might be specific to Rust.</p>

ThatGuyJamal: <p>I was never running PyTorch in a docker container but natively on the system, but thanks I will look into this.</p>
Torch optimizer related issues

xinbai: <p>Write the loss function and backpropagation code yourself, and then do not call the loss function packaged by pytorch, nor the backpropagation function that encapsulates the loss function, write the backpropagation function yourself, that is, calculate the gradient by yourself. In this case, can the optimizer packaged by torch be used to update the parameters?</p>

srishti-git1110: <p>I would think, yes.<br>
Optimizer uses the <code>.grad</code> attribute of the optimizable parameters passed to it during initialization to update the parameter values.</p>
<p>So as long as <code>.grad</code> is populated, it should work.</p>

xinbai: <p>Thank you for your answer, but I still have two questions to ask:</p>
<p>The first question: For example, after I perform a forward propagation, I perform the back propagation, calculate the parameter gradient I need, and then pass the calculated parameter gradient to the optimizer, right?</p>
<p>The second question: how do I pass the gradient of the parameters to the optimizer? !I only see a list of parameters that can be passed that need to be optimized, so how do I pass the gradient?</p>

srishti-git1110: <aside class="quote no-group" data-username="xinbai" data-post="3" data-topic="180150">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xinbai/48/59178_2.png" class="avatar"> xinbai:</div>
<blockquote>
<p>and then pass the calculated parameter gradient to the optimizer, right?</p>
</blockquote>
</aside>
<p>No. You will need to store the calculated gradients in the <code>.grad</code> attributes of the respective parameters and then call <code>optimizer.step()</code> to be able to update the parameters using their <code>grad</code>.</p>
<p>If I may ask, is there a specific need to implement your own backward pass and not use PyTorch’s automatic differentiation via <code>autograd</code>?</p>

xinbai: <p>Yes, I implemented backpropagation by myself without using pytorch automatic differentiation, so in this case, can I still call the optimizer packaged by pytorch?</p>
Resources for learning current best-practices for structuring PyTorch code?

mks: <p>Does anyone have any recommendations for a recent style/best practices guide for writing pytorch code?</p>
<p>From my understanding, many deep learning projects end up sharing a lot of similar structure (data loading, model instantiation, train/test loops, analyzing training / loss dynamics). Is there anywhere I can learn the most common practices for structuring these methods? I’d like to prepare myself for collaborating with other ML/DL engineers in an industry context.</p>
<p>Thanks!</p>
Freeze few layers after a certain epoch

Zillur_Rahman: <p>I have a compute vision model with ResNet18 backbone. On top, there are two sequential branches for two different tasks. After a few epochs I see, one branch accuracy becomes saturated while the other one gradually increases. Now, I want to freeze the parameter update of the saturated branch to prevent overfitting. So, the condition would be either let’s say 95% accuracy of that branch or 30 epochs. How can I do that?</p>

ptrblck: <p>You could use the current accuracy as a condition and set the <code>.requires_grad</code> attributes of the parameters which should be frozen to <code>False</code>.</p>
Distributed Data Parallel Overlap batch Training

Phammanh26: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60818d413ac213ed44ef52d1afb7d096ed82ec12.png" data-download-href="https://discuss.pytorch.org/uploads/default/60818d413ac213ed44ef52d1afb7d096ed82ec12" title="Ảnh màn hình 2023-05-18 lúc 01.20.13"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60818d413ac213ed44ef52d1afb7d096ed82ec12_2_690x392.png" alt="Ảnh màn hình 2023-05-18 lúc 01.20.13" data-base62-sha1="dLJoPLGP0GcS25chr9T8GwymwQa" width="690" height="392" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60818d413ac213ed44ef52d1afb7d096ed82ec12_2_690x392.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60818d413ac213ed44ef52d1afb7d096ed82ec12_2_1035x588.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60818d413ac213ed44ef52d1afb7d096ed82ec12_2_1380x784.png 2x" data-dominant-color="EEEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Ảnh màn hình 2023-05-18 lúc 01.20.13</span><span class="informations">2430×1382 407 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I be overlap when try to use DDP (distributed data parallel). This image, show each batch on each epoch  used on both 2 GPU 0 and GPU 1.</p>
What is different in torch.ao.quantization.get_default_qat_qconfig and torch.ao.quantization.get_default_qconfig?

yyd199948: <pre><code class="lang-auto">qconfig = torch.ao.quantization.get_default_qat_qconfig("qnnpack")
&gt;&gt;&gt; print(qconfig)
QConfig(activation=functools.partial(&lt;class 'torch.ao.quantization.fake_quantize.FusedMovingAvgObsFakeQuantize'&gt;, observer=&lt;class 'torch.ao.quantization.observer.MovingAverageMinMaxObserver'&gt;, quant_min=0, quant_max=255, reduce_range=False){}, weight=functools.partial(&lt;class 'torch.ao.quantization.fake_quantize.FusedMovingAvgObsFakeQuantize'&gt;, observer=&lt;class 'torch.ao.quantization.observer.MovingAverageMinMaxObserver'&gt;, quant_min=-128, quant_max=127, dtype=torch.qint8, qscheme=torch.per_tensor_symmetric){})
&gt;&gt;&gt; qconfig = torch.ao.quantization.get_default_qconfig("qnnpack")
&gt;&gt;&gt; print(qconfig)
QConfig(activation=functools.partial(&lt;class 'torch.ao.quantization.observer.HistogramObserver'&gt;, reduce_range=False){}, weight=functools.partial(&lt;class 'torch.ao.quantization.observer.MinMaxObserver'&gt;, dtype=torch.qint8, qscheme=torch.per_tensor_symmetric){})
</code></pre>
<p>Can torch.ao.quantization.get_default_qconfig be used for QAT training，what is the difference between them？</p>

HDCharles: <p>PTQ is post training quantization, i.e. once you’ve got a model, you simply apply this type of quantization on top of it. The qconfigs include information about what type of quantization you want to apply to your model. This is what the default qconfig is set up for.</p>
<p>QAT is quantization aware training i.e. while training the model, you are finding weights that will work well given a specific type of quantization you will apply at the end. The qconfig contains the information for this quantization as well as functionality to simulate the quantization while still using fp32 numerics during training, since an actual quantized module doesn’t have gradients.</p>
<p>They are not interchangable.</p>
<p>for more information see <a href="https://pytorch.org/docs/stable/quantization.html#general-quantization-flow" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.0 documentation</a></p>
Tf.unique_with_counts PyTorch equivalent?

cdahms: <p>Good morning,</p>
<p>I’m translating code from TensorFlow to PyTorch.  There is a function in TF, <code>unique_with_counts</code>, that I can best explain with the example from the TF documentation <a href="https://www.tensorflow.org/api_docs/python/tf/unique_with_counts" class="inline-onebox" rel="noopener nofollow ugc">tf.unique_with_counts  |  TensorFlow v2.12.0</a></p>
<pre><code class="lang-auto">import tensorflow as tf

x = tf.constant([1, 1, 2, 4, 4, 4, 7, 8, 8], tf.float32)
print('\n' + 'x: ')
print(x)

unique_vals, unique_idxs, unique_val_counts = tf.unique_with_counts(x)

print('\n' + 'unique_vals: ')
print(unique_vals)

print('\n' + 'unique_idxs: ')
print(unique_idxs)

print('\n' + 'unique_val_counts: ')
print(unique_val_counts)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">x: 
tf.Tensor([1. 1. 2. 4. 4. 4. 7. 8. 8.], shape=(9,), dtype=float32)

unique_vals: 
tf.Tensor([1. 2. 4. 7. 8.], shape=(5,), dtype=float32)

unique_idxs: 
tf.Tensor([0 0 1 2 2 2 3 4 4], shape=(9,), dtype=int32)

unique_val_counts: 
tf.Tensor([2 1 3 1 2], shape=(5,), dtype=int32)
</code></pre>
<p>What is the best way to replicate this function in PyTorch?</p>

cdahms: <p>Well this was an easy one, it looks like simply changing some of the default parameters to <code>torch.unique</code> achieves the same functionality:</p>
<pre><code class="lang-auto">import torch

x = torch.tensor([1, 1, 2, 4, 4, 4, 7, 8, 8], dtype=torch.float32)

unique_vals, unique_idxs, unique_val_counts = torch.unique(x, sorted=True, return_inverse=True, return_counts=True)

print('\n' + 'unique_vals: ')

print(unique_vals)

print('\n' + 'unique_idxs: ')

print(unique_idxs)

print('\n' + 'unique_val_counts: ')

print(unique_val_counts)
</code></pre>
<p>result:</p>
<pre><code class="lang-auto">unique_vals: 
tensor([1., 2., 4., 7., 8.])

unique_idxs: 
tensor([0, 0, 1, 2, 2, 2, 3, 4, 4])

unique_val_counts: 
tensor([2, 1, 3, 1, 2])
</code></pre>
Inter-shard shuffling with torchdata datapipes

falckt: <p>Is there a way in <code>torchdata</code> to shuffle data across shards in a multiprocessing setting? In my current pipeline, I shuffle my shards before I apply the <code>sharding_filter</code> to distribute them among the worker processes and then do an intra-shard shuffle on the individual workers. However, this way batches generated by a worker process still contain samples from only one shard. Is there an elegant way to also shuffle the results returned by the individual workers?</p>
<p>My current pipeline looks similar to this</p>
<pre><code class="lang-python">base_dp = (
    dp.iter.FileLister(input_path, masks="*.wds")
    .shuffle()  # shuffle shards
    .sharding_filter()
    .open_files(mode="b")
    .load_from_tar()
    .map(decode_data)
    .webdataset()
    .shuffle(buffer_size=&lt;smaller than shard size&gt;)  # this is on a single worker --&gt; intra-shard shuffle
    .batch(batch_size)
)
</code></pre>
<p>My current approach is to define a dummy <code>datapipe</code> which forces the data back into the main process, and to perform the shuffle there.</p>
<pre><code class="lang-python">@dp.functional_datapipe("collect_from_workers")
class WorkerResultCollector(dp.iter.IterDataPipe):
    def __init__(self, source: dp.iter.IterDataPipe):
        self.source = source

    def __iter__(self) -&gt; Iterator:
        yield from self.source

    def is_replicable(self) -&gt; bool:
        """Method to force data back to main process"""
        return False

new_dp = (
    base_dp
    .collect_from_workers()
    .shuffle(buffer_size=&lt;n * size of batch size used in individual workers&gt;, unbatch_level=1)
    .batch(batch size)
)
</code></pre>
<p>Any feedback appreciated. Other way to handle sharded data, “official” collector dp to replace my hackish version, …</p>
Lstm model prediction converging to a value

imricky: <p>Hi, I’ve trained an lstm model on a time seres prediciotn task and when I use it to predict the whole time series it almost match the original, so it’s pretty accurate. But when I use it to predict one step at a time, appending the last prediction to the data and using the last “window size” item count the model predict values converging to a certain value, depending on the first input. I’ve also tried checking if the prediction on one single window size elements count and the whole dataset at the specified index is the same or similar, but it’s not. Any ideas? Thanks in advance</p>
Does pytorch use the cudatoolkit in Docker or the system

tangolin: <p>I installed cuda10.1 in the docker container. Upon running <code>nvcc --version</code> , I get</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
</code></pre>
<p>Now on my actual machine when <code>nvcc --version</code> is ran, I get</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85
</code></pre>
<p>If my pytorch in the docker container was installed with <code>pip install torch==1.5.1+cu101 torchvision==0.6.1+cu101</code>, does this mean that pytorch will simply use its own version of cuda?</p>

ptrblck: <aside class="quote no-group" data-username="tangolin" data-post="1" data-topic="140303">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/dfb087/48.png" class="avatar"> tangolin:</div>
<blockquote>
<p>does this mean that pytorch will simply use its own version of cuda?</p>
</blockquote>
</aside>
<p>Yes, the pip wheels and conda binaries ship with their own CUDA runtime (as well as cuDNN. NCCL etc.) so you would only need to install the NVIDIA driver. If you want to build PyTorch from source or a custom CUDA extension, the local CUDA toolkit will be used.</p>

ChristLBUPT: <p>Hello. I came across another situation: my docker-machine has cuda11.4 installed and I installed torch only through <code>pip install torch==1.13.1</code> (the version specification doesn’t end with +cu11x). While I will have torch.cuda.is_available() return <code>True</code> even though I neither specified the corresponding cuda version nor found any <code>.so</code> library files in my environment’s torch installation. Does it mean that pytorch will use the system cudatoolkit (11.4) instead?</p>

ptrblck: <p>No, this shouldn’t be the case and you could check <code>torch.version.cuda</code> to see which CUDA version in used in your binaries.</p>

ChristLBUPT: <p>Thank you for your reply! That’s right, I’ve found that pip will automatically download the corresponding cuda, cudnn and cublas if I don’t specify the corresponding cuda version of pytorch.</p>
Accuracy (and other metrics) with hierarchical classes

piojanu: <p>Hi!</p>
<p>I have such a problem: my classes have a tree structure and my model can predict a class at any level.</p>
<p>Example:</p>
<ul>
<li>With these two ground truth samples: 1/4/75 and 1/4/84</li>
<li>My model can output any class from: 1, 4, 75, 84</li>
<li>If it outputs 75, then only 1/4/75 is a respective positive sample.</li>
<li>If it outputs 84, then only 1/4/84 is a respective positive sample.</li>
<li>If it outputs 1 or 4, then both 1/4/75 and 1/4/84 are positive samples.</li>
</ul>
<p>By positive, I mean that by returning 75 or 84 the accuracy on those two samples is 1/2 and by returning 1 or 4 the accuracy on those two samples is 1.</p>
<p>My question is, how can I efficiently, using tensors, implement e.g. the accuracy metric? Moreover, I’d like to implement the Mean Average Precision, as my problem in general is a multi-class classification problem.</p>
<p>Thanks for any help e.g. links to appropriate literature!</p>

piojanu: <p>I’ve found this paper <a href="http://arxiv.org/abs/1306.6802" rel="noopener nofollow ugc">Evaluation Measures for Hierarchical Classification</a> that pretty much covers the theory. I’ll use the set-based measures as they seem simpler and address my case. Now, I was looking for an efficient implementation of set operations e.g. <em>batch</em> intersection of two tensors and I found this <a href="https://stackoverflow.com/questions/55110047/finding-non-intersection-of-two-pytorch-tensors" class="inline-onebox" rel="noopener nofollow ugc">python - Finding non-intersection of two pytorch tensors - Stack Overflow</a>. This seems to be it, but I need to dive deeper into the responses to check if it really solves my problem.</p>
How to enable standard libjpeg library instead of Pillow

Rajveer_B: <p>In my torchvision installation, the torchvision library defaults to using Pillow’s libjpeg version instead of the standard libjpeg available.</p>
<p><strong>Is there a way to force torchvision to use the the standard jpeg library instead of Pillow’s?</strong></p>
<p>Installed versions:<br>
torch                     1.13.1<br>
torchvision            0.14.1</p>
<p>Ubuntu 20.04 Linux  - OS</p>
How to vmap inside a tuple input?

annada: <p>Consider the tensors,</p>
<pre><code class="lang-python">x = tensor([[1., 1.],
            [1., 1.],
            [1., 1.],
            [1., 1.],
            [1., 1.]])
y = tensor([0., 0.])
</code></pre>
<p>I want to batched <code>torch.hstack</code> to output something like,</p>
<pre><code class="lang-python">torch.stack([torch.hstack((i, y)) for i in x])
</code></pre>
<p>which outputs,</p>
<pre><code class="lang-python">tensor([[1., 1., 0., 0.],
        [1., 1., 0., 0.],
        [1., 1., 0., 0.],
        [1., 1., 0., 0.],
        [1., 1., 0., 0.]])
</code></pre>
<p>I guess python list expansions are slow and probably will not propagate the gradient across the computation graph (or will it?). So I wanted to convert it to <code>torch.func.vmap</code> with something like,</p>
<pre><code class="lang-python">torch.vmap(torch.hstack, in_dims=(0, None))((x, y))
</code></pre>
<p>but obviously that doesn’t work because input to <code>torch.hstack</code> is a tuple, not a unpacked *tuple. What would be the correct way to do it?</p>

ptrblck: <aside class="quote no-group" data-username="annada" data-post="1" data-topic="179568">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/8491ac/48.png" class="avatar"> annada:</div>
<blockquote>
<p>I guess python list expansions are slow and probably will not propagate the gradient across the computation graph (or will it?).</p>
</blockquote>
</aside>
<p>Concatenating tensors will not break the computation graph as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([[1., 1.],
                  [1., 1.],
                  [1., 1.],
                  [1., 1.],
                  [1., 1.]], requires_grad=True)
y = torch.tensor([0., 0.], requires_grad=True)

z = torch.cat((x, y.unsqueeze(0).expand_as(x)), dim=1)
print(z)
# tensor([[1., 1., 0., 0.],
#         [1., 1., 0., 0.],
#         [1., 1., 0., 0.],
#         [1., 1., 0., 0.],
#         [1., 1., 0., 0.]], grad_fn=&lt;CatBackward0&gt;)

z.mean().backward()
print(x.grad)
# tensor([[0.0500, 0.0500],
#         [0.0500, 0.0500],
#         [0.0500, 0.0500],
#         [0.0500, 0.0500],
#         [0.0500, 0.0500]])
print(y.grad)
# tensor([0.2500, 0.2500])
</code></pre>
<p>so you might not need to use <code>vmap</code> for a simple concatenation.</p>

annada: <pre><code class="lang-python">z = torch.cat((x, y.unsqueeze(0).expand_as(x)), dim=1)
</code></pre>
<p>Didn’t know about the <code>expand_as()</code> function. Thank you.</p>
Gradient Accumulation failing

santurini: <p>I am trying to implement a train loop with mixed precision and gradient accumulation but when I do the second step it gives me this error:</p>
<pre><code class="lang-auto">Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
</code></pre>
<p>This is the code I am using:</p>
<pre><code class="lang-auto"># Loop over the dataset multiple times
print("Start Training ...")
while steps &lt; cfg.train.trainer.max_steps:
    dt = time.time()
    model.train()

    print('Loading Batches ...')
    for i, data in enumerate(train_dl):
        lr, hr = data[0].to(device), data[1].to(device)

        with torch.cuda.amp.autocast():
            sr, lq = model(lr)
            loss_dict = compute_loss(loss_fn, loss_dict, sr, hr, lq)
            loss = loss_dict["Loss"] / num_grad_acc

        print("Loss:", loss_dict["Loss"].item())
        metrics_dict = compute_metric(metric, metrics_dict, sr, hr)

        print("Scaling Loss ...")
        scaler.scale(loss).backward()

        if (i + 1) % num_grad_acc == 0:
            print("Updating Parameters at Step {} ...".format(i))
            scaler.step(optimizer)
            scaler.update()
            scheduler.step()
            optimizer.zero_grad()
</code></pre>

ptrblck: <p>It seems you are reuing <code>loss_dict</code> recursively, so could you check if this is attaching to the previous computation graph?</p>

santurini: <p>Yes that was exactly the problem! As my supervisor is saying “It just took a look at Patrick to understand the problem while we raged an entire afternoon to understand the problem, he’s a f***ing legend!”</p>
How to gather not tensor object in torch.distributed?

JulioZhao97: <p>What I want to do is gather not tensor object in torch.distributed. For example, I launch job using torchrun and have these dicts on 4 ranks:</p>
<pre><code class="lang-auto">P 0: {0: 1}
P 2: {2: 1}
P 4: {4: 1}
P 6: {6: 1}
P 8: {8: 1}
All: {}
</code></pre>
<p>what I want is</p>
<pre><code class="lang-auto">P 0: {0: 1}
P 2: {2: 1}
P 4: {4: 1}
P 6: {6: 1}
P 8: {8: 1}
All: {0: 1, 2: 1, 4: 1, 6: 1, 8: 1}
</code></pre>
<p>Is there any way to do this?</p>
Different Results of Whisper GELU on x86 and ARM CPU

Pi-Chuan: <p>Hello everyone,</p>
<p>I am working on the Whisper ASR model inference and getting different results on two different CPU architectures. On an Intel CPU, I am able to get normal results. However, on an ARM CPU, the model is unable to predict normal tokens, resulting in empty text. After checking, I found that the GELU function output is different on both CPUs. The MSE error of GELU results in layer 1 is 0.0048.</p>
<p>I am using the pre-trained Whisper-tiny.en model directly from Hguuingface, and the Python and PyTorch versions are the same on both CPUs, with all relevant libraries installed.</p>
<p>The ARM CPU is Cortex-A53 on Xilinx KV260. The x86 CPU is Intel Core i7.</p>
<p>I would like to ask what could be the possible cause of this issue and how to resolve it? Thank you.</p>

ptrblck: <p>Could you post a minimal and executable code snippet showing the difference in the gelu output on both CPU architectures, please?</p>
Should DALI cuda version same as local cuda or pytorch cuda?

LuoXin-s: <p>Pytorch distributes its package with builtin cuda, so I can use a pytorch with cuda 11.8 even I have cuda 12.0 in local. The question is which cuda version of DALI should I install so it can works correctly with my environments?</p>

ptrblck: <p>DALI doesn’t specify any CUDA minor version, so <code>nvidia-dali-cuda110</code> should work.</p>
RuntimeError: Could not export Python function call '_DDPSink'

Vrushank98: <p>I am trying to trace a model which was trained using DDP. I get this error while tracing the model:</p>
<pre><code class="lang-auto"> ops.profiler._record_function_exit(_1)
  _517 = ^_DDPSink(&lt;torch._C._distributed_c10d.Reducer object at 0x7fd53d9d1f70&gt;, {'static_graph': False, 'num_iterations': 0})(image, boxes1, _516)
  _518 = {"image": image, "boxes": boxes1, "density_pred": _516}
  return _518

Traceback (most recent call last):
  File "/content/Few-Shot-Object-Counting-PyTorch/tools/trace_model.py", line 160, in &lt;module&gt;
    main()
  File "/content/Few-Shot-Object-Counting-PyTorch/tools/trace_model.py", line 150, in main
    traced_model.save(save_path)
  File "/usr/local/lib/python3.10/dist-packages/torch/jit/_script.py", line 713, in save
    return self._c.save(str(f), **kwargs)
RuntimeError: 
Could not export Python function call '_DDPSink'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:
/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py(506): apply
/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py(1207): forward
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py(1488): _slow_forward
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py(1501): _call_impl
/content/Few-Shot-Object-Counting-PyTorch/tools/trace_model.py(83): forward
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py(1488): _slow_forward
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py(1501): _call_impl
/usr/local/lib/python3.10/dist-packages/torch/jit/_trace.py(1056): trace_module
/usr/local/lib/python3.10/dist-packages/torch/jit/_trace.py(794): trace
/content/Few-Shot-Object-Counting-PyTorch/tools/trace_model.py(109): trace
/content/Few-Shot-Object-Counting-PyTorch/tools/trace_model.py(148): main
/content/Few-Shot-Object-Counting-PyTorch/tools/trace_model.py(160): &lt;module&gt;
</code></pre>

ptrblck: <p>You might want to trace the model before wrapping it into <code>DDP</code>.</p>