akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Access the weights of neural network after each epoch

username: Vaibhav_Chahar
cooked: <p>Hi I have a autoregressive neural network. While training my network I want to see weight and biases in every epoch. How to access them?</p>
<p>"class Neural_Made(nn.Module):<br>
def <strong>init</strong>(self,n,m):<br>
super(Neural_Made,self).<strong>init</strong>()<br>
self.n=n<br>
self.m=m<br>
self.in_size=n<em>m</em>m<em>m</em>m<br>
self.register_buffer(‘Mask1’,torch.ones([self.in_size]*2))<br>
self.register_buffer(‘Mask2’,torch.eye(self.in_size))<br>
self.Mask1=torch.tril(self.Mask1)<br>
self.Mask2=torch.sub(self.Mask1,self.Mask2)</p>
<pre><code>    self.fc1=nn.Linear(self.in_size,self.in_size)
    nn.init.xavier_uniform_(self.fc1.weight)
    self.fc1.weight.data=torch.mul(self.fc1.weight.data,self.Mask2)
    print('weights1',self.fc1.weight.data)

    self.fc2=nn.Linear(self.in_size,self.in_size)
    nn.init.xavier_uniform_(self.fc2.weight)
    self.fc2.weight.data=torch.mul(self.fc2.weight.data,self.Mask1)
    print('weights2',self.fc2.weight.data)

    self.out=nn.Linear(self.in_size,self.in_size)
    nn.init.xavier_uniform_(self.out.weight)
    self.out.weight.data=torch.mul(self.out.weight.data,self.Mask1)
    print('weights3',self.out.weight.data)

    self.prelu=nn.PReLU()
    self.sig=nn.Sigmoid()

def forward(self,x):
    y=self.fc1(x)
    z=self.prelu(y)
    y1=self.fc2(z)
    z1=self.prelu(y1)
    x1=self.out(z1)
    x2=self.sig(x1)
    return x2"
</code></pre>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/vaibhav_chahar">@Vaibhav_Chahar</a>,</p>
<p>If you want the weighs/biases of your network, you can simply call,</p>
<pre><code class="lang-auto">params = [p for p in net.parameters()]
</code></pre>
<p>or if you want the layer info as  well as <code>dict</code> works well,</p>
<pre><code class="lang-auto">params = dict(net.named_parameters())
</code></pre>
<p>If you’re calling at each epoch, you may need to deepcopy the parameters if you’re looking at the weight/biases across epochs.</p>
Impossible to package pex application with torch2.0.1+cu18

username: jcuquemelle
cooked: <p>Hi community, I have the following issue :</p>
<p>Following this memory leak issue <a href="https://github.com/pytorch/vision/issues/4378" class="inline-onebox" rel="noopener nofollow ugc">Torchvision decode_jpeg memory leak · Issue #4378 · pytorch/vision · GitHub</a> that seems to be fixed by upgrading to cuda 11.8, I’m trying to bump my application to this version and run into the following issue:</p>
<p>I’m using Pex to build and ship the app, and it struggles to create the pex file because the torch2.0.1+cu118 is larger than  2GB: <code>torch-2.0.1+cu118-cp39-cp39-linux_x86_64.whl is too big (2265684151 bytes) to fit in memory</code> (I think the message is misleading here, as I have plenty available memory when building)</p>
<p>When using torch2.0.1+cu117, it works because it is not shipped as a big wheel, but as a torch wheel + nvidia wheels (nvidia_cuda_runtime_xxxx, nvidia_cublas_xxxxx, etc) and as such all individual wheels are much smaller so Pex has no issue to build the app.</p>
<p>Is there an alternative repository or a way to install  torch2.0.1+cu18 with separate wheels for nvidia libraries ?</p>
Equivalent function from tensorflow ( tf.config.experimental.set_memory_growth) to limit number of processes on GPU

username: tanvi
cooked: <p>In tensorflow, there is a function called <code>tf.config.experimental.set_memory_growth</code> (<a href="https://www.tensorflow.org/guide/gpu" rel="nofollow noopener">Details</a>) which allocates as much memory to the process as needed. Moreover, it doesn’t release the memory till the process runs. This prevents any other process from concurrently occupying the memory.<br>
This is a very useful functionality for cases when the GPU is a shared resource and your process has high but dynamic memory requirement.</p>
<p>Could someone tell if there is a similar option in pytorch to limit the number of processes which can share memory concurrently?</p>

username: ptrblck
cooked: <p>If I’m not mistaken, there is no functionality in PyTorch, as the memory will be dynamically allocated and cached.<br>
As an untested hack, you could try to allocate a huge tensor at the beginning of your script and delete it right after its creation. This should trigger PyTorch to allocate the memory block and put it into the cache without freeing it.</p>

username: fabian_schutze
cooked: <p>I think by now pytorch offers such an API. See <code>set_per_process_memory_fraction</code>  <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_per_process_memory_fraction.html#torch.cuda.set_per_process_memory_fraction" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.set_per_process_memory_fraction — PyTorch 2.0 documentation</a> .</p>

username: ptrblck
cooked: <p>It’s not the same as PyTorch now allows to set a limit while TF will eagerly flood the specified section to prevent other processes from using it.</p>

username: fabian_schutze
cooked: <p>Interesting! Thanks for the clarification!</p>
CUDA error 209: no kernel image is available for execution on the device

username: Hawk_Singh1
cooked: <p>Hi. i’m new to Pytorch and CUDA, I’m trying to run a llm locally and attempting to offload some of its layer to the GPU and encountering this error. Please advise.</p>
<p>This is the output for: python -m torch.utils.collect_env</p>
<p>PyTorch version: 2.0.1+cu118<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.8<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 10 Home Single Language<br>
GCC version: Could not collect<br>
Clang version: Could not collect<br>
CMake version: version 3.26.0-msvc3<br>
Libc version: N/A</p>
<p>Python version: 3.11.2 (tags/v3.11.2:878ead1, Feb  7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)] (64-bit runtime)<br>
Python platform: Windows-10-10.0.19045-SP0<br>
Is CUDA available: True<br>
CUDA runtime version: 12.2.128<br>
CUDA_MODULE_LOADING set to: LAZY<br>
GPU models and configuration: GPU 0: NVIDIA GeForce 930MX<br>
Nvidia driver version: 536.67<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>CPU:<br>
Architecture=9<br>
CurrentClockSpeed=2703<br>
DeviceID=CPU0<br>
Family=198<br>
L2CacheSize=512<br>
L2CacheSpeed=<br>
Manufacturer=GenuineIntel<br>
MaxClockSpeed=2904<br>
Name=Intel(R) Core™ i7-7500U CPU @ 2.70GHz<br>
ProcessorType=3<br>
Revision=</p>
<p>Versions of relevant libraries:<br>
[pip3] mypy-extensions==1.0.0<br>
[pip3] numpy==1.25.2<br>
[pip3] torch==2.0.1+cu118<br>
[pip3] torchaudio==2.0.2+cu118<br>
[pip3] torchvision==0.15.2+cu118<br>
[conda] Could not collect</p>

username: ptrblck
cooked: <p>Your 930mx should have a compute capability of 5.0 and should thus still be supported. Could you post the failing code, please?</p>
Profiler: How to associate memory allocations with Kernels

username: fabian_schutze
cooked: <p>Can I somehow attribute memory allocations recorded with the pytorch profiler with the associated kernels?</p>
<p>When using the profiler, I can see the allocation of individual functions (such as <code>aten::empty</code>) but I do not know how to link this to individual kernels. A view like the traces with memory allocations instead of runtime would be helpful.</p>
<p>I see that there is a lot of development in pytorch’s main branch for the profiler, <a href="https://pytorch.org/docs/main/profiler.html" class="inline-onebox" rel="noopener nofollow ugc">torch.profiler — PyTorch main documentation</a> (in particular <a href="https://pytorch.org/docs/main/profiler.html#torch.profiler._KinetoProfile.export_memory_timeline" class="inline-onebox" rel="noopener nofollow ugc">torch.profiler — PyTorch main documentation</a> ). The function seems to return a JSON with two arrays, but I have not been able to use it to locate the memory allocations as I would like to.</p>
Pytorch Lite on Linux

username: ksm
cooked: <p>When running lite modules on Linux using Pytorch, are the outputs guaranteed to be the same as when using the lite interpreter on Android?</p>
<p>The Pytorch Mobile page says “Available for iOS, Android and Linux” but I did not see a Linux specific build of the Pytorch Lite interpreter.</p>

username: vjayd
cooked: <p>Pytorch Lite is supported via libtorch build</p>
Export model to onnx and save its initializer with several independent files

username: 111357
cooked: <p>Is it possible to export the torch model to the ONNX model and save its weights with several independent files? Like what <a href="https://huggingface.co/docs/accelerate/v0.21.0/en/package_reference/accelerator#accelerate.Accelerator.save_model" rel="noopener nofollow ugc">Accelerator.save_model</a> do?</p>
Number of channels in the input image

username: deep999
cooked: <p>What is meant by " Number of channels in the input image" and " Number of channels in the output image" in paramaters of torch.nn.Conv2d here:<a href="http://pytorch.org/docs/master/nn.html" rel="nofollow noopener">http://pytorch.org/docs/master/nn.html</a></p>
<p>I’m sorry I’m new, but can’t find asnwer…some people say it’s colour channels, but then why it’s 1,6 and 6,16 in  beginner tutorial(<a href="http://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html</a>)? Or are these channels number of different filters in one bundle of convolution layers?</p>

username: ajbrock
cooked: <p>In general, a “2d” tensor in CNNs is of size “Batch x Channels x Height x Width.” For the actual input to the network, channels is usually 3 for RGB or 1 if it’s greyscale. For the outputs of layers in the network, “output channels” is analagous to the number of neurons, or the number of hidden units, of a layer. So, the latter–output channels are the number of filters in one layer, while input channels are the number of filters in the incoming layer.</p>

username: Alio
cooked: <aside class="quote no-group" data-username="ajbrock" data-post="2" data-topic="9455">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajbrock/48/139_2.png" class="avatar"> ajbrock:</div>
<blockquote>
<p>3 for RGB or 1 if it’s greyscale</p>
</blockquote>
</aside>
<p>Where would you see this and be able to change in a typical pytorch CNN?</p>

username: mikbuch
cooked: <p>Usually you specify the number of channels when defining your architecture (see, e.g. this example: <a href="https://github.com/AghdamAmir/3D-UNet/blob/main/unet3d.py#L132" rel="noopener nofollow ugc">https://github.com/AghdamAmir/3D-UNet/blob/main/unet3d.py#L132</a>), if you are working with pre-defined architectures there is not much you can do in terms of changing the number of input channels. If you have a greyscale image, and you wish to use it in a 3-channel RGB-indended network, you can always duplicate the dimension on run it like that.</p>

username: mikbuch
cooked: <p>See also, for instance, this link on defining a neural network layer, with specific number of input channels: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html" class="inline-onebox" rel="noopener nofollow ugc">Conv3d — PyTorch 2.0 documentation</a></p>
Why a layernorm change the whole model?

username: dreamG
cooked: <p>I am using several residual blocks and then following the head part the model. I tried two different architectures  found that the models can different from each via the existing of the layer normalizaion.</p>
<h3><a name="the-is-my-model-structures-1" class="anchor" href="#the-is-my-model-structures-1"></a>The is my model structures</h3>
<p>input-&gt; residual blocks → residual blocks → head part;</p>
<h3><a name="each-of-residual-blocks-comprise-the-following-operation-2" class="anchor" href="#each-of-residual-blocks-comprise-the-following-operation-2"></a>each of residual blocks comprise the following operation:</h3>
<pre><code class="lang-auto">  -&gt; conv1d -&gt; layernorm -&gt; conv1d -&gt; GELU -&gt;  +
|______________________________________|^ 
                         skip  connection
</code></pre>
<h3><a name="the-following-are-two-different-head-parts-of-the-model-3" class="anchor" href="#the-following-are-two-different-head-parts-of-the-model-3"></a>The following are two different head parts of the model</h3>
<pre><code class="lang-auto">* the first is : 
     max(along the sequence dimension) -&gt; layerNorm -&gt; Linear(in channels, num of classes)
</code></pre>
<pre><code class="lang-auto">* the second is 
    max(along the sequence dimension) -&gt; linear(in channels, in channels,) -&gt; LayerNom-&gt; Linear(in channels, num of classes)
</code></pre>
<p>The result is that: the performance of the first model can achieve 100% on training set while 84% for test test. The same number of the second model is 0.1 %  on both train set and test set.</p>
<p>I can not understand why I put the linear operation between the max operation and the LayerNorm making so much negative effects on the performance of the model.</p>
How to create video dataset without loading from the harddrive?

username: MiraWelner
cooked: <p>Hello!<br>
I’m a beginner with pytorch but fairly good with python. I am creating a dataset where simple, short videos are catagorized with labels from 1 to 10.</p>
<p>I am trying to make a dataset object and have previously been using the built in LabeledVideoDataset class. However now I am using a dataset that I am not allowed to download because it is sensitive data. I can only scan the videos from my lab’s site without them being downloaded to my computer. I have data variables that I want to train my model on.</p>
<p>I am used to TensorFlow where you load the data and make it into a dataset from there. However for some reason it looks like every dataset loader i can find, you need to put in a path variable that tells you where to load the data from your file system. If I put the dataset on my hard drive I will be in legal trouble. If anybody could tell me how to do the tensorflow equivalent of model.fit(x,y) where the x and y are just variables, it would be much appreciated!</p>
<p>sorry for the beginner question</p>

username: ptrblck
cooked: <p>Tensorflow would also need to load the data eventually, so could you describe how it’s done?<br>
PyTorch datasets often use lazy data loading, so storing only the path and allowing the actual sample to be loaded and processed in the <code>__getitem__</code> once it’s needed will reduce the memory usage compared to preloading the entire dataset.</p>
Calculate the gradient of pytorch model

username: Mohamed_Nabih
cooked: <p>I have a model trained for four epochs and I saved this model after each epoch. Is there is a way to calculate the gradient as the difference between two models I mean between the saved model after the fourth epoch and the model saved after the third epoch.</p>

username: ptrblck
cooked: <p>I’m unsure what exactly your use case is, but in case you want to calculate any difference between parameters you could directly iterate these as seen here:</p>
<pre><code class="lang-python">modelA = models.resnet18()
modelB = models.resnet18()

diffs = {}
for (name, paramA), (paramB) in zip(modelA.named_parameters(), modelB.parameters()):
    diffs[name] = (paramA - paramB).abs()
</code></pre>

username: Mohamed_Nabih
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your reply, what I want to do it to calculate the gradient as the difference between the two model weights i.e.<br>
gradient = (W_n ) - (W_n-1), where n is the number of epochs.</p>

username: ptrblck
cooked: <p>My code shows how to calculate the difference between the parameters of two different models. In your case both models would have been trained for some epochs while I just use randomly initialized models as an example.</p>
Torch.cuda.synchronize() after to( device )

username: Zai
cooked: <p>Hi, when running certain operations, say a timer start, it is usually required to do a synch to assure correct behavior. However, I have seen several examples doing a synch after creating and transferring a tensor to a cuda device ( and nothing before ). Is that actually necessary, assuming non_blocking=False? Also isn’t a non-blocking transfer generally a synchronization point?</p>

username: ptrblck
cooked: <p>No, you won’t need a full device sync when transferring data. The <code>non_blocking</code> argument allows an async transfer w.r.t. the host. To move the data async w.r.t. other CUDA ops you would need to use a custom <code>CUDAStream</code> and synchronize it with the default stream before using the data, which is an advanced use case.</p>

username: Zai
cooked: <p>Thank you. Of course I was referring to HtoD transfers, not DtoD. Should have made that clearer. Just to reiterate the latter point: a ( blocking ) HtoD transfer will be a synchronization point ( for the respective stream, usually the default one ), or am I wrong?</p>

username: ptrblck
cooked: <p>My explanation also refers to HtoD copies, not DtoD, so I’m unsure what causes the misunderstanding.<br>
Yes, if <code>non_blocking=True</code> is used the data will be copied using the default stream asynchronously w.r.t. the host.</p>

username: Zai
cooked: <p>Ah, I see. I interpreted “async wrt other CUDA ops” as referring to a second device. So, nevermind. thank you.</p>
Logarithmic Number System (LNS) Extension in Pytorch

username: hossein.masoum
cooked: <p>Hi everyone,</p>
<p>I want to replicate the LNS-Madam paper (<a href="https://arxiv.org/pdf/2106.13914.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2106.13914.pdf</a>) that is using a logarithmic number system for DNN. Hence, I need to implement all the arithmetic operations in LNS format. For example, multiplications turn to addition and addition itself will be more like a lookup table.</p>
<p>In the paper, It is mentioned that "To evaluate accuracy, we simulate LNS-Madam using a PyTorch-based neural network quantization library that implements a set of common neural network layers (e.g., convolution, fully connected) for training and inference in both full and quantized modes [14]. The baseline library supports integer quantization in a fixed-point number system, and we further extend it to support LNS. ". Does this extension need to only involve the basic arithmetic operations (and then all of the operations/kernels in Pytorch will automatically adapt that)? Or do I need to reimplement all the kernels? In either case, I would appreciate any pointers on how to get started with this.</p>
<p>Thank you!</p>
Trying to optimize the gradient as part of the loss

username: shubbey
cooked: <p>I have a network that is trying to optimize a tensor x as well as the gradient of that tensor wrt another tensor r, eg dx/dr.</p>
<pre><code>        y = torch.autograd.grad(x,r)
</code></pre>
<p>when I run this without DDP, I can minimize both quantities as part of the loss. When I use DDP on 2 gpus, the tensor x shows the same loss value on both gpus, but the gradient tensor y has different values:</p>
<pre><code># training 
    ...
    Nx = 0
    Ny = 0
    x_err = 0
    y_err = 0

    training_sampler.set_epoch(epoch)
    for batch in training:
        x_loss,y_loss,nx,ny = batch.loss(model,cuda) # runs through network
        loss = x_loss/nx + y_loss/ny
        Nx += nx
        Ny += ny

        optimizer.zero_grad()
        loss.backward() # will sync?

        optimizer.step()
        x_err += reduce(x_loss).item()
        y_err += reduce(y_loss).item()
    
    print (x_err/Nx, y_err/Ny) 
</code></pre>
<p>where:</p>
<pre><code>    def reduce(T):
        dist.all_reduce(T, op=dist.ReduceOp.SUM)
        T /= float(dist.get_world_size())
        return T
</code></pre>
<p>output:</p>
<pre><code>27140.88438,  35.89582 # gpu 0
27140.88438,  37.94447 # gpu 1
</code></pre>
<p>I am extremely new to parallel computing and honestly the behind the scenes seems like magic to me, but it’s my understanding that loss.backward() handles the dirty work of syncing the gpus.  I’m assuming that my problem is due to the manual torch.autograd.  Is there some obvious explanation as to how this works and how I might fix it?  Thank you.</p>

username: fduwjj
cooked: <p>Can you provide a minimum code which we can repro the result? Especially, how you wrap your model with DDP. Thanks!</p>

username: shubbey
cooked: <p>Thanks.  I will try to make a simple example.  It’s very likely I’m doing something stupid or just don’t understand how this works.</p>

username: shubbey
cooked: <p>After much debugging I realized that this is not a bug.  The issue was that individually my nx,ny values are different per sample so the gpus were receiving different information.   My bad!</p>
Correct way to avoid torch.compile recompilations

username: vadimkantorov
cooked: <p>What is the correct way to use <code>torch.compile</code> to avoid dynamic shape recompilations or over-specialization?</p>
<pre><code class="lang-python">min_sum = torch.compile((lambda a, b: torch.min(a, b).sum(dim = -1)))
</code></pre>
<p>I’m interested in passing in shapes:</p>
<ul>
<li>(192, ) x (192, ) → scalar; (96, ) x (96, ) → scalar</li>
<li>(B, N, 1, 192) x (B, 1, N, 192) → (B, N, N); (B, N, 1, 96) x (B, 1, N, 96) → (B, N, N)</li>
</ul>
<p>Can torch.compile generate kernels which would avoid allocation and computation of the intermediate <code>(B, N, N, 192)</code> tensor?</p>
<p>So basically I need two modes: a static shape mode and a dynamic shape mode (+broadcasting).</p>
<p>Can I express this? How to get the best speed with this?</p>
<p>Ideally, I’m interested both in CPU and CUDA codegen.</p>
<p>Also, if I call this <code>torch.compile(lambda ...)</code> in the code multiple times, will it share the compilation cache between all invokations?</p>
<p>Can I somehow force AOT pre-compilation for some set of shapes dtypes?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: vadimkantorov
cooked: <p>I created a GitHub for my naive bite at this task: <a href="https://github.com/pytorch/pytorch/issues/106466" class="inline-onebox" rel="noopener nofollow ugc">min_sum 3x slower on torch.compile (torch 2.0.1, CPU) · Issue #106466 · pytorch/pytorch · GitHub</a></p>
Torchaudio-tranforms.py - mat1 and mat2 shape cannot be multiplied

username: mala
cooked: <p>Hi all,<br>
I am using torch audio = 0.5.0 (as the code already uses it, I too gave same version). While executing code error occurs as :RuntimeError: mat1 and mat2 shapes cannot be multiplied (258x577 and 201x192)<br>
Error is traced to file torchaudio/transforms/transforms.py in line<br>
mel_specgram = torch.matmul(specgram.transpose(-1, -2), self.fb).transpose(-1, -2).<br>
Kindly suggest a solution for the issue.</p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet reproducing the error, please?</p>

username: mala
cooked: <p>The code is executed for Voice conversion.<br>
try:<br>
%tensorflow_version 2.x<br>
except Exception:<br>
pass<br>
import tensorflow as tf<br>
!pip install soundfile<br>
!pip install --no-deps torchaudio==0.5.0</p>
<hr>
<p>Execution sequence - 4-&gt;3-&gt;2-&gt;1<br>
1 ) torch.set_default_tensor_type(‘torch.cuda.FloatTensor’)</p>
<p>specobj = Spectrogram(n_fft=6<em>hop, win_length=6</em>hop, hop_length=hop, pad=0, power=2, normalized=True)<br>
specfunc = specobj.forward<br>
melobj = MelScale(n_mels=hop, sample_rate=sr, f_min=0.)<br>
melfunc = melobj.forward</p>
<ol start="2">
<li>def melspecfunc(waveform):<br>
specgram = specfunc(waveform)<br>
mel_specgram = melfunc(specgram)<br>
return mel_specgram</li>
</ol>
<p>3)def prep(wv,hop=192):<br>
S = np.array(torch.squeeze(melspecfunc(torch.Tensor(wv).view(1,-1))).detach().cpu())<br>
S = librosa.power_to_db(S)-ref_level_db<br>
return normalize(S)</p>
<ol start="4">
<li>prep(x) → x = data[i]  , i in range(data.shape[0]) and ‘data’ is the wave form array</li>
</ol>
<p><strong>Error</strong> - Same error is encountered for the code even if I change the audio dataset.<br>
/usr/local/lib/python3.10/dist-packages/torchaudio/transforms/_transforms.py in forward(self, specgram)<br>
mel_specgram = torch.matmul(specgram.transpose(-1, -2), self.fb).transpose(-1, -2)<br>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (258x577 and 201x192)</p>
Scaler.step(optimizer) in FP16 or FP32?

username: caesar025
cooked: <p>When using the recipe for training with AMP and GradScaler, i.e.:</p>
<pre><code class="lang-auto">                scaler.scale(loss).backward()
                scaler.step(optimizer)              
                scaler.update()
</code></pre>
<p>Is the optimizer step performed in full or half precision? And could this lead to issues in regard to very, very small learning rates?</p>

username: ptrblck
cooked: <p>The model parameters are kept in <code>float32</code> and not transformed to the lower precision <code>dtype</code>. The <code>optimizer</code> thus applies the update on the <code>float32</code> parameters.</p>
Graph returned by model_to_graph is not subscriptable anymore?

username: parikshit
cooked: <p>After the bump from torch 1.11.0 to 2.0.1 it seems the graph returned from model_to_graph is not subscriptable or have the functions to get a node attribute changed?!</p>
<pre><code class="lang-auto">from torch.onnx.utils import _model_to_graph


graph, params_dict, torch_out = _model_to_graph(
        model=model,
        args=args,
        do_constant_folding=False,
        training=TrainingMode.EVAL,
        operator_export_type=operator_export_type,
        dynamic_axes={},
    )
</code></pre>
<p>After this if I perform operations on the graph via node like:</p>
<pre><code class="lang-auto">foo(node[att_name]) for att_name in node.attributeNames()
</code></pre>
<p>I get error stating <strong>‘torch._C.Node’ object is not subscriptable</strong><br>
Is this related to _model_to_graph or definition of torch.Node has changed</p>
How to speedup the following code

username: Shivani_Sri_Varshini
cooked: <p>I’m trying to compute a matrix with pairwise distances. Following is my code, how to optimize the code to run faster.</p>
<pre><code>          for (i,a),(j,b) in itertools.combinations(enumerate(X_meaned),2):
                cov[i,j],cov[j,i]=self.compute(a,b)
</code></pre>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/shivani_sri_varshini">@Shivani_Sri_Varshini</a>,</p>
<p>You’ll need to include a more complete minimal reproducible example before people can help debug your code.</p>

username: Shivani_Sri_Varshini
cooked: <p>@ <a href="https://discuss.pytorch.org/u/AlphaBetaGamma96">AlphaBetaGamma96</a></p>
<p>X_meaned is torch tensor with dimensions B * M * N where B is batch_size M * N is 2D data. I would like to compute the distances of 2D samples within the batch. Code that i mentioned is giving an output of dimension B*B which is expected but for that i’m considering each sample in batch and computing the distance with respect to all other samples in batch.<br>
self.compute can be euclidean distance or cosine distance function</p>
Dtensor: how to get the "global tensor" on each rank after sharding

username: LM001
cooked: <p>Hi,</p>
<p>Currently, I am sharding the tensor_a with 4 processes on cpu. I just wondering after sharding the tensors. Is there anyway to get back the global tensor on each rank?</p>
<pre><code class="lang-auto">    tensor_a = torch.rand(1,1,12,10)
    tensor_a = distribute_tensor(tensor_a,mesh,[Shard(dim=2)])
</code></pre>
<p>Thanks</p>

username: wconstab
cooked: <aside class="quote no-group" data-username="LM001" data-post="1" data-topic="185531">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/46a35a/48.png" class="avatar"> LM001:</div>
<blockquote>
<p><code>distribute_tensor</code></p>
</blockquote>
</aside>
<p>You should be able to do this.  Can you try something like the following?</p>
<p><code>replica_tensor = distribute_tensor(tensor_a,mesh,[Replicate()])</code></p>
TypeError: FullyShardedDataParallel.__init__() got an unexpected keyword argument 'ignored_parameters'

username: dhyani15
cooked: <p>I am trying to run this code <a href="https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/main/accessory/main_finetune.py" rel="noopener nofollow ugc">here</a> and the snippet below is giving me the error in the subject, Pytorch nightly build 2.1.0.dev20230801, cuda 12.1 Python 3.11.4</p>
<pre><code class="lang-auto">model = FSDP(
        model,
        process_group=fs_init.get_data_parallel_group(),
        auto_wrap_policy=functools.partial(
            transformer_auto_wrap_policy,
            transformer_layer_cls=[] if model.is_peft else [TransformerBlock],
        ),
        limit_all_gathers=True,
        use_orig_params=True,
        sync_module_states=True,
        mixed_precision=MixedPrecision(
            param_dtype=mixed_precision_dtype,
            reduce_dtype=mixed_precision_dtype,
            buffer_dtype=mixed_precision_dtype,
        ),
        sharding_strategy={
            "sdp": ShardingStrategy.SHARD_GRAD_OP,
            "ddp": ShardingStrategy.NO_SHARD,
            "fsdp": ShardingStrategy.FULL_SHARD,
        }[args.data_parallel],
        device_id=device,
        ignored_parameters=[param for param in model.parameters() if not param.requires_grad]
    )
</code></pre>
<p>Can you please help me understand why is this happening, in the docs ignored_parameters is indeed a argument, so why does it say an expected argument?</p>

username: wconstab
cooked: <p>Sorry for the confusion.  ignored_parameters is indeed gone, replaced by <code>ignored_states</code>. if you look at the latest docs (<a href="https://pytorch.org/docs/main/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel" class="inline-onebox" rel="noopener nofollow ugc">FullyShardedDataParallel — PyTorch main documentation</a>) or look at the code (<a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/fsdp/fully_sharded_data_parallel.py#L404" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/distributed/fsdp/fully_sharded_data_parallel.py#L404</a>) you can see the info about it - iiuc you can just continue to pass the same list of params to ignored_states, so it should be a minor change.</p>
<p>Note: the docs you may have looked at are probably for ‘stable’ (2.0) not latest (nightly/unstable).</p>
How do I identify the ID number of test data misclassified by BERT?

username: CSstudent
cooked: <p>I am using BERT to classify text segments extracted from ASR-generated transcripts. There are multiple segments per participant. I have stored the data in a dataframe with the following column names: Participant_ID, Segment_Text and Diagnosis (i.e. label). I have successfully trained my dataset and am trying to identify which segments of text (i.e. Segment_Text) and Participant_ID the model misclassified - how would I go about doing this? I have provided the code for the custom Dataset class and Evaluation class (i.e. testing the BERT model) below:</p>
<p>Custom Dataset Class</p>
<pre><code class="lang-auto">tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
labels = {'HC':0, 'PK':1} 

class Dataset(torch.utils.data.Dataset):

def __init__(self, df):

    self.participantIDs = df['Participant_ID']
    self.labels = [labels[label] for label in df['Diagnosis']]
    self.texts = [tokenizer(text, padding='max_length', max_length = 512, truncation=True, return_tensors="pt") for text in df['Transcript_Segment']]

def classes(self):
    return self.labels

def __len__(self):
    return len(self.labels)

def get_batch_labels(self, idx):
    # Fetch a batch of labels
    return np.array(self.labels[idx])

def get_batch_texts(self, idx):
    # Fetch a batch of inputs
    return self.texts[idx]

# def get_batch_participant_ids(self, idx):
#     # Fetch a batch of inputs
#     return self.participantIDs[idx]

def __getitem__(self, idx):

    batch_texts = self.get_batch_texts(idx)
    batch_y = self.get_batch_labels(idx)
    # batch_participant_ids = self.get_batch_participant_ids(idx)

    return batch_texts, batch_y
</code></pre>
<p>Evaluate BERT model class</p>
<pre><code class="lang-auto">def evaluate(model, test_data):

incorrect_samples = []
test = Dataset(test_data)

test_dataloader = torch.utils.data.DataLoader(test, batch_size=8)

use_cuda = torch.cuda.is_available()
device = torch.device("cuda" if use_cuda else "cpu")

if use_cuda:

    model = model.cuda()

total_acc_test = 0
with torch.no_grad():

    for test_input, test_label in test_dataloader:

          test_label = test_label.to(device)
          mask = test_input['attention_mask'].to(device)
          input_id = test_input['input_ids'].squeeze(1).to(device)

          output = model(input_id, mask)
          _, pred = torch.max(output,1)
          idxs_mask = ((pred == test_label) == False).nonzero()
          print(idxs_mask)
          incorrect_samples.append(input_id[idxs_mask].cpu().detach().numpy())

          acc = (output.argmax(dim=1) == test_label).sum().item()
          total_acc_test += acc

print(f'Test Accuracy: {total_acc_test / len(test_data): .3f}')
print(incorrect_samples)
</code></pre>
How to load my dataset?

username: Mah_Neh
cooked: <p>I read the pytorch docs and wrote a simple network.</p>
<p>I don’t understand how to load my data and pass it to the model.</p>
<p>Can you give me a minimal example please ? or any snippet to get started ?</p>
<p>I do not have labels in a particular format yet, so that does not matter much just now.</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html">This data loading tutorial</a> might be a good starter.</p>

username: Mah_Neh
cooked: <p>What I currently tried is:</p>
<pre><code class="lang-python">from os import path, listdir
from torchvision.io import read_image

for item in listdir("./images/"):
    fullpath = path.join("./images/", item)
    print(fullpath)
    read_image(fullpath)
</code></pre>
<p>would this make sense ?</p>

username: ptrblck
cooked: <p>Yes, your code would manually load the data but would not use a <code>Dataset</code> or <code>DataLoader</code>. As described in the tutorial you could use these to e.g. shuffle and batch the data.</p>

username: Mah_Neh
cooked: <p>Thank you, it looked impossible yesterday but got it done today. Doing it manually reduced the overhead of reading the code (I’m not a python dev.)</p>
CUDA error: unspecified launch failure whe running the program for the second time

username: DawnChou
cooked: <p>Hi,</p>
<p>I came across the problem</p>
<blockquote>
<p>CUDA error: unspecified launch failure</p>
</blockquote>
<p>when I tried running the model for a second time in a python script. The model is in the function <code>RunmyModel()</code>, and I need to call the function two times in the python script. However, when the program call the function at the second time, it showed the CUDA error. Specifically, the code ended at</p>
<blockquote>
<p>File <a href="https://vscode-remote+ssh-002dremote-002bhtgc1t-002ecs-002ecityu-002eedu-002ehk.vscode-resource.vscode-cdn.net/home/grads/yipingzou2/integration/~/integration/TCRclub.py:160" rel="noopener nofollow ugc">~/integration/testmodel.py:160</a>, in testNet.to(self)<br>
158 for name, param in self.<strong>dict</strong>.items():<br>
159 if name not in [‘k’]: → <br>
160 self.<strong>dict</strong>[name] = param.cuda()<br>
RuntimeError: CUDA error: unspecified launch failure</p>
</blockquote>
<p>It seems the error occurred due to <code>.to()</code> when transferring the model to GPU. Aslo , I noticed that the GPU memory was not free after the first running.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ea800ef11a400885066e2a60fa9ae01e2375487.png" data-download-href="https://discuss.pytorch.org/uploads/default/3ea800ef11a400885066e2a60fa9ae01e2375487" title="1690956617483"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ea800ef11a400885066e2a60fa9ae01e2375487.png" alt="1690956617483" data-base62-sha1="8Why3B9odsWI7ZxSTDZgP1IqqHB" width="690" height="372" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1690956617483</span><span class="informations">726×392 9.29 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried <code>torch.cuda.empty_cache()</code> but it didn’t work. However, when I ran the same script in another machine with the same version of torch and cuda, the error does not occurred even though the GPU memory still was not free after the first running. How to solve the problem?</p>
<p>PS. I checked <code>dmesg</code>, here is the info:</p>
<blockquote>
<p>[    5.474440] [drm] [nvidia-drm] [GPU ID 0x00000b00] Loading driver<br>
[4201635.872634] NVRM: GPU at PCI:0000:0b:00: GPU-cab56c2d-811d-98fb-d3de-52e2bb36782d</p>
</blockquote>

username: ptrblck
cooked: <p>I’m unsure what the <code>dmesg</code> output should indicate, but are you able to use the GPU at all afterwards? If not your system might be dropping it which should be indicated by XIDs in the <code>dmesg</code> output.</p>
How to compute cosine similarity for a mini batch of samples for 2d data

username: Shivani_Sri_Varshini
cooked: <p>I have an input X of dimension B<em>S</em>D where B is batch_size, S is sample_size, and D is embedding_dimension. I want to compute cosine similarity for X with itself i.e., samples in batch(all pairs). Cosine_similarity is done for 1D or 2D. How to compute efficiently for mini-batch of samples (for all possible pairs). The expected output of dimension 2 * B * S * S. Please help to solve this issue. Thanks in advance.</p>
Dataloader on macOS with Apple M1 chip creates multiple processes

username: XIE-Fangyuan
cooked: <p>Iterating a <code>Dataloader</code> on macOS with M1 Apple chip creates multiple processes. The number of process is <code>num_workers + 1</code>. Each process seems to load the <code>.py</code> file again, and executes the top-level statement again. The same script does not cause the problem on a Linux server. I guess this might have something to do with the <code>multiprocessing_context</code> argument of <code>Dataloader</code>, but I did not find out a solution.</p>
<p>Here is a short test code:</p>
<pre><code class="lang-python">
import logging

from torch.utils.data import DataLoader
from torchvision.datasets import FashionMNIST
from torchvision.transforms import ToTensor

logger = logging.getLogger(name=__name__)
logger.setLevel(level=logging.DEBUG)
console_handler = logging.StreamHandler()
console_handler.setFormatter(fmt=logging.Formatter(fmt=f'[PID=%(process)d][%(name)s] %(msg)s'))
logger.addHandler(console_handler)

logger.warning('This message will be printed by `num_workers` + 1 processes.')


def main():
    data_loader = DataLoader(
        dataset=FashionMNIST(root='./data', train=True, transform=ToTensor()),
        batch_size=128, shuffle=True, num_workers=4,
    )

    for _ in data_loader:
        i = 1


if __name__ == '__main__':
    main()
</code></pre>
<p>On macOS M1, it prints:</p>
<pre><code class="lang-plaintext">[PID=71509][__main__] This message will be printed by `num_workers` + 1 processes.
[PID=71513][__mp_main__] This message will be printed by `num_workers` + 1 processes.
[PID=71514][__mp_main__] This message will be printed by `num_workers` + 1 processes.
[PID=71515][__mp_main__] This message will be printed by `num_workers` + 1 processes.
[PID=71516][__mp_main__] This message will be printed by `num_workers` + 1 processes.
</code></pre>
<p>and on Linux, it prints:</p>
<pre><code class="lang-plaintext">[PID=71509][__main__] This message will be printed by `num_workers` + 1 processes.
</code></pre>
[resolved] Double unsqueeze

username: alexis-jacq
cooked: <p>Hi,</p>
<p>Is there a smart way to do: <code>my_tensor.unsqueeze(0).unsqueeze(0)</code> in one shot?</p>
<p>something like <code>my_tensor.unsqueeze(0,1)</code></p>
<p>It’s not a big deal but if you want to had 3 fake dimensions, the code line stops to look serious</p>

username: alexis-jacq
cooked: <p>How, just <code>my_tensor.view(1,1,-1)</code> did it… ok sorry for the topic…</p>

username: trougnouf
cooked: <p>This flattens the -1 dimension</p>

username: Medhat
cooked: <p>you can use <code>my_tensor(*my_tensor.shape,1,1)</code></p>

username: amiasato
cooked: <p>This was one of the first results in Google, so for anybody out there, if you don’t want to flatten the last dim, <code>my_tensor[None, None, ...]</code> currently does the job.</p>
Customizing PyTorch: How to test/fine-tune a model using a new data type that has different arithmetics for basic operations (+,-,/,*) compared to float in PyTorch?

username: phgh
cooked: <p>Hi,</p>
<p>I want to use a new data representation instead of float for fine-tuning/testing a model (e.g., DNN or LLM) in Pytorch. The basic operations (add/sub/multiply/division) in my data type is different from floating point. My question is if it is possible to implement these operations (+,-,*,/) and force all of functions in Pytorch (e.g., torch.add(), torch.sum(), torch.nn.Linear(), conv2d, etc.) to use my basic arithmetic implementation? If so, could you please guide me how can I do it?</p>
<p>Because I think otherwise it takes so much time and effort; first, I have to find which functions my model calls (which I dont know how to do it) and, then, I have to replace them one by one. This becomes complicated for a large model.</p>
<p>I found this <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">link</a> from Pytorch that shows how to extend pytorch. But it seems that it is not comprehensive enough to answer my question.</p>
<p>Thank you very much!</p>
How do I train an LSTM for experiment data which resets after an interval?

username: Harish_Uthravalli
cooked: <p>I am using an LSTM model for Multivariate Time series forecasting where my dataset is a collection of experiments that are performed for a certain time. To train the LSTM model I am creating sequences from that dataset of experiements where at the beginning of the experiements the values are reset to 0 or some initializations.</p>
<h1><a name="example-of-dataset-1" class="anchor" href="#example-of-dataset-1"></a>Example of dataset</h1>
<pre><code class="lang-auto">index    experiement    temperature    ph     output
  1       exp-1            30          7       0
  2       exp-1            32          7       2
  3       exp-1            33          7       3
  4       exp-1            34          7       4
  5       exp-1            35          7       3
  6       exp-2            30          7       0
  7       exp-2            32          7       5
  8       exp-2            35          7       6
  9       exp-2            36          7       4
 10       exp-2            33          7       3
</code></pre>
<p>As the temporal data gets reset at the beginning of the experiment one of my sequence data which is used for training will look like this</p>
<pre><code class="lang-auto">index    experiement    temperature    ph     output
  1       exp-1            30          7       0
  2       exp-1            32          7       2
  3       exp-1            33          7       3
  4       exp-1            34          7       4
  5       exp-1            35          7       3
                                               
                                Label: 0
</code></pre>
<p>Basically, as the sequence does not know that a new experiment has started it will learn that the above values when trained should predict an output of 0 which is incorrect. If I train only for one experiment the training data will not be enough as I will not be able to produce many sequences from a single experiment.</p>
<p>Is there a certain approach to train such data? How do I feed multiple experiments data to my LSTM without the issues explained above?</p>
<p>Example of training them without considering start of new experiement results in the following output which is not accurate <a href="https://i.stack.imgur.com/YAHym.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/YAHym.png" alt="enter image description here" width="543" height="449"></a></p>
AttributeError: module 'torch' has no attribute 'vmap'

username: David_Wolfson
cooked: <p>I am running pytorch version 1.13 on a Google Cloud instance. When I try to use torch.vmap, I get the attribute error. I know this was an issue in older versions of pytorch, but not sure why it isn’t working with my version.</p>
<p>Best,<br>
David</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="David_Wolfson" data-post="1" data-topic="185496">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_wolfson/48/62820_2.png" class="avatar"> David_Wolfson:</div>
<blockquote>
<p>I know this was an issue in older versions of pytorch, but not sure why it isn’t working with my version.</p>
</blockquote>
</aside>
<p><code>torch.vmap</code> was added in <code>2.0</code> so you might need to use <code>functorch.vmap</code> in older releases or update PyTorch to the latest stable or nightly release.</p>
How to trace gradients through the network

username: cheeks
cooked: <p>Hi. I’m trying to use a HyperNetwork, i.e. a network that generates weights for another network. However, I’m finding that the HyperNetwork weights are not being updated when running the opt.step() . Is there a way to trace the gradient graph, back to the input, to make sure the HyperNetwork weights are on there? It would be great if there was a way to visualise that but just being able to trace the gradients would be extremely helpful.</p>
<p>i.e.<br>
a = f1(input)<br>
b = f2(a)<br>
c = f3(b)<br>
loss = MSE(c, truth)<br>
loss.backward()<br>
From here, I’d like to be able to trace gradients back to make sure parameters of f3 have grad graph edge to params of f2 which in turn have grad graph edge to params of f1.</p>

username: soulitzer
cooked: <p>To check if a input is the graph you can use <code>torch.autograd.grad(output, inputs=(input,))</code><br>
If the <code>input</code> is not part of the graph, then autograd would raise an error.</p>

username: cheeks
cooked: <p>Thanks for the reply. I’m afraid that doesn’t solve my problem. I need to be able to trace what is in the gradient graph of output. <code>torch.autograd.grad</code> would raise an error if an input was not part of the gradient but no way to see what is in the graph.</p>
No Squeeze-Excite block in EfficientNetV2 implementation of Fused-MBConv

username: anniepank
cooked: <p>Although original paper <a href="https://arxiv.org/pdf/2104.00298.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2104.00298.pdf</a> uses Squeeze and Excite block in FusedMbConv (see Figure 2 in the paper), torchvision’s implementation does not use them. What is the reason? And why is that not documented?</p>
RuntimeError: vector::_M_range_check

username: dma_science
cooked: <p>Hi all,<br>
I have been running into the following error with my torch model and have been struggling to debug it:</p>
<p>RuntimeError: The following operation failed in the TorchScript interpreter.<br>
Traceback of TorchScript (most recent call last):<br>
RuntimeError: The following operation failed in the TorchScript interpreter.<br>
Traceback of TorchScript (most recent call last):<br>
RuntimeError: vector::_M_range_check: __n (which is 18446744073709551615) &gt;= this-&gt;size() (which is 2)</p>
<p>The crux of the issue is that I have a decently complex torch.nn model class, that I have successfully been able to jit.script. The forward pass has a direct prediction from several nn submodules and then another prediction that is computed using autograd w.t.r. to the input. Everything works fine when I run my model on a cpu, but when I map it to cuda I get the M_range error.</p>
<p>Any help for teasing out why this error is happening would be greatly appreciated, or even some information on what causes this error.</p>