akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Confused output from MSE loss

Aldebaran: <p>When using the following tensors:</p>
<pre><code class="lang-python">prediction = torch.randn(3, 5)
# tensor([[ 1.0732,  1.0451,  0.6778,  1.9992,  0.2216],
#         [-0.1783,  0.7814,  0.3577, -0.8458, -0.1871],
#         [ 0.7581,  1.2827,  0.2930, -1.0155, -1.0184]])

target = torch.randn(3, 5)
# tensor([[-1.1133,  0.7325, -0.8611,  0.3197, -0.0146],
#        [-0.5205,  1.2527,  0.9034,  0.1704, -1.1623],
#        [-0.7035,  2.2432,  1.2100, -1.9930, -0.9532]])
</code></pre>
<p>to calculate the <a href="https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html" rel="noopener nofollow ugc">MSE loss</a> without reduction I getting as a result:</p>
<pre><code class="lang-python">mse_loss = torch.nn.MSELoss(reduction="none")
loss = mse_loss(input, target)
# tensor([[4.7804e+00, 9.7749e-02, 2.3682e+00, 2.8209e+00, 5.5801e-02],
#         [1.1709e-01, 2.2212e-01, 2.9780e-01, 1.0326e+00, 9.5105e-01],
#         [2.1364e+00, 9.2253e-01, 8.4085e-01, 9.5541e-01, 4.2508e-03]])
</code></pre>
<p>But in that case, shouldn’t it be a tensor with only three values? Since the the number of elements (N) in target and prediction is equal to 3?</p>

Dwight_Foster: <p>No when you set reduction to none MSELoss only finds the squared difference between each of the targets and predictions. It does not compute the mean so the shape is the same as the shape of the target and predictions.</p>

Aldebaran: <p>However, the <a href="https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html" rel="noopener nofollow ugc">documentation</a> states:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/071eafe4a8ea07eba419bbc9a3b53e57c4e3aa76.png" alt="image" data-base62-sha1="10Z5wmEqiyxBz2KLpz7nqF4luaq" width="399" height="52"></p>
<p>where N is the batch size and x and y are tensors of arbitrary shapes. Then, with N=3, L should be:<br>
<code>L={L1, L2, L3}</code></p>

ado_sar: <p>Yep, this is what I am understanding from the docs, which IMO are very confusing in that case. Did you find out what happens?</p>

ado_sar: <p>So the <code>reduction</code> is not performed over the batch dimension? Because, this is what the docs are stating.</p>

ptrblck: <p>I don’t fully understand which part of the documentation is confusing as no reduction is mentioned unless the <code>reduction</code> argument is used. E.g. for <code>"mean"</code> the loss is calculated as <code>mean(L)</code>.</p>
KL Divergence in PixelCnn

Vaibhav_Chahar: <p>Hi, I tried to use Kl Divergence Loss function in Gated PixelCNN but till now no success. in KL divergence It is required to have y_true and y_pred. I am sampling my input from the Neural network itself. The code for sampling is here (the output gives me a probability):<br>
‘’’<br>
I_Par=torch.zeros(shape,dtype=torch.float64,device=self.device)<br>
Start1=time.time()<br>
for h in range(shape[2]):<br>
for w in range(shape[3]):<br>
for c in range(shape[1]):<br>
with torch.no_grad():<br>
Net_1=(self.forward(I_Par[:,:,:h+1,:]))<span class="hashtag-raw">#forward</span> pass of the neural network<br>
rand=torch.rand((bs),dtype=torch.float64,device=self.device)<br>
I_Par[:,c,h,w]=torch.where(Net_1[:,c,h,w] &gt; rand[:],-1,1)<br>
‘’’</p>
<p>I thought after sampling, If i pass my final input again in the neural network and then I get both the y_true and y_pred but it is not correct. Please help  me with this, if you have any idea. I can add my code also.</p>
Pyqtorch - a minimalistic quantum state vector simulator

seitzdom: <p>To anyone interested in building/ training quantum neural networks using a intuitive torch-based syntax to solve complex tasks like partial differential equation solving: have a look at <a href="https://github.com/pasqal-io/pyqtorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pasqal-io/pyqtorch: PyTorch-based state vector simulator</a> (&lt; 1000 lines) and its docs <a href="https://pasqal-io.github.io/pyqtorch/" class="inline-onebox" rel="noopener nofollow ugc">Redirecting</a>. We would greatly appreciate it!</p>
How does Fork/Join work in pipeline.py

James_Zhang1: <p>Try to understand some implementation details in pipeline parallelism.</p>
<p>In pipeline.py, phony is used to create dependency  to ensure that batches[i-1] is executed after batches[i] in  # backpropagation by an explicit dependency.</p>
<p>Question 1). fork returned both detached tensor, how the dependency can be created in the join operation?<br>
Question 2). Both batch[I] and batch[I-1] are replaced by detached tensor, how the system can track the following graph for example, in following copy operation (_copy(batches[i], prev_stream, next_stream))</p>
<p>def _depend(fork_from: Batch, join_to: Batch) → None:<br>
fork_from_idx = fork_from.find_tensor_idx()<br>
join_to_idx = join_to.find_tensor_idx()</p>
<pre><code>fork_from[fork_from_idx], phony = fork(fork_from[fork_from_idx])
join_to[join_to_idx] = join(join_to[join_to_idx], phony)
</code></pre>

James_Zhang1: <p>I understand this is deprecated, but would like to understand the logic. Appreciate it!</p>
Error using fft: "RuntimeError: fft: ATen not compiled with MKL support"

neklein: <p>Hi, I am having an issue with using <code>fft</code>. I installed pytorch using <code>pip3 install torch torchvision</code> on Mac OSX 10.11.6 with Python 3.6. I could not find much about this error online so I apologize if it’s something simple and appreciate any pointers.</p>
<p>Here is a minimal example:</p>
<pre><code class="lang-auto">import torch

x = torch.randn(4, 3, 2)
torch.fft(x, 2)
</code></pre>
<p>leads to the error</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-11-ca08a48f699a&gt; in &lt;module&gt;()
      5 
      6 x = torch.randn(4, 3, 2)
----&gt; 7 torch.fft(x, 2)

RuntimeError: fft: ATen not compiled with MKL support
</code></pre>

albanD: <p>Hi,</p>
<p>The problem is that the pip package does not ship MKL with it and MKL is required at the moment for fft support on cpu. If you use gpu tensors, that should work.</p>
<p>If you do need it on cpu, you can either install using conda as the conda package ships with MKL. Otherwise, you will have to install MKL on your machine and then compile from source.</p>

neklein: <p>Thanks! I had tried installing MKL but missed the step about compiling from source.</p>

Yujing_Wang: <p>Hi,<br>
I have met the same question. I want to know how to compile from source. Thanks!</p>
Read Torch export Exir/ExportedProgram from C++

Antonio_Nevado: <p>Given a model that has been exported using the torch.export API from Python and serialized as a EXIR/ExportedProgram.</p>
<p>Is there a way of loading the serialized model for inspection/traversal using the C++ API in a similar way we can do it with TorchScript IR?</p>
<p>Note that I’m not referring to inference, I’m aware that it is possible to compile the model into a shared library with AOTInductor. I’m more interested on export a model with torch.export into EXIR/ExportedProgram and read the contents of the model (traverse the graph, inspect nodes…etc) from C++ API.</p>
<p>If it is not possible, are there any plans to add an C++ API for being able to do this?</p>
Using hook functions to call on a custom function to modify a layer given parameters of another layer

Mihai_M: <p>Hi all,</p>
<p>I’m slightly struggling with implementing a hook function within my SNN. I’m looking to call on a function at every epoch which modifies a layer’s voltages (in this example), <em>layer[3]</em> given another layer’s voltages and weights, <em>layer[0]</em></p>
<p>The custom function unfolds the tensors in an array and does some computation there. <em>layer[3]</em>'s dimension will also be changed.</p>
<p>I am looking to call the hook function to do something like the following:</p>
<blockquote>
<p>hook_handle = net.register_forward_hook(custom_function(net.layer[0], net.layer[3]))</p>
</blockquote>
<p>Looking for a little guidance here as I am quite new to hook functions.</p>

ptrblck: <p>I don’t fully understand why forward hooks are used, but I assume you are using the forward activation for the actual manipulation?<br>
It also seems you want to directly manipulate the <code>weight</code> attribute of a layer based on this computation?<br>
Could you give a small example of the actual manipulation and when the parameter should be changed?</p>
Sinusoidal encoding and gradient flow

0x97: <p>In models requiring sinusoidal encoding, likewise the following, taken from <a href="https://github.com/openai/guided-diffusion" rel="noopener nofollow ugc">github.com/openai/guided-diffusion</a>:</p>
<pre><code class="lang-auto">def timestep_embedding(timesteps, dim, max_period=10000):
    """
    Create sinusoidal timestep embeddings.

    :param timesteps: a 1-D Tensor of N indices, one per batch element.
                      These may be fractional.
    :param dim: the dimension of the output.
    :param max_period: controls the minimum frequency of the embeddings.
    :return: an [N x dim] Tensor of positional embeddings.
    """
    half = dim // 2
    freqs = th.exp(
        -math.log(max_period) * th.arange(start=0, end=half, dtype=th.float32) / half
    ).to(device=timesteps.device)
    args = timesteps[:, None].float() * freqs[None]
    embedding = th.cat([th.cos(args), th.sin(args)], dim=-1)
    if dim % 2:
        embedding = th.cat([embedding, th.zeros_like(embedding[:, :1])], dim=-1)
    return embedding
</code></pre>
<p>I was wondering if, when passing the encoding to the interested modules, a <code>.detach()</code> should be performed, since in my (probably wrong) view, the actual new input should be the encoded one.</p>
Getting error while generating pte for WAV2VEC2_ASR_BASE_960H

mi_gupta: <p><code>    raise AssertionError( AssertionError: original output #2 is None, but only the following types are supported: (&lt;class 'torch.Tensor'&gt;, &lt;class 'torch.SymInt'&gt;, &lt;class 'torch.SymFloat'&gt;, &lt;class 'torch.SymBool'&gt;)</code></p>
<p>can anybody suggest the input size and model class code to be put in the example directory?<br>
current example input is torch.randn(input_shape) i.e.<br>
(self.batch_size,1, self.input_frames)</p>
Cnn in deep q learning

Mamora: <p>I have a network like this:</p>
<p>c, h, w = input_dim</p>
<p>self.online = nn.Sequential(<br>
nn.Conv2d(in_channels=c, out_channels=32, kernel_size=8, stride=4), nn.ReLU(), nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2), nn.ReLU(), nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1), nn.ReLU(), nn.Flatten(), nn.Linear(35<em>27</em>64,512), nn.ReLU(), nn.Linear(512, output_dim) )</p>
<p>my input shape is (3,250,315) and my actual inputs are images, which I use transforms.ToTensor before passing to neural net.</p>
<p>but when I run it, I get this error:</p>
<p>RuntimeError: Given groups=1, weight of size [32, 315, 8, 8], expected input[8, 3, 250, 315] to have 315 channels, but got 3 channels instead.</p>
<p>my actual inputs are images, which I use transforms.ToTensor before passing to neural net.</p>

ptrblck: <p>Based on the error message the first <code>nn.Conv2d</code> layer raises the error since <code>c=315</code> is set. Since you are using 3 input channels, set <code>in_channels=3</code> and it should work. If you are still running into this issue, your images might be in channels-last memory format. In this case you would need to <code>permute</code> them to the expected channels-first layout.</p>

Mamora: <aside class="quote no-group" data-username="Mamora" data-post="1" data-topic="200309">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/a183cd/48.png" class="avatar"> Mamora:</div>
<blockquote>
<p>I use transforms.ToTensor</p>
</blockquote>
</aside>
<p>Thank you. Indeed the problem was c. I think the main problem was using torchvision.transforms.ToTensor  which already changes the shape of the input in the order CHW, so I did not need to do it myself, but I did.</p>
Getting different loss values with gradient accumulation but same effective batch size

KimberleyJensen: <p>I did two experiments one with batch size 5 with accumulation_steps = 2 and another with batch size 10 with accumulation_steps = 1, no other code changes and I expected to see similar loss values on wandb for both experiments but they weren’t. I used the same seed, learning rate (no lr schedule), no dropout, set deterministic to true and data shuffling off, also my model uses instance norm and not batch norm. Here is the code if anyone can help please</p>
<pre><code class="lang-auto">optimizer = Adam(model.parameters(), lr=config.training.lr)
scaler = GradScaler()    
counter = 0  
total_loss = 0.0
accumulation_steps = 2
optimizer.zero_grad()
for batch in tqdm(train_loader):   

    y = batch.to(device)
    x = y.sum(1)  
    if config.training.target_instrument is not None:
        i = config.training.instruments.index(config.training.target_instrument)
        y = y[:,i]
    with torch.cuda.amp.autocast():        
        y_ = model(x)   
        loss = nn.MSELoss()(y_, y) 

    scaler.scale(loss).backward()
    if (counter + 1) % accumulation_steps == 0:  
        scaler.step(optimizer)
        scaler.update()
        optimizer.zero_grad(set_to_none=True)
    total_loss += loss.item()
    counter += 1
    if counter % 100 == 0:
      average_loss = total_loss / 100
      wandb.log({"loss_100_steps": average_loss})
      total_loss = 0.0
</code></pre>

ptrblck: <p>Based on your code you are not dividing the loss (or gradients) by the accumulation steps and are thus directly accumulating the gradients. Disable <code>amp</code> for a quick test and print the <code>.grad</code> attributes of some parameters. In the <code>accumulation_step = 2</code> case the gradients should have a larger magnitude, which could explain the difference in your loss values.</p>

KimberleyJensen: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Do you know what code i can add/edit to fix this problem please?<br>
I tried adding loss = loss / accumulation_steps but the loss values are still quite different</p>

ptrblck: <p>Yes, scaling the <code>loss</code> should work. If you get stuck, could you post a minimal and executable code snippet reproducing the unexpected divergence?</p>

cmm: <p>Hi,<br>
were you able to find a solution? I am also facing the issue even if loss is normalized</p>
Remove Computation in Forward & Backward Pass

livingshade: <p>Hi, I am trying to do some hack to pytorch to see if an research idea works.</p>
<p>Specificly,  I want to remove the computation(i.e. cuda kernel launch) in both forward and backward pass, without breaking the training script interface. Ideally, after remove the computation kernels, I can still use <code>pytorch train.py</code> to do a training loop and see how much time it takes. (though the training itself no longer make senses here since no meaningful activation and gradient is computed)</p>
<p>I check the document and it seems that there is no obvious way to do that. From my understanding: In the forward pass, I need to keep the memory allocation of those <code>saved tensor</code> as weel as the computation graph. In the backward pass, I need to keep the memory allocation for the gradient.</p>
<p>I believe that modifying <code>aTen</code> and <code>autograd</code> library might work, but it will be very difficult. I wonder if there is any easier solution. For example, can I change the <code>grad_fn</code> after forward to make it a dummy that allocs and returns <code>torch.zeros()</code> ?</p>
<p>Thanks in advance!</p>
Seeking Efficient Data Transformation for Complex Number Phase Prediction in PyTorch

nazalli: <p>Greetings,<br>
I come seeking guidance on a perplexing issue regarding data transformation for efficiently predicting the phase of complex numbers using PyTorch. Despite extensive experimentation, I have yet to find a transformation method that yields satisfactory results.</p>
<p>I have explored various techniques including Cox-Box, Yeo-Johnson, and quantile transformations, but unfortunately, none of them have provided the desired outcome for predicting the phase of complex numbers accurately.</p>
<p>In my pursuit of a solution, I devised a custom transformation function as follows:</p>
<pre><code class="lang-auto">def transform_y(y, a, b):
    return np.log(y**a + 1)**b

def inverse_transform(y_transformed, a, b):
    return (np.exp(y_transformed**(1/b)) - 1)**(1/a)
</code></pre>
<p>I then fine-tuned the parameters <code>a</code> and <code>b</code> to minimize the Root Mean Squared Error (RMSE) of the transformed data. Surprisingly, this transformation appears to perform well in predicting the transformed data, with an RMSE of approximately 10e-4.</p>
<p>However, the issue arises when applying the inverse transformation. Despite the promising results obtained during prediction, the application of the inverse transformation results in an infinite RMSE, indicating a severe discrepancy.</p>
<p>I am reaching out to the PyTorch community for insights, suggestions, or alternative approaches to tackle this challenge. Any advice or recommendations on more effective data transformation techniques or adjustments to the existing transformation method would be greatly appreciated.</p>
<p>Thank you in advance for your assistance and expertise.</p>
Help building and storing very large datasets?

Goldname: <p>Given an embedding model, I want to embed a lot of text and store the embeddings for later downstream tasks. What’s the best way to do this? My embedding model is quite large and the dataset is quite large so they can neither fit into cpu nor gpu memory.</p>
<p>How can I create a pytorch dataset, save it to disk, then continuously append chunks of torch.tensors to it without needing to load the dataset back into memory?</p>
<p>I also don’t want to have hundreds of .pt files. I just want a single large pytorch dataset. Embedding size of 4096 x 4 bytes (float32) x 10000 items so I expect the dataset to be about 200 GB in size.</p>

Goldname: <p>Furthermore, after creating this dataset I’d need a good way to read out chunks from it. Could anyone point me to any ways of doing this elegantly?</p>

Goldname: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  do you have any advice?</p>
How to Set QNNPACK as Quantization Backend in C++ for PyTorch on Raspberry Pi 4?

edoabrgel: <p>I’m transitioning from Python to C++ for deploying a quantized model on an rpi4 device and aiming to achieve maximum inference speed. In Python, setting <code>torch.backends.quantized.engine = 'qnnpack'</code> significantly improves my model’s FPS performance, it boosts from ~4 FPS to ~45 FPS.</p>
<p>How can I achieve a similar backend switch to QNNPACK in C++?<br>
Is there any C++ equivalent to:<br>
`torch.backends.quantized.engine = ‘qnnpack’?</p>
<p>Thank you in advance,<br>
Ido</p>

edoabrgel: <p>Answer:</p>
<p><span class="hashtag-raw">#include</span> &lt;ATen/ATen.h&gt;<br>
<span class="hashtag-raw">#include</span> &lt;torch/torch.h&gt;</p>
<p>int main() {<br>
// Set the quantization engine to QNNPACK<br>
at::globalContext().setQEngine(at::QEngine::QNNPACK);<br>
return 0;<br>
}</p>
How to use Opacus with TemporalGNN e.g. A3TGCN2?

simon2: <p>Hello everyone,</p>
<p>Can you help me to train a temporal GNN with Opacus?<br>
I try to set up Differential Private learning on a Graph Temporal Neural Network, using opacus for the differential privacy and A3TGCN2 from the pytorch-gemoemtric-temporal library for the TemporalGNN part. In particular I try to combine the guides for <a href="https://opacus.ai/tutorials/building_image_classifier" rel="noopener nofollow ugc">opacus</a> and for <a href="https://github.com/benedekrozemberczki/pytorch_geometric_temporal/blob/master/notebooks/a3tgcn_for_traffic_forecasting.ipynb" rel="noopener nofollow ugc">a3tgcn</a> training.</p>
<p>To check if the resulting Model is opacus compliant, i test it with<br>
the opacus ModuleValidator and got no errors. So I assume the model is compliant.</p>
<p>Unfortunately I still run into errors. Could you please help me understand where my errors come from, how I can fix them and optimally provide a minimal working example?</p>
<p>Here are the relevant parts of my code:</p>
<p>Using the privacy engine:</p>
<pre><code class="lang-auto">from opacus import PrivacyEngine
secure_mode = False privacy_engine = PrivacyEngine(secure_mode=secure_mode)  
# Privacy engine hyper-parameters 
MAX_GRAD_NORM = 1.5 
DELTA = 1e-5 
EPSILON = 50.0 
EPOCHS = 20  priv_model, priv_optimizer, priv_train_loader =
privacy_engine.make_private_with_epsilon(
    module=model,     
    optimizer=optimizer,     
    data_loader=train_loader,     
    max_grad_norm=MAX_GRAD_NORM,     
    target_delta=DELTA,     
    target_epsilon=EPSILON,     
    epochs=EPOCHS )
</code></pre>
<p>Attempting to train the Model:</p>
<pre><code class="lang-auto">model.train()

snapshot = next(iter(train_set))
edge_index = snapshot.edge_index.to(device)

for epoch in tqdm(range(EPOCHS)): 
    loss = 0
    loss_list = []
    for encoder_inputs, labels in priv_train_loader:
    
        y_hat = priv_model(encoder_inputs, edge_index)
        loss = criterion(y_hat, labels)
        loss.backward()
        priv_optimizer.step()
        priv_optimizer.zero_grad()
</code></pre>
<p>The error message:</p>
<pre><code class="lang-auto">ValueError: Poisson sampling is not compatible with grad accumulation. You need to call optimizer.step() after every forward/backward pass or consider using BatchMemoryManager
</code></pre>
<p>Attempt with BatchMemoryManager</p>
<pre><code class="lang-auto">for epoch in tqdm(range(EPOCHS)): 
loss = 0
loss_list = []
with BatchMemoryManager( data_loader=priv_train_loader, max_physical_batch_size=2, optimizer=optimizer) as new_data_loader:
    for encoder_inputs, labels in new_data_loader:
        y_hat = priv_model(encoder_inputs, edge_index)
        loss = criterion(y_hat, labels)
        loss.backward()
        priv_optimizer.step()
        priv_optimizer.zero_grad()
</code></pre>
<p>The second error message:</p>
<pre><code class="lang-auto">AttributeError: 'SGD' object has no attribute 'signal_skip_step'
</code></pre>
Why is CUDA running out memory for Llama 2 inference?

Goldname: <p>I am using a GPU of 48 GB memory and Llama 2 7b. The model easily fits into gpu memory, but when I perform inference with a long sequence length of 8k-10k tokens I run out of memory. Does anyone know why?</p>
<p>Sequence length of 10k tokens should only be about 10k x 10k x 4 400MB memory usage since transformer memory is O(n^2). Since I am only doing inference previous activations can be discarded.</p>
<p>My code looks something like this:</p>
<pre><code class="lang-auto">with torch.no_grad():
    llama_name = "meta-llama/Llama-2-7b-chat-hf"
    llama = LlamaModel.from_pretrained(llama_name).to(device)
    print('loaded model')
    for batch_idx, (times,texts) in enumerate(train_loader):
        texts = texts.to(device)
        logits = llama(texts)
</code></pre>

Goldname: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Do you have any advice please?</p>

ptrblck: <p>I don’t believe a memory increase of only 400MB is expected as all intermediate activations would also increase in their size.</p>

Goldname: <p>I understand that’s probably the case for training, but during inference there’s no need to keep the past activations while the input passes through the layers. So is that optimization not implemented in pytorch?</p>

ptrblck: <p>Yes, intermediates won’t be kept, but still computed and passed to the next layer. The peak memory would thus be smaller than during training but would still increase more than the input size delta.</p>

Goldname: <p>Why would the intermediates be larger in size than the input? The dimension and sequence length usually remains the same when passing through transformer layers.</p>

Youness_EL_BRAG: <p>Hey Could u please check your inbox i sent message if you have time to look at it</p>

insightbuilder: <p>Before we dive into debugging, can you share the memory that is already taken up in your Nvidia Card?</p>
<p>Run <code>watch -n 1 nvidia-smi</code> on your terminal and check the memory used just by loading the model. Without the model, ~14MB of VRam is usually consumed.</p>
<p>Then run the inference and see how the memory is increasing on the terminal. This must give you some idea as to where the issue is.</p>

Goldname: <p>Yeah I’ve been doing that. Loading the model and optimizer takes up around 20 gb usually. Then during inference i get oom.</p>

insightbuilder: <p>20GB of VRAM… that means you are left with another 27GB atleast.</p>
<p>I am just trying to find the root cause…</p>
<ul>
<li>
<p>Could you please share the full code, so it can reviewed. (I must have asked it earlier… my bad)</p>
</li>
<li>
<p>Check if lesser number of tokens like 1k or 2k is getting executed. Measure the memory usage with them.</p>
</li>
<li>
<p>Another point is can you tell what is the batch_size that you are using in the train_loader? Is it 1/ 4/ 8 or 16?</p>
</li>
</ul>
<p>Lets then try to dive into the code mechanics.</p>
<p>I play with 12GB Nvidia RTX 4070, and face a lot of OOM errors. Used the above tactics to get the bigger models and token counts to execute. (working with lesser resources does teach a lesson or two)</p>
Inner workings of torch.sum function

John_Selmys: <p>Can anyone tell me the algorithm used by the sum() function. I discovered that sum() in numpy uses the pairwise algorithm. I don’t think torch uses it or the Kahan algorithm but maybe I’m wrong. Does anyone know for sure?</p>

John_Selmys: <p>The reason I’m asking is because I want to write the torch.sum() function in C. I have tried many summation algorithms that are intended to reduce the precision errors that occur when using floating point numbers: kahan (variations), pairwise, sortfirst, recursivesum, etc. None of my C programs gave the same results as torch.sum().<br>
Should anyone want to try, here is the data set I’m using:<br>
[ 7.8045e-05, -2.6427e-05,  1.2772e-05,  9.2774e-05,  4.3943e-05,<br>
1.5254e-04, -1.2603e-05, -2.4027e-04,  9.0581e-06,  7.7683e-06,<br>
-5.3032e-06,  1.3365e-04,  9.2983e-05,  8.4483e-05,  8.1089e-05,<br>
9.9249e-05, -1.6948e-04,  3.3411e-06, -2.6907e-05,  1.1957e-04,<br>
4.8341e-05, -2.1752e-04,  3.5117e-05,  2.0250e-04, -6.8962e-05,<br>
-2.2960e-05, -1.5692e-04, -1.4009e-04,  1.3505e-05, -8.7326e-05,<br>
-5.3032e-06, -1.3066e-04]</p>
<p>I will appreciate any ideas.</p>
Video classification dataloader

pyt: <p>Hi all. Does anyone have experience in classifying videos using deep learning with pytorch? I’m having a bottleneck in reading videos with the dataloader. The videos are stored in mp4 format and I use the OpenCV library.<br>
I have tried to increase the batch size but it doesn’t improve the speed, it even seems to slow down.<br>
Can anyone tell me how to improve the reading speed?<br>
Here is the part of the code where I read the videos.</p>
<pre data-code-wrap="python"><code class="lang-python">frames = []
v_cap = cv2.VideoCapture(filename)
start_frame_number = frame_list[0]
last_frame_number = frame_list[-1]

v_cap.set(cv2.CAP_PROP_POS_FRAMES, start_frame_number)

for fn in range(start_frame_number, last_frame_number + 1):
    success, frame = v_cap.read()
    if fn in frame_list:
        frame = Image.fromarray(frame)
        img_tens = transform(frame)
        frames.append(img_tens)

v_cap.release()</code></pre>

ptrblck: <p>You could give <a href="https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html">DALI</a> a try and check if e.g. decoding the video on your GPU would speed up the training.</p>

pyt: <p>Thanks for your response.<br>
Another problem I am having is that I have a significant speed difference (three times) in the execution of the code on different PCs. On the computer with the more powerful GPU I have less speed in reading the video. In both I am working with the same versions of Pytorch and OpenCV. What can these differences be due to?</p>
erro:Expected input batch_size (16) to match target batch_size (32)

pak_key: <p>from tqdm.auto import tqdm</p>
<h1><a name="h-1" class="anchor" href="#h-1"></a>随机种子</h1>
<p>torch.manual_seed(42)</p>
<h1><a name="h-2" class="anchor" href="#h-2"></a>循环次数</h1>
<p>epoch = 5</p>
<p>for epoch in tqdm(range(epoch)):<br>
print(f’第{epoch}次…')</p>
<pre><code># 统计训练损失
train_loss = 0

for _,(X,y) in enumerate(train_data_lodael):
   
   # 开始训练
    mode_l.train()
    y_pred = mode_l(X)
     
    # 计算损失
    loss = loss_fn(y_pred,y)
    train_loss += loss
    
    # 优化
    optimizer.zero_grad()
    
    # 后传
    loss.backward()
    
    # 优化stop
    optimizer.step()
    
    if _ % 400 == 0:
        print(f'------------------')

train_loss /= len(train_data_lodael)

# 测试
test_loss,test_acc =0,0
mode_l.eval()
with torch.inference_mode():
    for X_test,y_test in test_data_lodael:
        # 前传
        test_y_pred = mode_l(X_test)
        
        # test
        test_loss += loss_fn(y_pred,y_test)
        
        test_acc += accuracy_fn(y_true=y_test,y_pred=test_y_pred)
        
    # 统计
    test_loss /=len (test_data_lodael)
    
    test_acc /= len(test_data_lodael)
    
print(f'\n训练损失:{train_loss:.4f}| 测试损失:{test_loss:.2f}__测试准确度:{test_acc}')
</code></pre>

ptrblck: <p>Based on the error message in the title the loss calculation fails due to a shape mismatch.<br>
Could you post your model definition so we could check if you are reshaping the inputs in an invalid way?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>
Valid linting when using register_buffer/register_parameter?

bean_the_great: <p>Hi!</p>
<p>I am using register_buffer and register_parameter to initialise a parameter for a pytorch model. However, when doing so, the attribute that I have set via either function does not get picked up as a class attribute by the python linter. Has anyone managed to find a way to address this, please?<br>
Thanks</p>
Speed Optimize of a customized layer that utlizes "torch.meshgrid" and "for loop"

Darryl_S: <p>Hi all, I am implementing a customized layer and found out it runs super-slow. I guess it might be due to the way that I used torch.meshgrid" and “for loop” inside the forward which cannot benefit from GPU.</p>
<p>Can anyone help and guide how to further optimize the code to speed up? Thanks.</p>
<p>Here is the layer:</p>
<pre><code class="lang-auto">
def e2e_abs_max_diff(a, b):

    ##calculate max of the element wise abs difference between two 1D tensor with all pair combinations 
    ## same 1xn for a and b

    a_c, b_c = torch.meshgrid(a.view(-1), b.view(-1))
    # print(a_c.size(), b_c.size())
    v = torch.sub(a_c, b_c)
    # print(v)
    result = torch.max(torch.abs(v))

    return result


class MaxDiffMap(nn.Module):
    
    #input: X, Y 4 D tensors in (N,C,D,H,W)
    #output: result_windows in (N,C,D,H', W')
    #param: win_a, win_b, stride_a, stride_b
    #property:
    # Implementation of maxpooled difference map
  

    def __init__(self, win_a, win_b, stride_a, stride_b):
        super().__init__()
        self.win_a = win_a
        self.win_b = win_b
        self.stride_a = stride_a
        self.stride_b = stride_b

    def forward(self, X, Y):
        ## check the input size of X and Y
        ## assume X Y same size in (N,C,D,H,W)

        XX = X.unfold(3, self.win_a, self.stride_a).unfold(4, self.win_b, self.stride_b)
        XX = XX.contiguous().view(*XX.size()[:-2], -1)
        result_shape = XX.size()

        YY = Y.unfold(3, self.win_a, self.stride_a).unfold(4, self.win_b, self.stride_b)
        YY = YY.contiguous().view(*YY.size()[:-2], -1)

        XX = XX.view(*XX.shape[:5], -1)
        YY = YY.view(*YY.shape[:5], -1)

        XX = XX.view(-1, self.win_a * self.win_b)
        YY = YY.view(-1, self.win_a * self.win_b)



        ## Note: type conversion of tensors.
        result_windows = torch.zeros(XX.shape[0:-1]).type_as(X)
        result_windows = result_windows.view(-1, 1)

        #print(result_windows.size())

        for i in range(0, XX.shape[0]):
            # print(i)
            result_windows[i] = e2e_abs_max_diff(XX.view(-1, self.win_a * self.win_b)[i, :],
                                                 YY.view(-1, self.win_a * self.win_b)[i, :])

        result_windows = result_windows.view(result_shape[0:-1])

        return result_windows
</code></pre>

ptrblck: <p>Could you add the missing pieces of code to make the current code snippet executable, please? This would allow us to check if e.g. <code>torch.compile</code> could speed up your code.</p>
Weighted Multi-label Focal Loss Implementation

Shix0: <p>I want to confirm the below implementation for a Multi-label <a href="https://arxiv.org/abs/1708.02002v2" rel="noopener nofollow ugc">Focal Loss</a> function that also accepts the class_weights parameter to handle class imbalance (<a class="mention" href="/u/ptrblck">@ptrblck</a>  would like to get your feedback if possible <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> ):</p>
<blockquote>
<p>class MultiLabelFocalLoss(torch.nn.Module):<br>
“”"Implementation of a Multi-label Focal loss function</p>
<pre><code>    Args:
        weight: class weight vector to be used in case of class imbalance
        gamma: hyper-parameter for the focal loss scaling.
"""
def __init__(self, weight=None, gamma=2):
    super(MultiLabelFocalLoss, self).__init__()
    self.gamma = gamma
    self.weight = weight #weight parameter will act as the alpha parameter to balance class weights
    self.loss = torch.nn.BCELoss(reduction='none')

def forward(self, outputs, targets):
    ce_loss = self.loss(outputs, targets) * self.weight 
    pt = torch.exp(-ce_loss)
    focal_loss = ((1-pt)**self.gamma * ce_loss).mean() # mean over the batch
    return focal_loss
</code></pre>
</blockquote>
Cross Entropy for batch data

tungyen: <p>Hello, I am now working on implementing the PointNet Model for 3D point cloud Segmentation. However, I encountered some problems in loss function. Here is my data shape for each batch:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6a43953fe88ec4543261265939933492175bd7b.png" data-download-href="https://discuss.pytorch.org/uploads/default/c6a43953fe88ec4543261265939933492175bd7b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6a43953fe88ec4543261265939933492175bd7b.png" alt="image" data-base62-sha1="slgo0PznanAAXFvgrhyaMaO3G4j" width="690" height="75" data-dominant-color="252424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">735×80 6.17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
where 4 is batch size, and 1500 is the point number for each point cloud, and 4 in the ouput is class number.</p>
<p>However, when I put them into the nn.CrossEntropyLoss() function, I got the error below:</p>
<pre><code class="lang-auto">RuntimeError: Expected target size [4, 4], got [4, 1500]
</code></pre>
<p>I remember that if I don’t set up the argument “reduction”, then the default should be “mean”, which will handle the batch for me. Then I am very confused where is the problem. Thank you guys.</p>

ptrblck: <p>Your <code>output</code> has an invalid shape and should be <code>[batch_size, nb_classes, additional_dim]</code>, so <code>[4, 4, 1500]</code> in your case. You can <code>permute</code> the tensor to swap dim1 and dim2.</p>

tungyen: <p>Thank you very much. It solve my problem!</p>
Torch.tensor(): CUDA-capable device(s) is/are busy or unavailable

Christian_Stano: <p>Our torch.tensor() calls are failing with the below error:<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Our driver setup / printout is the below:<br>
NVIDIA-SMI 535.161.07             Driver Version: 535.161.07   CUDA Version: 12.2<br>
__Python VERSION: 3.10.11 (main, Nov 30 2023, 18:20:49) [GCC 7.5.0]<br>
__pyTorch VERSION: 2.1.0+cu121<br>
__CUDA VERSION 12.1<br>
__CUDNN VERSION: 8902<br>
__Is CUDA available: True<br>
__Number CUDA Devices: 1<br>
Active CUDA Device: GPU 0<br>
Available devices 1<br>
Current cuda device 0<br>
GPU count: 1</p>
<p>Trying to understand if there’s a weird version compatibility here causing this issue- in our container image we specify CUDA 12.3 &amp; Torch 2.2.1, so it looks like there’s an override happening somewhere</p>

ptrblck: <aside class="quote no-group" data-username="Christian_Stano" data-post="1" data-topic="200324">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/christian_stano/48/63089_2.png" class="avatar"> Christian_Stano:</div>
<blockquote>
<p>in our container image we specify CUDA 12.3 &amp; Torch 2.2.1, so it looks like there’s an override happening somewhere</p>
</blockquote>
</aside>
<p>The PyTorch binaries ship with their own CUDA runtime dependencies and your locally installed CUDA toolkit will be used of you build PyTorch from source or a custom CUDA extension.<br>
Make sure your container can run any CUDA application, as the error might point to your setup.</p>