akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to understand thread_local of capture_error_mode in cuda graph capture

yjguo: <p>at <a href="https://pytorch.org/docs/stable/generated/torch.cuda.graph.html" class="inline-onebox" rel="noopener nofollow ugc">graph — PyTorch 2.3 documentation</a>, it says that:</p>
<ul>
<li><strong>capture_error_mode</strong> (<a href="https://docs.python.org/3/library/stdtypes.html#str" rel="noopener nofollow ugc"><em>str</em></a><em>,</em> <em>optional</em>) – specifies the cudaStreamCaptureMode for the graph capture stream. Can be “global”, “thread_local” or “relaxed”. During cuda graph capture, some actions, such as cudaMalloc, may be unsafe. “global” will error on actions in other threads, “thread_local” will only error for actions in the current thread, and “relaxed” will not error on actions.</li>
</ul>
<p>at <a href="https://github.com/pytorch/pytorch/blob/main/test/test_cuda.py#L3537" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/test_cuda.py at main · pytorch/pytorch · GitHub</a> for the test test_cuda_graph_error_options, there are two threads:<br>
the main thread: do the cuda graph capture, and there’s memory allocation within the capture workload.<br>
the second thread: this thread is created within graph capture, and there’s also memory allocation in the thread function body.</p>
<p>Within the test code,</p>
<ol>
<li>
<p>for capture_error_mode “relaxed”:<br>
it is expected that we are able to capture the graph successfully, because “relaxed” will not error on actions.</p>
</li>
<li>
<p>for capture_error_mode “global”:<br>
it is expected that we are unable to allocate the memory in the second thread (and so the capture is not successful), because “global” will error on actions in other threads.</p>
</li>
<li>
<p>for capture_error_mode “thread_local”:<br>
my expectation is that the main thread is error (and so the capture is not successful), because “thread_local” will only error for actions in the current thread. But the test shows that the actual result is that  we are able to capture the graph successfully, why? thanks.</p>
</li>
</ol>
Cuda.graph can not capture self written cuda kernel

Eutenacity: <p>import torch</p>
<p>from torch.utils.cpp_extension import load</p>
<p>wait_test = load(name=‘wait_flag’, sources=[‘cuda_flag_wait.cpp’, ‘cuda_flag_wait.cu’])<br>
a_s = [torch.tensor([False]*128,dtype=torch.bool).cuda() for _ in range(20)]<br>
w1 = torch.randn(4096,14336,dtype=torch.bfloat16).cuda()<br>
x1 = torch.randn(1,4096,dtype=torch.bfloat16).cuda()</p>
<p>g = torch.cuda.CUDAGraph()<br>
j=0<br>
<span class="hashtag-raw">#warmup</span><br>
for i in range(1):</p>
<pre><code>torch.matmul(x1,w1)
wait_test.wait_flag(2000000,a_s[j])


torch.matmul(x1,w1)


# torch.cuda.synchronize()
</code></pre>
<p>j=0<br>
with torch.cuda.graph(g):<br>
torch.matmul(x1,w1)</p>
<pre><code>wait_test.wait_flag(2000000,a_s[j])

torch.matmul(x1,w1)
</code></pre>
<p>g.replay()</p>
<p>###########</p>
<p><span class="hashtag-raw">#include</span> &lt;torch/extension.h&gt;<br>
<span class="hashtag-raw">#include</span> &lt;torch/torch.h&gt;<br>
<span class="hashtag-raw">#include</span> &lt;ATen/cuda/CUDAContext.h&gt;<br>
<span class="hashtag-raw">#include</span> &lt;ATen/cuda/CUDAEvent.h&gt;<br>
static at::cuda::CUDAStream captureStream = at::cuda::getStreamFromPool();<br>
static at::cuda::CUDAEvent cuda_events;<br>
void wait_test( int clock_count,bool * flag);//, cudaStream_t a<br>
void set_flag(bool* flag);</p>
<p>void wait_flag(int clock_count,torch::Tensor device_modules)<br>
{<br>
wait_test(clock_count,device_modules.data_ptr());<br>
// cuda_events.record(captureStream);<br>
}</p>
<p>// Binding<br>
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)<br>
{<br>
m.def(“wait_flag”, &amp;wait_flag,  “wait”);<br>
}<br>
#########<br>
<span class="hashtag-raw">#include</span> “cuda_runtime.h”<br>
<span class="hashtag-raw">#include</span> “device_launch_parameters.h”<br>
<span class="hashtag-raw">#include</span> &lt;cub/cub.cuh&gt;<br>
<span class="hashtag-raw">#include</span> &lt;thrust/device_vector.h&gt;<br>
<span class="hashtag-raw">#include</span> &lt;thrust/sort.h&gt;<br>
<span class="hashtag-raw">#include</span> &lt;thrust/unique.h&gt;<br>
<span class="hashtag-raw">#include</span> &lt;string.h&gt;</p>
<p><strong>global</strong> void clock_block(clock_t clock_count, bool* flag)<br>
{<br>
int idx = blockIdx.x * blockDim.x + threadIdx.x;<br>
clock_t start_clock = clock();<br>
clock_t clock_offset = 0;<br>
// clock_t clock_count_tmp = 1000000;<br>
//<br>
int count = 0;<br>
bool state_v = flag[0];<br>
while ( (clock_offset &lt; clock_count)&amp;&amp;(!state_v)  )<br>
{</p>
<pre><code>    clock_offset = clock() - start_clock;
   
    state_v = flag[count];
    count+=1;
    count = count % 128;
}
</code></pre>
<p>}</p>
<p>void wait_test(int clock_count, bool* flag )<br>
{</p>
<pre><code>int m = 1;
int n = 1;
int block1D = 1;
dim3 block(block1D, block1D);
dim3 grid(m/block1D, n/block1D);


//execute the kernel

clock_block&lt;&lt;&lt; grid, block&gt;&gt;&gt;(clock_count,flag);
</code></pre>
<p>}</p>
Doubt in the usage of t5 model

vamsi_balijepally: <pre><code class="lang-auto">modelSeq2Seq = AutoModelForSeq2SeqLM.from_pretrained(model_name) 
model = AutoModel.from_pretrained(model_name)
</code></pre>
<p>so this modelseq2seq model has an extra Fully connected layer 756 to NUM tokens (lm_head) but the plain model does not output that why?<br>
Even while in the pre-training phase there has to be a layer to convert the embeds to logits right ?</p>

vamsi_balijepally: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> i hope you can help ?</p>

ptrblck: <p>I’m not an expert in HF Models but from the docs:</p>
<blockquote>
<p>One can use T5ForConditionalGeneration (or the Tensorflow/Flax variant), which includes the language modeling head on top of the decoder.</p>
</blockquote>
<p>it seems this model abstraction would contain the <code>lm_head</code>. You might also want to cross-post this question in the HF discuss board as the devs would know exactly how these models are defined.</p>

vamsi_balijepally: <p>hey <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the response …<br>
i will post in the HF discussion board</p>
FSDP model can't save checkpoint

JulioZhao97: <p>I failed to save model when using torch.FSDP, what I tried to do is something like:</p>
<pre><code class="lang-auto">model = unwrap(self.model)
state_dict = model.state_dict()
</code></pre>
<p>But the process hangs forever, I have tried something like enable offload_to_cpu=True in <a href="https://discuss.pytorch.org/t/fsdp-failed-to-save-model-checkpoints/178232">this</a> and <a href="https://github.com/pytorch/pytorch/issues/98823" rel="noopener nofollow ugc">this</a> but nothing works.</p>
<p>I am using torch=2.1.0+cu117</p>

George_feng: <p>I ran into same issue. Have you solved it yet?</p>
Remove softmax layer from x3d_l model

talhaanwarch: <p>I want to remove last softmax layer from pytorch x3d_l model.<br>
This is what I tried</p>
<pre><code class="lang-auto">model = torch.hub.load('facebookresearch/pytorchvideo', 'x3d_l', pretrained=True)
model.blocks[5]=nn.Sequential(*list(model.blocks[5].children())[:-2])
</code></pre>
<p>Shape of my input is <code>torch.Size([2, 3, 16, 320, 320])</code><br>
When I pass it to the model, I got this error</p>
<pre><code class="lang-auto">RuntimeError: mat1 and mat2 shapes cannot be multiplied (4096x1 and 2048x400)
</code></pre>
<p>This last block look like this<br>
<a href="https://i.stack.imgur.com/9o9lJ.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/9o9lJ.png" alt="enter image description here" width="690" height="226"></a></p>
<p>After removing last two layer, it look like this<br>
<a href="https://i.stack.imgur.com/zjzjl.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/zjzjl.png" alt="enter image description here" width="690" height="183"></a></p>

ptrblck: <p>You are currently trying to replace the <code>ResNetBasicHead</code> with a simple <code>nn.Sequential</code> container, which might not work as some calls would be missing from the <code>forward</code> method as seen <a href="https://github.com/facebookresearch/pytorchvideo/blob/5984809510df14c0eb37a8edc8d8f77e3fb4865e/pytorchvideo/models/head.py#L371-L391">here</a>. In your particular case I would guess the flattening operation is missing and also be careful about the usage of the “project module” via <code>self.proj</code> since the permutations would also be missing.</p>

Ayanokoji: <p>Hi <a class="mention" href="/u/talhaanwarch">@talhaanwarch</a>, Did you resolve this problems? How did you do?</p>
Inference of fasterrcnn_resnet50_fpn only gives exactly 100 annotations

Snacked: <p>when i try infering my fasterrcnn_resnet50_fpn i only get exactly 100 annotations per image when there are around 200-1000 which should be detected. The bbox-annotations are usually 16x16 pixels big and my image size is 2048x3072. I am not sure if the model itself is limited to output only 100 or if the task is unsuitable for that size of annotations but i have not read it anywhere. What could cause that? I tried changing a view parameters as the AnchorSize and RPN.</p>

Aniruth_Sundararaja1: <p>The default number of detections_per_image is 100.<br>
Refer docs <a href="https://pytorch.org/vision/main/_modules/torchvision/models/detection/faster_rcnn.html#fasterrcnn_resnet50_fpn" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models.detection.faster_rcnn — Torchvision main documentation</a></p>

Snacked: <p>feeling a little embarrassed now. Must have overread it at least two times. Thanks a lot anyway!</p>
Weight initialization variance (cnn) different from expected?

rk7697: <p>I wanted to see the variance of 1 weight of a CNN layer with input size=1 and output size = 1:</p>
<pre data-code-wrap="import"><code class="lang-import">import torch.nn as nn
variance=0
samplesize=10000
for i in range (samplesize):
    layer=nn.Conv2d(in_channels=1,out_channels=1,kernel_size=1)
    variance+=(layer.weight.item())**2

print(variance/samplesize)
</code></pre>
<p>But this seems to converge to 1/3.</p>
<p>In <a href="https://github.com/pytorch/pytorch/blob/f2f4dde2d37d2e2e6c76cdeeeea376cc7ffb8016/torch/nn/modules/conv.py#L178C5-L187C56" rel="noopener nofollow ugc">reset parameter method</a> of <code>torch/nn/modules/conv.py</code> it calls <code>init.kaiming_uniform_(self.weight, a=math.sqrt(5))</code>, and in <a href="https://github.com/pytorch/pytorch/blob/f2f4dde2d37d2e2e6c76cdeeeea376cc7ffb8016/torch/nn/init.py#L440" rel="noopener nofollow ugc">kaiming uniform</a>, it sets nonlinearity to ‘leaky_relu’, so after this code:</p>
<pre data-code-wrap="fan"><code class="lang-fan">    gain = calculate_gain(nonlinearity, a)
    std = gain / math.sqrt(fan)
    bound = math.sqrt(3.0) * std  # Calculate uniform bounds from standard deviation
    with torch.no_grad():
        return tensor.uniform_(-bound, bound, generator=generator)
</code></pre>
<p>I think calculate gain is returning ~ sqrt(2), and fan = 1, so bound=sqrt(6). But the variance of a uniform distribution U(-sqrt(6),sqrt(6)) should be (sqrt(6)^2)/3=6/3=2, right? I’m confused why I’m getting 1/3 for the variance. Did possibly I make any math or code mistakes?</p>
<p>I also notice when <code>reset_parameters</code> initializes the biases, ultimately it does use a uniform distribution U(-1,1), and the resulting variance matches up with (1^2)/3 =1/3. But I get this result from both the variance on layer.weight and layer.bias, so I wasn’t sure. Thx</p>

rk7697: <p>(I realized the answer is because when a=sqrt(5) is passed into Kaiming uniform, nonlinearity is set to “leaky_relu”, and the first elif under nonlinearity=“leaky_relu” sets negative_slope = sqrt(5) and  returns sqrt(2/(1+5))=sqrt(1/3))</p>
Struggling with approximating a 2D function by MLP

dmeister: <p>I’m trying to train an MLP to learn a simple 2D function such as Gaussian or a step function, but the loss converges only to a certain relatively high value. The approximation is almost constant. I tried to tweak learning rate, network size, weight initialization, activation functions, etc, but without any success.</p>
<pre><code class="lang-auto">import numpy as np
import torch
import matplotlib.pyplot as plt
from torch import optim, nn

device = "cuda" if torch.cuda.is_available() else "cpu"
torch.set_default_device(device)


class Net(nn.Module):
    def __init__(self):
        super().__init__()

        self.n = 16

        self.fc1 = nn.Linear(2, self.n)
        self.fc2 = nn.Linear(self.n, self.n)
        self.fc3 = nn.Linear(self.n, self.n)
        self.fc4 = nn.Linear(self.n, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = torch.relu(self.fc3(x))
        return self.fc4(x)


def loss_fn (z_pred, z):
    loss = torch.mean((z_pred - z) ** 2)
    return loss


def f(x, y):
    return np.logical_and(x &lt; 0.5, y &lt; 0.5).astype(float)
    # return np.exp(-(x**2 + y**2))


if __name__ == '__main__':
    n = 50
    learning_rate = 0.001
    epochs = 1500

    t = torch.linspace(0, 1, n)
    X = torch.cartesian_prod(t, t).transpose(0, 1)
    print(X.size())
    x = X[0]
    y = X[1]
    g = torch.from_numpy(f(x.cpu().detach().numpy(), y.cpu().detach().numpy()))
    g = g.to(device)
    print("x ", x.size(), " y ", y.size(), " g ", g.size())

    Xt, Yt = np.meshgrid(t.cpu(), t.cpu())

    model = Net()
    optimizer = optim.Adam(model.parameters(), lr=learning_rate)
    X = torch.cat((x.reshape(-1, 1), y.reshape(-1, 1)), dim=1)

    for e in range(epochs):
        g_pred = model(X)
        loss = loss_fn(g_pred, g)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if e % 100 == 0:
            print(e, loss.item())
</code></pre>

soulitzer: <p>What is the largest model size you’ve tried?</p>

dmeister: <p>I found a bug in the loss function (z_pred and z had different shape). My bad.</p>
Having different number of in_features and out_feature in nn.Linear?

luan_nguy_n: <p>I understand that the out_features in Linear are often lower than the in_features to get more meaningful features but sometimes I see the out_features is higher than the in_features, sometimes it’s equal.</p>
<p>I noticed in the VGG19, that the first two of the three last FC layers have the same 4096 channel.</p>
<p>In swin-transformer we have:</p>
<pre><code class="lang-auto">Sequential(
      (0): SwinTransformerBlockV2(
        (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
        (attn): ShiftedWindowAttentionV2(
          (qkv): Linear(in_features=768, out_features=2304, bias=True) #Higher
          (proj): Linear(in_features=768, out_features=768, bias=True) #Equal
          (cpb_mlp): Sequential(
            (0): Linear(in_features=2, out_features=512, bias=True)
            (1): ReLU(inplace=True)
            (2): Linear(in_features=512, out_features=24, bias=False) 
          )
</code></pre>
<p>Does having this help anything to the networks?<br>
I want to ask:</p>
<ol>
<li>What are the purposes of having higher, equal, lower out_features in the network?</li>
<li>Can you provide me with some papers about this matter and network architectures having this?</li>
</ol>
<p>I have done some experiments on the last layers of my custom network by having higher out_features than in_feature, followed by the equal, then followed with lower for the output. It gives better results sometimes</p>

Aniruth_Sundararaja1: <p>Hi <a class="mention" href="/u/luan_nguy_n">@luan_nguy_n</a></p>
<p>That is an interesting question.</p>
<p>So generally we have a lower to higher feature when we want the model to increase its learning capacity and learn more patterns.</p>
<p>We have equal features, when we just want a linear transformation of features, without actually changing dimensionality.</p>
<p>We have Higher to lower features when we want to lose unwanted information or features. This helps in speeding up computation and avoid overfitting.</p>

luan_nguy_n: <p>Thank you for your answer <a class="mention" href="/u/aniruth_sundararaja1">@Aniruth_Sundararaja1</a> .<br>
I experimented with my custom network and it gave me better results, quite curious about why it does. My hypothesis is the same as yours, but finding any research talks about it really hard.</p>
Dirichlet/Gamma Distribution Sampling And Reparam. Gradients

maxwass: <p>I’m attempting to find the methods used to sample from, and compute the reparameterized gradients of, the Dirichilet and Gamma distributions.</p>
<p>The <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributions/dirichlet.py#L71" rel="noopener nofollow ugc">Dirichlet</a> distribution and the <a href="https://pytorch.org/docs/stable/_modules/torch/distributions/gamma.html#Gamma" rel="noopener nofollow ugc">Gamma</a> Distribution both include seemingly hidden functions.</p>
<p>For Dirichlet: <code>torch._sample_dirichlet</code>, <code>torch._dirichlet_grad,</code></p>
<p>For Gamma: <code>torch._standard_gamma</code></p>
<p>Is there documentation on the sampling/gradient methods, and these functions?</p>

maxwass: <p>Bumping this. <a class="mention" href="/u/ptrblck">@ptrblck</a>, <a class="mention" href="/u/smth">@smth</a> do you folks know who the correct person to ask about this would be?</p>

smth: <p>Hey Max.</p>
<p>Looks like these functions were never properly documented, but the CPU implementations for these are pretty readable.</p>
<p>This is where <code>_sample_dirichlet</code> goes to: <a href="https://github.com/pytorch/pytorch/blob/a6ac6447b55bcf910dee5f925c2c17673f162a36/aten/src/ATen/native/Distributions.cpp#L512-L557" class="inline-onebox">pytorch/aten/src/ATen/native/Distributions.cpp at a6ac6447b55bcf910dee5f925c2c17673f162a36 · pytorch/pytorch · GitHub</a> . I figured this out by looking for <code>_sample_dirichlet</code> in the <a href="https://github.com/pytorch/pytorch/blob/a6ac6447b55bcf910dee5f925c2c17673f162a36/aten/src/ATen/native/native_functions.yaml">native_functions.yaml</a> file.</p>
<p>This is where <code>_standard_gamma</code> goes to: <a href="https://github.com/pytorch/pytorch/blob/a6ac6447b55bcf910dee5f925c2c17673f162a36/aten/src/ATen/native/Distributions.cpp#L482-L510" class="inline-onebox">pytorch/aten/src/ATen/native/Distributions.cpp at a6ac6447b55bcf910dee5f925c2c17673f162a36 · pytorch/pytorch · GitHub</a></p>
<p>For both of these, the core <code>gamma</code> sampling function that they call is implemented here (lifted straight up from NumPy sources): <a href="https://github.com/pytorch/pytorch/blob/a6ac6447b55bcf910dee5f925c2c17673f162a36/aten/src/ATen/native/Distributions.h#L63" class="inline-onebox">pytorch/aten/src/ATen/native/Distributions.h at a6ac6447b55bcf910dee5f925c2c17673f162a36 · pytorch/pytorch · GitHub</a></p>
<p>Hope this helps.</p>

maxwass: <p>Yes, this is very helpful! Thanks for diving into the belly of the beast for me…I would have never found these!</p>
<p>Can I assume the GPU implementations are in effect the ~same algorithms?</p>

smth: <p>yes the gpu implementations are the same algorithms</p>
TorchScript and C++

jakkes12: <p>Hi,</p>
<p>I am trying to load a torch script i C++ as per <a href="https://pytorch.org/tutorials/advanced/cpp_export.html#step-3-loading-your-script-module-in-c" rel="noopener nofollow ugc">Loading a TorchScript Model in C++ — PyTorch Tutorials 2.3.0+cu121 documentation</a>, using <code>torch::jit::load</code>. However, I cannot seem to find this function, was it newly added/removed or something? Is the guide out of date? I am running libtorch 2.1.0cu121, but I see people online using it pre v2.0, yet the documentation refers to v2.3.</p>
<p>Thankful for any insights.</p>

BiggestZocker: <p>Sometimes the imports to Libtorch functions are very well hidden. You can try<br>
<code>#include &lt;torch/csrc/jit/serialization/import.h&gt;</code></p>
How to reach the exact source code of convolution and linear fucntions in Pytorch?

rzjhdi: <p>Hello, I wish to apply some changes in the plain multiplication operation of convolution and Linear layers of a Neural Network implemented by Pytorch. I know there are other topics asking the same thing and some of them got to sources. However, when I see those cpp source codes, they are irrelevant (for example some of the replies are: third_party/ideep/mkl-dnn/src/cpu/ , or ConvolutionMM2d.cpp, or aten/src/ATen/native/Convolution.cpp:804 , or …)</p>
<p>I even installed pytorch from source in DEBUG mode in order to find the source cpp implementation using GDB debugger, but nothing was found. I am fed up trying all the ways. It is so weird that you follow up a function and hope to finally find the plain convolution implementation, but reach some unknown strange functions at the end.</p>
<p>I would appreciate it if someone helps me. Thank you in advance.</p>
Getting nan in loss when training with torchtune on ROCM system

lemon07r: <p>Been trying to troubleshoot this for a while, on my Fedora 40 and RX 6900 XT system.</p>
<p>I have torchtune compiled from the github repo, installed ROCM 6.0 from the official fedora 40 repos, which I uninstalled to install ROCM 6.1.2 from AMD’s ROCM repo following their documentation for RHEL 9.4. I originally had pytorch 2.5-rocm6.0, which I’ve updated to the latest nightly for 2.5-rocm6.1.</p>
<p>I still always get nan in loss when training. One of the torchtune devs gave me a recipe for training in fp16, this more than tripled my training speed from 25t/s to 79t/s but still shows my loss as nan. All testing has been done on training a lora for phi mini using a small 10k line dataset and 32 seq length for testing purposes. Both my bf16 and fp16 recipes have been confirmed working fine on nvidia machines without nan in their training loss by others.</p>
<p>Sidenote, I also had an issue with an hipblaslt error which I worked around with <code>export TORCH_BLAS_PREFER_HIPBLASLT</code> (see <a href="https://github.com/pytorch/torchtune/discussions/1108" class="inline-onebox" rel="noopener nofollow ugc">HIPBLASLT error, and the work around for AMD/ROCM users who are getting it · pytorch/torchtune · Discussion #1108 · GitHub</a> for more details).</p>
<p>Update: the issue only happens with the creativegpt dataset by N8programs. When I use the alpaca dataset I can see loss just fine. However I’ve had others try this dataset with my same recipe/configs on their Nvidia machines and they don’t get any issues like me.</p>
What is the linear embedding layer in Swin Transformer?

jasonrichdarmawan: <p>my goal is to understand what is Linear Embedding layer actually is in Swin Transformer</p>
<p>The problem is the Swin Transformer’s paper do not explain what is the linear embedding layer.</p>
<p>arXiv:</p>
<ol>
<li>
<p>arxiv org/pdf/2103.14030</p>
<blockquote>
<p>A linear embedding layer is applied on this raw-valued feature to project it to an arbitrary dimension (denoted as C).</p>
</blockquote>
<p>This is the only explanation given for linear embedding in the paper.</p>
</li>
</ol>
<p>PyTorch documentation:</p>
<ol>
<li>Linear pytorch org/docs/stable/generated/torch.nn.Linear.html</li>
<li>Embedding pytorch org/docs/stable/generated/torch.nn.Embedding.html</li>
</ol>
<p>YouTube:</p>
<ol>
<li>Embedding youtube com/watch?v=aWFllV6WsAs</li>
</ol>
<p>StackOverflow:</p>
<ol>
<li>The difference between Linear and Embedding in PyTorch stackoverflow com/questions/65445174/what-is-the-difference-between-an-embedding-layer-with-a-bias-immediately-afterw</li>
</ol>
<p>Honestly, I do not know where to start</p>

soulitzer: <p>From the PyTorch implementation:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/microsoft/Swin-Transformer/blob/main/models/swin_transformer.py#L437">
  <header class="source">

      <a href="https://github.com/microsoft/Swin-Transformer/blob/main/models/swin_transformer.py#L437" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/microsoft/Swin-Transformer/blob/main/models/swin_transformer.py#L437" target="_blank" rel="noopener nofollow ugc">microsoft/Swin-Transformer/blob/main/models/swin_transformer.py#L437</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="427" style="counter-reset: li-counter 426 ;">
          <li></li>
          <li>    def flops(self):</li>
          <li>        flops = 0</li>
          <li>        for blk in self.blocks:</li>
          <li>            flops += blk.flops()</li>
          <li>        if self.downsample is not None:</li>
          <li>            flops += self.downsample.flops()</li>
          <li>        return flops</li>
          <li></li>
          <li></li>
          <li class="selected">class PatchEmbed(nn.Module):</li>
          <li>    r""" Image to Patch Embedding</li>
          <li></li>
          <li>    Args:</li>
          <li>        img_size (int): Image size.  Default: 224.</li>
          <li>        patch_size (int): Patch token size. Default: 4.</li>
          <li>        in_chans (int): Number of input image channels. Default: 3.</li>
          <li>        embed_dim (int): Number of linear projection output channels. Default: 96.</li>
          <li>        norm_layer (nn.Module, optional): Normalization layer. Default: None</li>
          <li>    """</li>
          <li></li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The Conv2D should be responsible for both the patch splitting and then the embedding</p>
Pytorch calculate loss backward when divide tensor into subgroup and put into different network

ryan_chandra: <p>I am new to PyTorch coding. I want to calculate keypoints X and Y coordinate prediction. I am thinking to divide keypoints into subgroups and put those keypoints that belong to same group to same network. For example, I have tensor with shape [bs, 14, 128] (batch size, number of keypoints, feature size). I use tensor slicing to divide into 3 groups:</p>
<p>I am new to PyTorch coding. I want to calculate keypoints X and Y coordinate prediction. I am thinking to divide keypoints into subgroups and put those keypoints that belong to same group to same network. For example, I have tensor with shape [bs, 14, 128] (batch size, number of keypoints, feature size). I use tensor slicing to divide into 3 groups:</p>
<p>group_one[:, :4, :]</p>
<p>group_two[:, 4:9, :]</p>
<p>group_three[:, 9:14, :]</p>
<p>and then put those 3 subgroups into different network and concatenate back into [bs, 14, 128]. Is the loss backward can be calculated as usual and the parameters are updated correctly?</p>

soulitzer: <p>Views should be differentiable yes, so the gradients computed for each view should flow back to the base.</p>
Mat1 and mat2 cannot be multiplied

FightTheByte: <p>Hello all, I’m trying to follow the tutorial but can’t seem to get a clean run of training the model. The error I get is:<br>
“RuntimeError: mat1 and mat2 shapes cannot be multiplied (49152x256 and 65536x512)”</p>
<p>Now I’ve tried changing the value of the in and out for every available option but nothing will make the mats match up, this includes following the answer to similar topics - such as adding x = x.unsqueeze(0) . Any help would be greatly appreciated, my code is below:</p>
<pre><code class="lang-auto">transform = transforms.Compose([
    transforms.Resize((256, 256))
])

dataset_train = CustomImageDataset(annotations_file='train_csv/labels.csv', img_dir='train', transform=transform)
dataset_val = CustomImageDataset(annotations_file='val_csv/labels.csv', img_dir='val', transform=transform)

train_dataloader = DataLoader(dataset_train, batch_size=64, shuffle=True)
val_dataloader = DataLoader(dataset_val, batch_size=64, shuffle=True)


device = (
    "cuda"
    if torch.cuda.is_available()
    else "mps"
    if torch.backends.mps.is_availble()
    else "cpu"
)

class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(256*256, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 9),
        )

    def forward(self, x):
        x = x.unsqueeze(0)

        x = x.to(torch.float32) 
        logits = self.linear_relu_stack(x)
        return logits

model = NeuralNetwork().to(device)
</code></pre>

soulitzer: <p>Your input needs to have shape of (*, 65536)</p>

FightTheByte: <p>So I think the answer here is that I need to represent the image in all three colour spectrums (R G B * 256 * 256) so 3 * 256 * 256 so that when it’s flattened it represents a 2d vector of the entire image across all colour spectrums?</p>

soulitzer: <p>Yes, and that is because you have a linear layer that requires a tensor with 256^2 as the embedding dim.</p>
Adding Custom Operator under torch.compile

bwyogatama: <p>Hi,</p>
<p>I want to introduce custom operators in Pytorch 2.0. The goal is to make my custom operator fusable with other existing operators when running it using torch.compile.  I know I probably would have to make some modification to TorchInductor/TorchDynamo and introduce new Aten ops but I am not sure what is the best way to do this. Can someone tell me which file I should modify/take a look at in order to achieve this? Pointer to URL/github commit would also be useful. Step-by-step tutorial is even more appreciated.</p>
<p>Thank you!</p>

marksaroufim: <p>You can follow this thread for updates, I believe this doesnt work quite yet <a href="https://github.com/pytorch/pytorch/issues/101191" class="inline-onebox">custom_op API follow-ups · Issue #101191 · pytorch/pytorch · GitHub</a></p>

richard: <p>Please see <a href="https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html" class="inline-onebox">PyTorch Custom Operators Landing Page — PyTorch Tutorials 2.3.0+cu121 documentation</a> for the newest up-to-date guide.</p>
Timeseries forecasting for more than 1 day in the future

Karls: <p>I’m trying to forecast future trends basing of future data. I’ve go through some tutorials like here:</p>
<ul>
<li><a href="https://github.com/jinglescode/time-series-forecasting-pytorch?tab=readme-ov-file" rel="noopener nofollow ugc">jinglescode/time-series-forecasting-pytorch: Acquiring data from Alpha Vantage and predicting stock prices with PyTorch’s LSTM (github.com)</a></li>
<li><a href="https://www.geeksforgeeks.org/time-series-forecasting-using-pytorch/" rel="noopener nofollow ugc">Time Series Forecasting using Pytorch - GeeksforGeeks</a></li>
</ul>
<p>And I was able to reproduce it on my data. But both approaches provides the same concept:</p>
<ol>
<li>ANN gets values for days in the range from x to x+w (where w is a lookback window, usually 20 - 50 points).</li>
<li>ANN returns predictions for days x+1 to x+w+1 (so overlapping with input)</li>
<li>Dropping all days except the last - this is the result</li>
</ol>
<p>So basically, it always models only one day in the future. You can use it in a loop, by adding the newest value to the end of input but as both train and test dataset is evaluated versus only one-day shift, the future predictions are usually poor, with a lot of oscillation. Also, because 90% of the modeled data overlaps with the input, it mask the error on the last day. So even if the “new” day is poorly modeled, the other overlapping days will hide it because the error is mean value. Is it the only way, or can I do it better? If so can you provide me with som MWE or tutorials how to it?</p>
Synthesis of multivariate timeseries data using vanila transformer architecture

jitendra_tiwari: <p>Hi Folks,<br>
I am working on my thesis and using the C-MAPSS dataset. There, I have been using Transformer architecture to synthesize the data. Here, I use the first 80 engine IDs for training purposes and the remaining 81-90 engine IDs for validation purposes, and thereafter, I use 91-100 engine IDs for test purposes. I did not use train_test_split functionality to get train and validation datasets. I have also created two extra attributes in the data frame to classify binary and multiclass classification. However, after training, my training loss is greater than my validation loss. Is there any way can resolve this issue?</p>
Configuration and compilation in python env

1128925dfb: <p>Hello,</p>
<p>I have to work with a PyTorch project, but I rather new with it, and I don’t really know where to start.</p>
<p>At first a description of the current architecture of the project :</p>
<ul>
<li>The project is mainly a big Python module. I need only a little part of it associated with ML, but</li>
<li>The input for the ML is preprocessed by a C++ module for performance reason. It also use a custom convolution filter for cuda, also written in C++ with cuda api</li>
<li>As they had problem of retrocompatibility with cuda updates and installation of GPU drivers they put everything in a docker. As I am under windows I have to use wsl. So, there is 3 layer of virtualisation : wsl, docker, conda. This is awful to install.</li>
</ul>
<p>So, what I would like to do :</p>
<p>Create a new project in a virtual env (with pip ? Is there a special reason to use conda ?), without docker and wsl. The existing code is rather good and clear, so I would like to keep and re-use it. I would like something easy to clone from a computer to another.</p>
<p>For now, the main blocking point is this C++ embed in a Python module. This is the first time I work on this kind of project. For now, the compilation is launch each time <code>pip install -e .</code> is launched in the project, and it rather works when I am not fighting against conda and docker. But I am don’t really understand how it works and I don’t find many documentation on the net.</p>
<ul>
<li>how should I organise my new project ? Is there good patterns for this ?</li>
<li>how am I supposed to manage and compile the C++ part of the code ?</li>
</ul>
<p>Thanks for your help.</p>
Fsdp: implementation question of _check_order

Huang_Henry: <p>fsdp: torch/distributed/fsdp/_exec_order_utils.py:_ExecOrderData:_check_order</p>
<pre><code class="lang-auto">            if not torch.distributed._functional_collectives.is_torchdynamo_compiling():
                # TODO(voz): Don't graph break on this - dynamo hates the n1 != n2
                # tensor comparison control flow.
                # https://github.com/pytorch/pytorch/issues/107055
                for (r1, n1), (r2, n2) in itertools.combinations(
                    (
                        (rank, world_num_valid_indices[rank])
                        for rank in range(self.world_size)
                    ),
                    2,
                ):
                    if n1 != n2:
                        raise RuntimeError(
                            f"{msg_prefix} rank {r1} is all-gathering {n1} parameters "
                            f"while rank {r2} is all-gathering {n2} parameters"
                        )
</code></pre>
<p>It seems that to confirm all idx in world_num_valid_indices are same.<br>
Why not just compare rank1～rankn idx with rank0 idx?</p>
<p>test program</p>
<pre><code class="lang-auto">import itertools
import torch
import time

def ori_cmp(ws, world_size):
    c = itertools.combinations( ((rank,ws[rank]) for rank in range(world_size)), 2)
    for (r1, n1), (r2, n2) in c:
        if n1 != n2:
            print('diff', r1, r2)

def opt_cmp(ws, world_size):
    for i in range(1, world_size):
        if ws[i] != ws[0]:
            print('diff', 0, i)

t0 = time.time()
world_size = 1024
ws = [torch.tensor(1, dtype=torch.int32) for i in range(world_size)]
ori_cmp(ws, world_size)
t1 = time.time()
opt_cmp(ws, world_size)
print(t1 - t0, time.time() - t1)
</code></pre>
<p>result:<br>
1.0049409866333008 0.0021049976348876953</p>
Efficient (lazy) dataset loading of large .npy file?

sad_robot: <p>Hi, I want to know the most efficient <code>Dataset</code>/<code>DataLoader</code> setup to lazy load a large .npy array dataset. I’ve already tried these methods and found them be needlessly inefficient:</p>
<p>•	Loading the <code>.npy</code> file as a persistent numpy mmap (via <code>np.load(mmap_mode=‘r’)</code>) → The caching is great, but unfortunately it doesn’t play nice with parallel data loaders and occasionally makes <code>__getitem__()</code> take like x4-x6 longer than it does with <code>num_workers=0</code> (e.g. 30-60 seconds vs 4 seconds).<br>
•	Loading the <code>.npy</code> file as a persistent torch mmap (via <code>torch.from_file</code>) → This is sometimes faster than numpy, but it has the same problem when <code>num_workers</code>&gt;0 and it sometimes OOMs from too much shared memory (i.e. greater cache usage).<br>
•	Lazily opening (&amp; closing) temporary mmaps inside <code>__getitem__()</code> → <a href="https://stackoverflow.com/questions/46190787/does-mmap-directly-access-the-page-cache-or-a-copy-of-the-page-cache" rel="noopener nofollow ugc">There is a strange problem in linux</a> where opening multiple mmaps on the same file causes extra overhead. …And I just found this method slower than the other options.</p>
<p>How can I achieve the ideal scenario where data loading/preprocessing costs are hidden by the <code>DataLoader()</code> in this situation?</p>
<p>P.S. Once by chance (I cannot reproduce this) I got the dataloaders &amp; the mmap working together; I had achieved like 80 ms per batch.</p>
Beginner problem with forward function

josephP: <p>Hello<br>
I’m working my way through the github tutorial and I have run into an error that I can not get passed. The error message that I recieve is “Module [CircleModelV0] is missing the required “forward” function” do any of you know what this means and what might be wrong with my code? thanks</p>
<p>class CircleModelV0(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
<span class="hashtag-raw">#2creating</span> the layerscapable of handleing x inuts and y outputs<br>
self.layer_1 = nn.Linear(in_features=2,out_features=5)<br>
self.layer_2= nn.Linear(in_features=5,out_features=1)</p>
<pre><code>    #3 defining the forward method containing the forward pass computation
    def forward(self, x):
        #return the output layer_2, a single feature in the shape as y
       
        return self.layer_2(self.layer_1(x))# first goes through layer 1 than the output of 1
</code></pre>
<p><span class="hashtag-raw">#4</span> instantiate<br>
model_0 = CircleModelV0().to(device)<br>
model_0</p>
<p>untrained_preds = model_0(x_test.to(device))<br>
print(f"Length of predictions{len(untrained_preds)}, shape: {untrained_preds.shape}“)<br>
print(f"length of test samples:{len(y_test)}, shape: {y_test}”)<br>
print(f"\nfirst 10 test predictions:\n{untrained_preds[:10]}“)<br>
print(f”\nFirst 10 test lables:\n{y_test[:10]}")</p>

soulitzer: <p>It means your CircleModelV0 module needs to define a method called “forward”. You can check out the docs here <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module" class="inline-onebox" rel="noopener nofollow ugc">Module — PyTorch 2.3 documentation</a></p>
Masked Query Gradient Flow to Keys and Values

DeerFreak: <p>Hi,<br>
I was wondering why the gradient in this example does not flow to the key and value. What am I doing wrong? How can I use padded batches with different target sequence lengths?</p>
<pre><code class="lang-auto">import torch
from torch.nn.functional import scaled_dot_product_attention

k = v = torch.rand(3, 4, 8)
q = torch.rand(3, 5, 8)

q.requires_grad = True
k.requires_grad = True
v.requires_grad = True

mask = torch.ones(3, 5, 4, dtype=torch.bool)
mask[:, :, -1] = 0
mask[:, -1, :] = 0

out = scaled_dot_product_attention(q, k, v, attn_mask=mask)
torch.mean(out[:, :-1, :]).backward()
</code></pre>
<p>Thank you</p>

soulitzer: <p>To take the example of v, the gradient of v is softmax(q @ k + bias).T @ grad_output<br>
softmax(q @ k + bias) produces a row with entirely nans because you masked out an entire row (implies bias of -inf), so it’s transpose has an entire column of nans.</p>
<p>grad_output has an entire row full of zeros, due to viewing the corresponding row of out.<br>
You may have expected the row of zeros of the rhs to cancel out the column of nans in the lhs since they are the last row and the last column respectively.<br>
However is not true in autograd today as zero times nan is still nan, so since the gradient of v is a linear combination of the columns of the lhs, this results the entirety of grad of v becoming nan.</p>
<p>One workaround for this is to specify a floating mask instead of a boolean mask, and use a large negative number instead of -inf so that softmax no longer produces nans.</p>

DeerFreak: <p>Would it be a good idea to fill the mask with ones for masked queries? As the forward pass will be performed anyways, but the masked query does not influence the output, as I mask it in the loss function anyways?</p>

soulitzer: <p>Hm I’d expect this to affect the normalization for softmax. Like if you had a large outlier, normalizing that with the rest of your row would mean your outputs are scaled down.</p>

DeerFreak: <p>Okay, I might have understood what I got wrong. In this context pertains the attention mask only the key/value pairs for specific queries. Me masking all key/value pairs for queries I would like to mask (padded ones), creates this zero row, which destroys the gradient because 0 times the nan value still influences both key and value gradients. Me ignoring the masked queries in the calculation of the loss function should fix this, as the results of the masked queries do not influence the loss function and therefore not the gradients. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to export tensorboard data to .csv file

AlphaBetaGamma96: <p>Hi All,</p>
<p>I’m going through the tutorial of tensorboard (tutorial here: <a href="https://pytorch.org/docs/stable/tensorboard.html" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.tensorboard — PyTorch 2.3 documentation</a>), but I can’t see a way to export the data from a log to say <code>.csv</code> to allow post-processing to take place in <code>pyplot</code> etc. Is this possible?</p>
<p>I briefly saw in <code>tensorflow</code> there’s a <code>tb.data.experimental.ExperimentFromDev</code> function, (as seen here: <a href="https://www.tensorflow.org/tensorboard/dataframe_api" class="inline-onebox" rel="noopener nofollow ugc">Accessing TensorBoard Data as DataFrames &nbsp;|&nbsp; TensorFlow</a>). Is there something equivalent in PyTorch?</p>
How to make group sum of single row tensor?

fahadh4ilyas: <p>So, I have a row tensor like this</p>
<pre data-code-wrap="python"><code class="lang-python">values = torch.tensors([0.2, -0.1, 0.9, -0.4, 0.3, 0.1, -0.5, -0.7])
</code></pre>
<p>And also, I have a tensor that is the cumsum of the length of sub Tensor</p>
<pre data-code-wrap="python"><code class="lang-python">cum_seqlens = torch.tensors([0,2,2,4], dtype=torch.long)
</code></pre>
<p>I want to sum the subset of <code>values</code> based on the <code>cum_seqlens</code> like this:</p>
<pre data-code-wrap="python"><code class="lang-python"># because the cumsum is 2, 2, 4, the tensor is split into [0.2, -0.1], [0.9, -0.4], and [0.3, 0.1, -0.5, -0.7]
torch.Tensor([0.2+(-0.1), 0.9+(-0.4), 0.3+0.1+(-0.5)+(-0.7)])
</code></pre>
<p>I could just split the tensor, but the result is a tuple of sub tensors. I want to keep the tensors intact because I need the grad of process.</p>
<p>I already search for the solution and found this similar problem <a href="https://discuss.pytorch.org/t/sum-over-various-subsets-of-a-tensor/31881">here</a>. But, the accepted answer is user <code>encoding_indices</code> while I don’t have it. I also have row tensor while theirs is column tensor.</p>

soulitzer: <aside class="quote no-group" data-username="fahadh4ilyas" data-post="1" data-topic="204992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fahadh4ilyas/48/70874_2.png" class="avatar"> fahadh4ilyas:</div>
<blockquote>
<p>I want to keep the tensors intact because I need the grad of process.</p>
</blockquote>
</aside>
<p>If this is the reason you need the tensors intact, splitting into smaller should be fine for you, as long as you use view operations that autograd can understand, e.g. the slice operation.<br>
It is definitely trickier however, if you have a really large tensor, and worry about performance.</p>

fahadh4ilyas: <p>My tensor length could reach to 81920 elements and be splitted into 8000 sub tensor. will it hurt the performance if I’m using split?</p>

soulitzer: <p>Yes definitely, though depending on how perf sensitive your case is, maybe it is still worth checking .</p>
Dynamically add and remove layers through module hooks?

Zenmar: <p>Hi</p>
<p>I’m trying to add adapters to ViT which I would love to be able to dynamically add and or remove.</p>
<p>I am currently able to add these layers to the graph through the hooks, and to freeze/unfreeze everything as needed, but using the handles to remove the hooks (and I expect the layers / parameters) doesn’t seem to work.</p>
<pre><code class="lang-auto">class Hooker:
    def __init__(self):
        self.hooks = {}
        #self.hk_idx = 0
    def register_hook(self, module, hook, layer_name):
        handle = module.register_forward_hook(hook)
        self.hooks[layer_name] = handle
        #self.hooks[self.hk_idx] = handle
        print(f"Hook registered for {layer_name}")
        print(f"Handle name: {handle}")
        self.hk_idx+=1

    def remove_hook(self, layer_name):
        if layer_name in self.hooks:
            self.hooks[layer_name].remove()
            print(f"Hook removed for {layer_name}")
            # del self.hooks[layer_name]
        else:
            print(f"No hook found for {layer_name}")

    def remove_all_hooks(self):
        for layer_name, handle in self.hooks.items():
            handle.remove()
            print(f"Hook removed for {layer_name}")
        # self.hooks = {}
</code></pre>
<p>When I count my parameters before and after adding the hooks, it correctly is adding the parameters, but when calling the function to remove them, they stay the same; am I missing something?</p>

soulitzer: <p>I don’t actually expect parameters to be added when hooks are registered to a module, do you know where that is happening?</p>

Zenmar: <pre><code class="lang-auto">def add_adapters_with_hooks(image_encoder, bottleneck_dim):
    adapters = nn.ModuleList()
    hook_manager = Hooker()

    for idx, block in enumerate(image_encoder.blocks):
        adapter = Adapter(block.norm1.normalized_shape[0], bottleneck_dim)
        adapters.append(adapter)

        def hook(module, input, output, adapter=adapter):
            return adapter(output)

        hook_manager.register_hook(block, hook, f'block_{idx}')

    return adapters, hook_manager
</code></pre>
<p>and:</p>
<pre><code class="lang-auto">image_encoder = model.image_encoder
adapters, hook_manager = add_adapters_with_hooks(image_encoder, bottleneck_dim)
model.image_encoder.adapters = adapters
model.image_encoder.hook_manager = hook_manager
</code></pre>

soulitzer: <p>I see, thanks for clarifying. Since you manually appended the adapters into the module list, I’d expect you to also have to manually remove the adapters from the module list yourself. Removing hooks would not removing the parameters.</p>

Zenmar: <p>I’m not completely sure how I would about doing this, would you maybe know?</p>

soulitzer: <p>Maybe I am missing something, but it should be something like - as you append to the adapters, remember what index of the list you appended to. Then when you desire, you delete the corresponding adapter from the list.</p>
Using GPU in NVIDIA GEForce MX450

Neel_Patel: <p>I’m having NVIDIA GEForce MX450 and I want to use it for the Deep learning .So can anyone guide which Cuda version to use ?<br>
Also I want to use it with tensorflow.</p>

ptrblck: <p>Your GPU should be supported in all of our current PyTorch binaries as these ship with all CUDA runtime dependencies and also support your Turing GPU. I don’t know what TF supports and you might want to ask in their discussion board.</p>