akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why there isn't a method named torch.softmax

bat67: <p>Hi~</p>
<p>For sigmoid, there’s <a href="https://pytorch.org/docs/stable/nn.html?highlight=sigmoid#torch.nn.Sigmoid" rel="nofollow noopener"><code>torch.nn.Sigmoid</code></a> to generate a instance of Sigmoid class, and <a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=sigmoid#torch.nn.functional.sigmoid" rel="nofollow noopener"><code>torch.nn.functional.sigmoid</code></a> function. I noticed another <a href="https://pytorch.org/docs/stable/torch.html?highlight=sigmoid#torch.sigmoid" rel="nofollow noopener"><code>torch.sigmoid</code></a> function and some related topics (<a href="https://discuss.pytorch.org/t/torch-nn-sigmoid-vs-torch-sigmoid/57691">link1</a>, <a href="https://discuss.pytorch.org/t/is-there-any-different-between-torch-sigmoid-and-torch-nn-functional-sigmoid/995">link2</a>).</p>
<p>However, for softmax function for example, there’s only <a href="https://pytorch.org/docs/stable/nn.html?highlight=softmax#torch.nn.Softmax" rel="nofollow noopener"><code>torch.nn.Softmax</code></a> and <a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=softmax#torch.nn.functional.softmax" rel="nofollow noopener"><code>torch.nn.functional.softmax</code></a>, but <strong>no</strong> <code>torch.softmax</code>. I am confused and want to know what is the thinkings behind designing these. And are there other functions designed like this?</p>
<p>Thanks!</p>

tom: <p>So the idea is to put more deep-learning-oriented functions in <code>torch.nn.functional</code> and keep general-purpose functions in under torch directly. <code>softmax</code> was deemed to fall into the former, <code>sigmoid</code> in the latter category.<br>
While there is <code>torch.softmax</code>, this is by accident (which is why it is not documented) rather than as a design (previous version of PyTorch didn’t have the fancy <code>torch._C._nn</code> module to put the <code>C++</code>-implementations of <code>torch.nn.functional</code>-functions). I would advise to only use the documented variants to stay out of trouble should someone start to clean up.</p>
<p>Best regards</p>
<p>Thomas</p>

bat67: <p>Thank you very much!</p>

isarandi: <p>Apparently <code>torch.softmax</code> is now a documented alias.</p>
Significant time difference to transfer tensor from gpu to cpu between 2 models

aleoris: <p>I have a Python script running two models on CUDA. One model is larger than the other, but their inference times are quite similar (~3ms vs. ~9ms). The issue arises when I move the output tensors from the GPU back to the CPU. The output tensor of the smaller model takes around <strong>15ms</strong> to transfer, while the output tensor of the larger model takes <strong>~500ms</strong>—even though the larger model’s output is actually <strong>smaller</strong> than that of the smaller model.</p>
<p>I find this behavior puzzling. Does anyone have an explanation?</p>

ptrblck: <aside class="quote no-group" data-username="aleoris" data-post="1" data-topic="217539">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3ab097/48.png" class="avatar"> aleoris:</div>
<blockquote>
<p>The issue arises when I move the output tensors from the GPU back to the CPU.</p>
</blockquote>
</aside>
<p>You are adding a synchronization to the code and the host will wait until the transfer is done. All previously scheduled CUDA kernels will still be executed and the host timer will accumulate their time unless you explicitly synchronize the code before starting and stopping host timers.</p>
Setting module buffers without using register_buffer?

ferrophile: <p>The typical method to assigning buffers to a module is obviously using <code>register_buffer</code>:</p>
<pre><code class="lang-python">foo = torch.random.rand(3, 4)
self.register_buffer('buffer', foo)
</code></pre>
<p>and the buffer can be accessed by <code>self.buffer</code> elsewhere in the module class.</p>
<p>My question is, is it possible to first explicitly perform an assignment (<code>self.buffer = ...</code>), then somehow “mark” this attribute as a buffer? While this does not affect the actual runtime behavior, I rely a lot on my IDE’s hinting / auto-complete features, and the IDE cannot detect attributes if they are not assigned values by an assignment statement somewhere else (preferrably in <code>__init__</code>) in the first place.</p>
<p>It seems that parameters / modules can be registered by assignment, instead of using the dedicated methods <code>register_parameter</code> and <code>add_module</code>. I wonder if the same can be done for buffers.</p>

ptrblck: <p>It’s not possible to directly register a buffer via a plain assignment, but you could call <code>self.register_buffer</code> afterwards as seen here:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.my_buffer = torch.randn(1)
        self.register_buffer('buffer', self.my_buffer)
        
model = MyModel()
print(dict(model.named_buffers()))
# &gt; {'buffer': tensor([-0.3787])}
</code></pre>
<p>Would this work for you?</p>

isarandi: <p>I think <a class="mention" href="/u/ferrophile">@ferrophile</a> wanted the IDE to know about the existence of <code>self.buffer</code>.</p>
<p>The current best practice seems to be:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.buffer = nn.Buffer(torch.randn(1))
</code></pre>
"Error when running official test cases: test cases not found."

mmzhangna: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07f427ed447441c796cc00368ae2ec68bd0ceb69.png" data-download-href="https://discuss.pytorch.org/uploads/default/07f427ed447441c796cc00368ae2ec68bd0ceb69" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07f427ed447441c796cc00368ae2ec68bd0ceb69.png" alt="image" data-base62-sha1="18mrwRHgHMBFs54jkZA8gJPs7eV" width="690" height="143" data-dominant-color="282727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">877×183 16.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The test was added in <a href="https://github.com/pytorch/pytorch/commit/66ee4f1c8136">this commit</a> so you might want to update your code base if it’s not available.</p>
Large output differences when using torch.no_grad() with TransformerEncoder (FlashAttention?)

ekf: <p>I’m noticing large differences (elements differ by more than 1) in tensors produced by running a forward pass with a TransformerEncoder with and without torch.no_grad. I believe this is caused by one using FlashAttention and the other not.</p>
<p>Is this just a consequence of accumulated floating point differences? I don’t observe the behavior when no mask is provided (max abs difference is ~1e-6). Torch version is 2.6.0</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

torch.manual_seed(10)

SEQ_LENGTH=30
EMBEDDING_DIM=128
BATCH_SIZE=16

fake_tokens = torch.ones(BATCH_SIZE, SEQ_LENGTH, EMBEDDING_DIM, requires_grad=True)
fake_mask = torch.ones((BATCH_SIZE, SEQ_LENGTH), dtype=bool).cpu()
fake_mask[:, :-SEQ_LENGTH//4] = 0
encoder_layer = torch.nn.TransformerEncoderLayer(d_model=EMBEDDING_DIM, nhead=4, batch_first=True, dropout=0.0)
encoder = torch.nn.TransformerEncoder(encoder_layer, 8).eval()

fake_out = encoder(fake_tokens, src_key_padding_mask=fake_mask, is_causal=False)
with torch.no_grad():
    fake_out_no_grad = encoder(fake_tokens, src_key_padding_mask=fake_mask, is_causal=False)
    
print(torch.abs(fake_out - fake_out_no_grad).max())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor(2.8289, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
`compile` a function with `autograd`

madirbah: <p>Hi,<br>
I’m trying to speed up my code, which involves computing derivatives. e.g.</p>
<pre><code class="lang-auto">import torch
model = torch.nn.Sequential(
    torch.nn.Linear(1, 10),
    torch.nn.Tanh(),
    torch.nn.Linear(10, 1)
)

f = lambda x: (4* torch.pi**2)* torch.sin(2* torch.pi* x)
x = torch.linspace(0, 1, 10).reshape(-1, 1).requires_grad_()

def res(x):
    u = model(x)
    u_x = torch.autograd.grad(u, x, torch.ones_like(u), create_graph = True)[0]
    u_xx = torch.autograd.grad(u_x, x, torch.ones_like(u_x), create_graph = True)[0]
    return u_xx + f(x)
</code></pre>
<p>I expected <code>torch.compile</code> to speed up the computation, but when I add <code>@torch.compile</code> before <code>def res</code>, the function fails. I haven’t found examples of compiling functions that involve derivatives with <code>autograd</code>, so I’m unsure why this happens.</p>
<p>I did a small test using <code>torch.func</code> instead of <code>torch.autograd</code> for derivative calculations, and it seems to work. However, I suspect that using <code>autograd</code> directly might be faster, as it avoids multiple function calls (<code>grad grad</code>, <code>vmap</code>, and functional calls of the model).</p>
<p>Does anyone know if <code>torch.compile</code> can work efficiently with <code>autograd</code>. Or is <code>torch.func</code> the only viable approach?</p>
<p>Thanks,</p>

soulitzer: <p>Are you looking for something like <a href="https://pytorch.org/tutorials/intermediate/compiled_autograd_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Compiled Autograd: Capturing a larger backward graph for torch.compile — PyTorch Tutorials 2.6.0+cu124 documentation</a></p>
Do pre-trained model weights e.g. ResNet50 get updated?

td00: <p>My team uses the ResNet50 backbone for some of our models. Lately we’ve observed  small but consistent performance discrepancies between model versions trained a year ago and those we train now using the same backbone. As far as we can tell, all other elements of our training environment e.g. package versioning, training data, etc. are constant in our A/B tests. This leads me to wonder if the backbone model weights queried through the torch API have changed – hence my question. If they are not static, is there somewhere to view their version control history?</p>
<p>Any insight here would be greatly appreciated!</p>

Eduardo_Lawson: <p>You can use specific versions and the <code>DEFAULT</code> option that is updated with the best results. Here is some information <a href="https://pytorch.org/vision/0.20/models.html" rel="noopener nofollow ugc">https://pytorch.org/vision/0.20/models.html</a></p>
Issue with Sparse Tensor Matrix Multiplication and Broadcasting

ZhaoBo_Hu: <p>I’m encountering a <code>NotImplementedError</code> when trying to perform matrix multiplication with sparse COO tensors that involves broadcasting. Here’s a minimal reproducible example:</p>
<pre><code class="lang-auto">x = torch.matmul(torch.rand(327, 36).to_sparse_coo(), torch.rand(1, 36, 1))
torch.matmul(torch.rand(2000, 327).to_sparse_coo(), x)
</code></pre>
<p>This produces the following error:</p>
<pre><code class="lang-auto">NotImplementedError: Could not run 'aten::as_strided' with arguments from the 'SparseCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::as_strided' is only available for these backends: [CPU, CUDA, Meta, QuantizedCPU, QuantizedCUDA, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMeta, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].
</code></pre>
DataParallel with single GPU

emma_ng: <p>Hi,<br>
How exactly is DataParallel work in the case of single GPU?</p>
<p>I noticed that when I’m using a single GPU, I got different loss and accuracy if I wrap the model in DataParallel</p>
<pre><code class="lang-auto">model = torch.nn.DataParallel(model)
model = model.to("cuda:0")
</code></pre>
<p>compared to when I don’t use it</p>
<pre><code class="lang-auto">model = model.to("cuda:0")
</code></pre>
<p>I thought that if I’m using a single GPU, there should be no difference between using DataParallel or not. Do you know what can cause this difference?</p>
<p>Thank you.</p>

Yuyao_Huang: <p>Did you set a seed and enabled deterministic operations? If you don’t set a specific seed for your model, you would get different loss and accuracy between runs anyway.</p>

emma_ng: <p>Yes, I did. I can get the same loss and accuracy when rerun the same model (e.g. with DataParallel) but not between the 2 cases above</p>

William-HYWu: <p>Hi. Did you fix this issue? I noticed the same thing.</p>
Model output drastically different after upgrade

gnodab: <p>We are trying to upgrade our target system from Ubuntu 18.04 to Ubuntu 24.04. In this process, torch, torchvision, and torchaudio versions are upgraded, among other packages.</p>
<p>We are seeing mostly similar model output. But every once in a while, the label and confidences completely change. I’m not looking for a solution, but rather a place to start debugging. Is it possible that the change in torch.save/load methods could cause this? This was my first thought. And is my first path to debugging.</p>
<p>Are there other things that could contribute to such large changes in model outputs?<br>
Thank you for your time.</p>

gnodab: <p>I found that torchaudio.load() is returning different data on the same mp3 file. I’m going to cross post in the torchaudio section.</p>
<p>Edit: Found the issue. For future reference: <a href="https://discuss.pytorch.org/t/loading-mp3-file-using-torchload/133545" class="inline-onebox">Loading mp3 file using torchload</a></p>
PyTorch v.2.3.1 unable to configure

John_Doe4: <p>Hello,</p>
<p>Im trying to build pytorch v.2.3.1 with HIP. Im configure the project with:</p>
<pre><code class="lang-auto">cmake \
-DCMAKE_C_FLAGS:STRING="-O2 -fPIC" \
-DCMAKE_CXX_FLAGS:STRING="-O2 -fPIC" \
-DCMAKE_INSTALL_PREFIX="/usr" \
-DCMAKE_CXX_STANDARD=17 \
-DLIBSHM_INSTALL_LIB_SUBDIR="lib64" \
-DTORCH_INSTALL_LIB_DIR="lib64" \
-DPYTHON_EXECUTABLE=$(which python3) \
-DBUILD_TEST=OFF \
-DWITH_BLAS=open \
-DBUILD_PYTHON=ON \
-DBUILD_CUSTOM_PROTOBUF=OFF \
-DUSE_GOLD_LINKER=ON \
-DONNX_ML=OFF \
-DONNX_GEN_PB_TYPE_STUBS=OFF \
-DUSE_CUDA=OFF \
-DUSE_CUDNN=OFF \
-DUSE_FFMPEG=ON \
-DUSE_KINETO=OFF \
-DUSE_MAGMA=ON \
-DUSE_NNPACK=OFF \
-DUSE_OPENMP=OFF \
-DUSE_OPENCL=ON \
-DUSE_OPENCV=ON \
-DUSE_NUMA=ON \
-DUSE_NUMPY=OFF \
-DUSE_ROCM=ON \
-DUSE_ROCM_KERNEL_ASSERT=ON \
-Wno-dev \
-GNinja \
-DCMAKE_BUILD_TYPE=Release ..
</code></pre>
<p>And receive 4 error for now.</p>
<pre><code class="lang-auto">CMake Error: File /root/tour/pytorch/pytorch-v2.3.1/c10/hip/impl/hip_cmake_macros.h.in does not exist.
CMake Error at c10/hip/CMakeLists.txt:10 (configure_file):
  configure_file Problem configuring file


CMake Error at c10/hip/CMakeLists.txt:55 (add_subdirectory):
  add_subdirectory given source "test" which is not an existing directory.

...
HIP is using new type enums
CMake Error at aten/CMakeLists.txt:79 (add_subdirectory):
  add_subdirectory given source "src/THH" which is not an existing directory.


ROCm is enabled.
CMake Error: File /root/tour/pytorch/pytorch-v2.3.1/aten/src/ATen/hip/HIPConfig.h.in does not exist.
CMake Error at aten/src/ATen/CMakeLists.txt:45 (configure_file):
  configure_file Problem configuring file

-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "3.3.1")

</code></pre>
<p>Any one could help ?</p>
<p>Thanks</p>

fngarrett: <p>Are you still experiencing issues with this?</p>
<p>Can you verify that you are still getting this issue in newer versions of PyTorch? (This is example uses 2.3.1, latest stable is 2.6).</p>
<p>If you must use 2.3.1, can you verify that you have run <a href="https://github.com/pytorch/pytorch/blob/v2.3.1/tools/amd_build/build_amd.py" rel="noopener nofollow ugc"><code>amd_build.py</code></a>?</p>
How to train two independent networks

estebarb: <p>I’m trying to implement a simple char by char language model using Test Time Training. I started with the code from <a href="https://arxiv.org/abs/2407.04620" class="inline-onebox" rel="noopener nofollow ugc">[2407.04620] Learning to (Learn at Test Time): RNNs with Expressive Hidden States</a> .</p>
<p>The network is actually divided in two: 1) a memory side that is trained on a self supervised task, where the task is generated by two learned projections. 2) the language model.</p>
<p>I’m having problems implementing it, as pytorch mentions that the weights of (1) were changed. I’m not sure about what I should look as a possible fix:<br>
How I can instruct PyTorch to create two backpropagation graphs? Freeze weights and do several passes?</p>

ptrblck: <aside class="quote no-group" data-username="estebarb" data-post="1" data-topic="217349">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/estebarb/48/75332_2.png" class="avatar"> estebarb:</div>
<blockquote>
<p>I’m having problems implementing it, as pytorch mentions that the weights of (1) were changed.</p>
</blockquote>
</aside>
<p>Unsure if I understand the issue correctly, but I assume PyTorch fails during the backward pass claiming some parameters were already updated? Depending on the logic you want to apply to train your model you might need to <code>detach</code> the forward pass if you don’t want to backpropagate through the first part of your model anymore.</p>

J_Johnson: <p>DCGAN might align with some of what you’re attempting to do and could be worth a look.</p>
<p>With a DCGAN, you have two networks, a discriminator and a generator. It may help motivate how to properly handle the detach in your training pipeline. Here is a tutorial:</p>
<p><a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a></p>
GPU memory can't be freed by launching code from VS Code

Vineyo: <p>The symptom is like this: The training starts, and first several iterations run well, but the GPU memory increase linearly, and finally I get out of resource error. And any memory release technics won’t work, <code>del</code>, <code>detach()</code>, <code>empty_cache()</code>, etc. I thought it was because I was using Intel’s GPU, which might had not adapted to PyTorch properly. Finally, I try to run it from PowerShell instead of VS Code, and everything run well!</p>
Rust lang for runtime

yukino: <p>Has the Executorch team ever considered using the Rust programming language for the runtime before? C++ was chosen because it is friendly for embedded systems, but I think Rust might also be a good choice for the same reason.</p>
<p>I would be interested to hear the reasoning behind the decision!</p>

dbort: <blockquote>
<p>C++ was chosen because it is friendly for embedded systems</p>
</blockquote>
<p>This is the biggest reason: C++ has the most toolchain support across embedded architectures, and is also viable for mobile/desktop use. But also, ExecuTorch interacts with existing C++ types and libraries provided by the core PyTorch project.</p>
<p>I think that it’d be interesting to write the core runtime in Rust, especially for safer .pte file parsing. But the core runtime is only a small piece: there are hundreds of C++ kernel implementations, as well as a growing number of C++ backend implementations. I don’t expect that we’d replace all of those. And Rust today can’t call C++ (because of C++'s poorly-defined ABI), so we’d need some kind of Rust-to-C-to-C++ wrapper layer in there.</p>
<p>If someone wanted to play around with Rust in executorch, I think that starting with some kernels would be a good experiment. Kernel implementations might benefit from Rust’s macros, and would definitely benefit from its memory safety. The core runtime loop expects to call a C++ function pointer, but that function could point to a C++ wrapper that then calls into Rust. Some of the biggest work would be to write adapters for the ExecuTorch EValue types (like Tensor), since the kernel code would need to read and write those types.</p>

barakugav: <p>If anyone is interested in using Executorch from Rust, I wrote a bindings crate for it, implemented the mentioned Rust-to-C-to-C++ layer.<br>
<a href="https://github.com/barakugav/executorch-rs" rel="noopener nofollow ugc">https://github.com/barakugav/executorch-rs</a></p>
Loss at step between epochs

davide: <p>I’m training a LSTM model but when I plot the loss and accuracy they have this recurring pattern and I don’t know how to solve it.</p>
<p>The way I save the value is by:</p>
<pre><code class="lang-auto">train_losses.append((train_running_loss*x.shape[0])/len(data))
train_acc.append(train_running_acc/((i+1)*batch_size*window_size))
</code></pre>
<p>And I set the running loss/acc at 0 at the start of each epoch.<br>
But, this is what I got</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b790e30a1738671642d40cbbe35f15a1b90d31b.png" data-download-href="https://discuss.pytorch.org/uploads/default/8b790e30a1738671642d40cbbe35f15a1b90d31b" title="Screenshot From 2025-03-04 01-38-33"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b790e30a1738671642d40cbbe35f15a1b90d31b_2_690x199.png" alt="Screenshot From 2025-03-04 01-38-33" data-base62-sha1="jTPJaaZZIzXyERl7NndZ2mIY3ZF" width="690" height="199" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b790e30a1738671642d40cbbe35f15a1b90d31b_2_690x199.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b790e30a1738671642d40cbbe35f15a1b90d31b_2_1035x298.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b790e30a1738671642d40cbbe35f15a1b90d31b.png 2x" data-dominant-color="F7F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot From 2025-03-04 01-38-33</span><span class="informations">1222×354 39.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

J_Johnson: <p>The charts appear to describe the LSTM accruing loss between time steps in the series, and giving a more accurate prediction after seeing more of the time series(higher accuracy). And it appears like you’re starting again from the first time step after about 6,000 steps.</p>
What is the correct way to implement a transformer based vision classifier with nn.TransformerEncoder?

JimW: <p>I am trying to implement a transformer based temporal sequence classifier.<br>
For each time step, the we have an input tensor of 1x32. So if the classification will be based on 25 consecutive time steps, the input shape will be 1x32x25.</p>
<p>So I implemented a transformer encoder like this:</p>
<pre><code class="lang-auto">encoder_layer = nn.TransformerEncoderLayer(d_model=25, nhead=5)
transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
embed = transformer_encoder(torch.rand(1,32, 25))
</code></pre>
<p>Since the output of the generated embedding also involves 25 time steps. How should I consume this embedding?  Should I simply grab the last element of the embedding (i.e.  embed[:,:,-1]) and pass it to a simple linear layer to generate classes?  I thought the vector at any time step in the embed should contain all the information of this sequence of 25 time steps. Not sure if my understanding is correct.</p>
<p>Or I need a way to aggregate the</p>