akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I resolve this build issue with libtorch

michaeleisel: <p>I’ve run <code>USE_CUDA=ON BUILD_SHARED_LIBS=OFF python setup.py build</code>, and also tried <code>make local</code>, but when it gets to building Caffe2, I get:</p>
<pre><code class="lang-auto">[1/625] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim160_fp16_sm80.cu.o
FAILED: caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim160_fp16_sm80.cu.o 
/usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler -DAT_PER_OPERATOR_HEADERS -DCPUINFO_SUPPORTED_PLATFORM=1 ...
</code></pre>
<p>The command prints out “nvcc fatal   : Could not open output file caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim128_bf16_sm80.cu.o.d”, and in fact caffe2/CMakeFiles doesn’t exist at all.</p>

wfoy: <p>Came across the same error, found a fix here: <a href="https://github.com/pytorch/pytorch/issues/111526" class="inline-onebox" rel="noopener nofollow ugc">Build consumes 32GB RAM triggers OOM crash when compiling CUDA object · Issue #111526 · pytorch/pytorch · GitHub</a></p>
Unexpected hook behavior with 3D tensor and inplace operation

nvmdava: <p>Somehow when you combine both 3D input and inplace operation on the tensor, the backward hook seems to not trigger. I fail to understand the reason why this is happening, especially since it seems that 2D input seems okay.</p>
<pre><code class="lang-auto">import torch
h = 4
layer = torch.nn.Linear(h, h)
 
def CaseNot3D():
    inp = torch.randn((4, 4))
    out = layer(inp)
    out.register_hook(lambda grad_output: print('Not 3D Case'))
    out += inp
    out = out.sum()
    out.backward()
 
def CaseNotInplace():
    inp = torch.randn((4, 4, 4))
    out = layer(inp)
    out.register_hook(lambda grad_output: print('Not Inplace Case'))
    out = out + inp
    out = out.sum()
    out.backward()
 
def CaseBroken():
    inp = torch.randn((4, 4, 4))
    out = layer(inp)
    out.register_hook(lambda grad_output: print('Broken Case'))
    out += inp
    out = out.sum()
    out.backward()
 
 
CaseNot3D()
#PRINTS "Not 3D Case"
CaseNotInplace()
#PRINTS "Not Inplace Case"
CaseBroken()
#DOES NOT PRINT "Broken Case"
</code></pre>

ptrblck: <p>I haven’t seen use cases where <code>.register_hook</code> is called on a non-leaf variable, so unsure if that’s supported. <a class="mention" href="/u/alband">@albanD</a> would know and can correct me.</p>

albanD: <p>It is ~expected I’m afraid.<br>
The output here is a view into the result of the linear op. So when inplace happens, this particular view is not used anymore to compute the final result. And so no gradient is computed and the hook is not fired.<br>
In the 2D case, there are no views, and so the output itself is in the graph and used. So the hook fires.<br>
I’m not sure what’s the best move here tbh, could you open an issue on GitHub about this?</p>
Conversion of 2d and 4d vectors in c++ to torch tensor doesn't have the right data

Saba1: <p>I’m trying to convert these two vectors to torch tensor but the output doesn’t have the right data:</p>
<pre><code class="lang-auto">auto input_options = torch::TensorOptions().dtype(torch::kInt32);
auto inputs = torch::from_blob(vec.data(), {1, 10, 194, 144}, input_options);
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">..Columns 13 to 18 -8.0412e+08  9.1974e+08  4.9000e+01  0.0000e+00  1.6578e+09  2.1909e+04
  1.6578e+09  2.1909e+04  1.6581e+09  2.1909e+04  1.6559e+09  2.1909e+04
  1.8698e+09  1.9702e+09  7.5708e+08  1.7000e+09  1.9199e+09  0.0000e+00
  1.9704e+09  1.8524e+09  1.7017e+09  5.3978e+08  1.9366e+09  5.4103e+08..

</code></pre>
<p>also</p>
<pre><code class="lang-auto">std::vector&lt;std::vector &lt;int&gt;&gt; indx1={{126, 22}, {166, 102}, {46, 46}, {166, 54} };
auto indx_options = torch::TensorOptions().dtype(torch::kInt32);
 auto indx = torch::from_blob(indx1.data(), { indx1.size(),2}, indx_options);
</code></pre>
<p>output:</p>
<pre><code class="lang-auto"> 1.1586e+09  2.2036e+04
 1.1586e+09  2.2036e+04
 1.1586e+09  2.2036e+04
 1.1586e+09  2.2036e+04
[ CPUIntType{4,2} ]

</code></pre>

ptrblck: <p>Are the source arrays going out of scope? If so, <code>.clone()</code> the tensor before the underlying data is freed.</p>

Saba1: <p>Thanks for your respond. I added <code>.clone()</code> but still got the same result.<br>
Using arrays instead of vectors has solved the problem.</p>
"Illegal instruction (core dumped)" when importing PyTorch

sphynx: <p>When I try to import pytorch, I keep getting “Illegal instruction (core dumped)”. I’ve tried CUDA 12.1, Ubuntu 22.04 and different installation methods.</p>
<p>Current setup:<br>
Ubuntu 20.04<br>
Nvidia RTX 3060<br>
CUDA 11.8<br>
Python 3.8.10</p>
<p>On this forum I found that the output of gdb might be useful, which is:</p>
<pre><code class="lang-auto">(gdb) run test.py
Starting program: /home/sphynx/Documents/test/.venv/bin/python3 test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 13834]

Program received signal SIGILL, Illegal instruction.
0x00007fffef02c732 in void std::vector&lt;dnnl_post_ops::entry_t, std::allocator&lt;dnnl_post_ops::entry_t&gt; &gt;::_M_realloc_insert&lt;&gt;(__gnu_cxx::__normal_iterator&lt;dnnl_post_ops::entry_t*, std::vector&lt;dnnl_post_ops::entry_t, std::allocator&lt;dnnl_post_ops::entry_t&gt; &gt; &gt;) () from /home/sphynx/Documents/test/.venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
</code></pre>
<p>Any help is appreciated.</p>

sphynx: <p>For anyone encountering this in the future. For me, installing PyTorch with</p>
<pre><code class="lang-auto">pip3 install https://download.pytorch.org/whl/cu118/torch-2.0.0%2Bcu118-cp38-cp38-linux_x86_64.whl
</code></pre>
<p>instead of the command mentioned on the <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">PyTorch website (Start Locally)</a> solved the problem.</p>

ptrblck: <p><code>dnnl</code> points to Intel’s oneDNN lib and does not seem to be supported on your CPU. Downgrading PyTorch might be a workaround but not a solution. Which CPU are you using?</p>

sphynx: <p>Thanks for your reply. I am using an AMD Phenom II X4 965.</p>

ptrblck: <p>I don’t believe this CPU supports AVX, while it might be a requirement.<br>
CC <a class="mention" href="/u/malfet">@malfet</a> do you know if and which AVX requirement the current binaries have?</p>
How does dropout2d works in CNN?

vivekstorm: <p>Hi Community,<br>
I would like to know how does dropout2d works? Does the weights are made to zero or output of activation of previous layer is made to zero in dropout?</p>

ptrblck: <p>Dropout layers masks the activations, not the weights or a layer:</p>
<pre data-code-wrap="python"><code class="lang-python">drop = nn.Dropout2d()
x = torch.randn(1, 3, 4, 4)
out = drop(x)
print(out)
# tensor([[[[-1.2433, -2.1234, -1.8535,  0.1276],
#           [ 5.8782, -2.1130,  1.8116,  1.3773],
#           [-2.0260, -1.5322,  1.3397,  0.8360],
#           [ 0.8796, -0.7228, -0.7606, -4.3226]],

#          [[ 0.0000, -0.0000, -0.0000,  0.0000],
#           [-0.0000,  0.0000, -0.0000,  0.0000],
#           [-0.0000,  0.0000,  0.0000,  0.0000],
#           [-0.0000,  0.0000,  0.0000, -0.0000]],

#          [[-0.0000,  0.0000,  0.0000, -0.0000],
#           [ 0.0000,  0.0000,  0.0000, -0.0000],
#           [ 0.0000,  0.0000, -0.0000, -0.0000],
#           [ 0.0000, -0.0000,  0.0000, -0.0000]]]])
</code></pre>
ReduceLROnPlateau is not decreasing the learing rate

Vaibhav_Chahar: <p>Hi,<br>
In my code, I want to reduce the learning rate when my monitor matrix (loss_mean) attained minimum values but ReduceLROnPlateau is not decreasing the learning rate, it is constant throughout the training.<br>
the below code is showing how I realize ReduceLROnPlateau in my code. The value of my monitor tensor, loss_mean are:<br>
-193.92658996582<br>
-193.953338623047<br>
-193.979461669922<br>
-193.974502563477<br>
-193.963928222656<br>
-193.976669311523<br>
-193.98210144043<br>
-193.942535400391<br>
-193.969467163086<br>
-193.977722167969<br>
-193.976165771484<br>
-193.969940185547<br>
-193.953491210938<br>
-193.986679077148<br>
-193.95051574707<br>
-193.958099365234<br>
-193.962783813477<br>
-193.970840454102<br>
-193.988967895508<br>
-193.992889404297<br>
(the last few epochs)</p>
<p>scheduler={<br>
“scheduler”: ReduceLROnPlateau(<br>
optimizer=optimizer,mode=“min” ,factor=0.92, patience=50, threshold=1e-2 ,min_lr=1e-6),“monitor”:“loss_mean”}.</p>
<p>Please have a look and help me.</p>

ptrblck: <p>The <code>patience</code> value is set to 50 steps which seems to be high considering:</p>
<blockquote>
<p>patience (int) – Number of epochs with no improvement after which learning rate will be reduced. For example, if patience = 2, then we will ignore the first 2 epochs with no improvement, and will only decrease the LR after the 3rd epoch if the loss still hasn’t improved then. Default: 10.</p>
</blockquote>
<p>By default <code>threshold_mode</code> will also use a relative comparison and from the few values you have posted the loss decreases after a few steps again, so you might want to adapt the settings a bit.</p>

Vaibhav_Chahar: <p>Hi,<br>
I reduces the patience to 5 and learning rate starts decreasing.Can you tell me what does the threshold and threshold_mode does with the example (using the monitor metric that i mentioned) ? I have read the docs and all the available materials but could not understand it. Please explain me these If possible simply.</p>
Mask BOS token for GPT-2

TeDataPro: <p>Hello,</p>
<p>I am trying to fine-tune a GPT-2 model on next token prediction task.</p>
<p>I manually add BOS and EOS tokens for each sentence.</p>
<p>However, I don’t know if I have to mask these tokens or not during training ? I was thinking about masking them (at least EOS for sure). But I have a doubt for the BOS. Indeed, during inference, the BOS will be the first token used to predict next token based on its embedding.</p>
<p>So should I mask it or not ?</p>
<p>Thank you !</p>
Cannot use Blas=Eigen when build pytorch

ikorifey2002: <p>I try to build PyTorch from sources</p>
<p>I have sucsessfully built it using official instriction,<br>
But now I’d like to build it using Eigen as a Blas implementation.</p>
<p>I run:<br>
BLAS=Eigen python setup.py develop</p>
<p>And in build logs I could see:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/9552c3f1bf3d286d20ea54ba0b339adbf489db46.png" alt="image" data-base62-sha1="liYsXMqDhaU4mGlODvhbkozfdoa" width="595" height="256"></p>
<p>It tried to find Eigen but couldn’t.</p>
<p>Which more variables schould I set, or how can I install Eigen to make my pytorch notice it.</p>
<p>Linux Rocky 8.8<br>
using conda env - python 3.10<br>
I made conda install eigen (but doubt that it was enough)</p>
Bert attention mask for pytorch dot product

Skyrax: <p>Hello, how can i pass bert attention mask to pytorch scaled dot product attention. For example, i have qkv shape (32,8,64,128) and my mask (32, 64). Is it correct to use unsqueeze and expand for changing shape?<br>
mask = mask.unsqueeze(1).expand(-1, 64,-1)<br>
mask = mask.unsqueeze(1).expand(-1,8,-1,-1)</p>
Shared data pool with DDP

CDhere: <p>Hi, I’m writing an algorithm where the data pool is affected by the outcome of the trained model at each epoch. This means that while I can use DDP to create copies of the model on GPUs, the data pool where the training samples are drawn from should be shared among all processes. Is there a good way to achieve this? How do I collect results from multiple processes to update the shared pool, before continuing to the next training epoch?</p>
<p>Many thanks!</p>

rvarm1: <p>Hi, if you’re within a single node you can probably coordinate these updates with something like a <code>multiprocessing.Manager</code>, see <a href="https://docs.python.org/3/library/multiprocessing.html" class="inline-onebox" rel="noopener nofollow ugc">multiprocessing — Process-based parallelism — Python 3.9.1 documentation</a>.</p>
<p>Alternatively, if your training is across different nodes entirely, you have a few options:</p>
<ol>
<li>Have a per-node data pool, and just proceed with the above approach</li>
<li>Have a global data pool that is replicated across each node with the MP manager. You can then probably use pytorch APIs such as <code>dist.broadcast_object_list</code> and <code>dist.scatter_object_list</code> to share the required data.</li>
</ol>

CDhere: <p>Thank you for the suggestion! So far I’m satisfied with single-machine, multi-gpu setting. I did a bit of research and here’s a good post on how to use <code>mp.Manager</code>: <a href="https://stackoverflow.com/questions/10415028/how-can-i-recover-the-return-value-of-a-function-passed-to-multiprocessing-proce" rel="noopener nofollow ugc">https://stackoverflow.com/questions/10415028/how-can-i-recover-the-return-value-of-a-function-passed-to-multiprocessing-proce</a>, in case anyone else is interested. BTW I should point out that <code>torch.multiprocessing</code> is just a thin layer of wrapping around Python’s native <code>multiprocessing</code>, so most stuff are similar and can be used directly in the same style.</p>
<p>Basically a dict can be shared among processes spawned, which suits my case. I guess any pickable primitives like List will also do, though.</p>
<p>Will be back and update this thread once I’ve extended to a multi-machine setting.</p>

mrshenli: <aside class="quote no-group" data-username="CDhere" data-post="3" data-topic="109963">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cdhere/48/29866_2.png" class="avatar"> CDhere:</div>
<blockquote>
<p>Will be back and update this thread once I’ve extended to a multi-machine setting.</p>
</blockquote>
</aside>
<p>For multiple machine data sharing, one option would be letting one process to serve as the data store, and use <code>torch.distributed.rpc</code> to push/pull data across processes.</p>

External_happy: <p>Can you give more details on how you did that ?</p>
Decomposition error with `scaled dot product flash attention`

jasmine_n: <p>When I tried decompose <code>aten._scaled_dot_product_flash_attention.default</code> with my own backend using <code>get_decompositions([aten._scaled_dot_product_flash_attention.default])</code>, I get the following error</p>
<pre><code class="lang-auto">File "torch/_functorch/_aot_autograd/functional_utils.py", line 335, in assert_functional_graph
   assert (
torch._dynamo.exc.BackendCompilerFailed: backend='mybackend' raised:
AssertionError: aot_autograd expected to have an entirely functional graph, but found %masked_fill_ : [num_users=1] = call_function[target=torch.ops.aten.masked_fill_.Scalar](args = (%zeros_like, %logical_not, -inf), kwargs = {})
</code></pre>
<p>How can I decompose <code>aten._scaled_dot_product_flash_attention.default</code>? This is with torch 2.2.0.</p>
PyTorch Installing Error on Fedora

inopy: <p>Fedora Workstation 39, Python 3.12.0, PIP 23.3.1<br>
I don’t have PyTorch installed, it gives this error:</p>
<pre><code class="lang-auto">python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://download.pytorch.org/whl/cpu
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch
</code></pre>
<p>Help me solve it, please.</p>

ptrblck: <p>Downgrade Python to 3.10 or 3.11 and it should work.</p>

Z4CK: <p>How to downgrade cos I tried several times which did’t work</p>

ptrblck: <p>I would recommend asking in a Fedora-specific discussion board if you have trouble installing a defined Python version.</p>
C++ frontend examples on training with customized dataset

abarunprasath22: <p>Dear Pytorch users,<br>
I am coming up with a pytorch enabled surrogate reference for my so-called multi-scale simulation framework, that inherits the material properties from it’s micro-structure considerations. The idea here is to use my macro-model in conjunction with the surrogate model that effectively substitutes the finer scale material constitutive behavior. With a careful selection of the training dataset in hand, i am looking at options in pytorch based c++ frontend to train the surrogates with this dataset and reference them with the target variables the macro model is throwing out.</p>
<p>The training dataset has the following simple structure that comprises the deformation gradient and stresses. You might find several of these lines for many different simulations that constitute this dataset.<br>
F1 F2 F3 F4 F5 F6 F7 F8 F9  S1 S2 S3 S4 S5 S6 S7 S8 S9</p>
<p>The macro-model will examine the trained set for the macro-model deformation gradients of this type and returns the stresses<br>
F1 F2 F3 F4 F5 F6 F7 F8 F9</p>
<p>We made some progress on this imposing the dataset on a NN model creating a torch script module following this approach <a href="https://pytorch.org/tutorials/advanced/cpp_export.html#step-3-loading-your-script-module-in-c" rel="noopener nofollow ugc">Torch Script Module</a>. Later this turned out to be an incorrect selection, as the length of the training dataset and evaluation set is required to be identical, without which the approach will be infeasible.</p>
<p>The pytorch examples <a href="https://github.com/pytorch/examples/tree/main/cpp" rel="noopener nofollow ugc">here</a> has references to some good examples on usage of NN models within an optimization setting, but has limited focus on creating the kind of datasets to be used within this framework. The simplest way one can compile this dataset is to put all these 18 numbers in a sequence of vectors in a CSV file and convert them to the data structure the data loader demands. The better and the recommended way is to use a MNIST dataset format which has these numbers encoded in a hand written format that has a natural connection with these examples. Given these I am not very clear on the following 1) How good is the selection of MNIST dataset, that has a better representation of the training dataset mentioned above 2) If all seems well, how this dataset can be pushed in to a MNIST dataset (any references to examples are appreciated).</p>
<p>Thanks a lot for looking at this case. Any suggestions/leads on handling this issue will be much appreciated.<br>
Thanks and regards,<br>
Arun</p>
Gradient Penalty for GANs in pytorch

dlsf: <p>I am trying to implement this tensorflow code from a method which implements gradient penalty:</p>
<pre><code class="lang-auto"># gradient penalty
real_grads = tf.gradients(tf.reduce_sum(real_output), source)[0]
gp = tf.reduce_sum(tf.square(real_grads), axis=[1, 2, 3])
gp_loss = tf.cast(tf.reduce_mean(gp * (10 * 0.5)), tf.float32)
</code></pre>
<p>A solution for this in pytorch or a pointer to the solution would be greatly appreciated, thx.</p>

ptrblck: <p>You can compute the gradients via e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">grads = torch.autograd.grad(out, model.parameters())
</code></pre>
<p>and could then compute the penalty:</p>
<pre data-code-wrap="python"><code class="lang-python">p = (grads.sum([1,2,3]) * 5).mean()
</code></pre>

dlsf: <p>Thanks a lot for your input. In the end I managed to implement their gradient penalty and resolve the issues related to torch.autograd.grad() with the following piece of code:</p>
<pre><code class="lang-auto"># gradient penalty
grads = torch.autograd.grad(real_score.sum(), s, create_graph=True)[0]
gradient_penalty = torch.mean((grads**2).sum([1,2,3]))
</code></pre>
Batched cross-correlation

Hadar_Sharvit: <p>so i have a tensor (a signal) <code>f</code> with shape <code>(B,T,1)</code> and another signal <code>g</code> with the same shape.<br>
i want to perform “pairwise” cross correlation between samples with the same batch index. Namely, if I were to iterate the samples in the batch, I’d perform something like this:</p>
<pre><code class="lang-auto">all_xcorrs = []
for b in range(B): # where B is the batch size
    f_b = f[[b]].permute(0,2,1) # shape (1,1,T)
    g_b = g[[b]].permute(0,2,1) # shape (1,1,T)
    xcorr_b = F.conv1d(f_b,g_b, padding='same') # shape (1,1,T)
    all_xcorrs.append(xcorr_b)
all_xcorrs=torch.cat(all_xcorrs,axis=0) # should have shape (B,1,T)
all_xcorrs = all_xcorrs.permute(0,2,1) # (B,T,1)
</code></pre>
<p>how can I vectorize this process?<br>
By the way - a workaround may be to calculate conv1d on the entire transposed tensors (which is similar to a nested loop <code>for i in [B] for j in [B]: conv1d(f_i,f_j)</code> ) and extracting the diagonal only:</p>
<pre><code class="lang-auto">f = f.transpose(1, 2)  # shape (B,1,T)
g = g.transpose(1, 2)  # shape (B,1,T)
all_xcorrs = F.conv1d(f, g, padding='same')
diag_indices = torch.arange(all_xcorrs.shape[0])
xcorrs = all_xcorrs[diag_indices, diag_indices, :].unsqueeze(-1)
</code></pre>
Batch with Basic RNN

astra1234567: <p>Hi everyone ! Can someone explain me how use batch with RNN ?<br>
It is basic model from documentation:</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

</code></pre>
<p>train function:</p>
<pre><code class="lang-auto">def train():
    hidden = model.initHidden()
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        data = data.to(device)
        target = target.to(device)


        loss = F.nll_loss(output, target)
        
        loss.backward()
        optimizer.step()
</code></pre>
<p>What should I do to send batch (for example [4,2,5] ) to model and how make correct training whit this?</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html">This RNN tutorial</a> might be a good starter as it walks you through a classification use case.</p>

erkiserk: <p>This doesn’t show how to batch!!</p>
Passing custom tensor using __torch_dispatch__ to nn.Parameter

tohtana: <p>Hi, I’m trying tensor subclassing examples in <a href="https://github.com/albanD/subclass_zoo/blob/main/empty_tensor.py" rel="noopener nofollow ugc">subclass_zoo/empty_tensor.py at main · albanD/subclass_zoo (github.com)</a>. The custom tensor class <code>EmptyTensor</code> does not have an actual tensor storage but returns dummy data on the fly. The behavior is implemented by <code>__torch_dispatch__</code>.</p>
<p>The custom tensor class behaves as expected. However, I am confused with a behavior when it is passed to <code>torch.nn.Parameter</code>.</p>
<pre><code>rt = torch.randn(2, 2)
et = EmptyTensor(rt)
empty_weight = torch.nn.Parameter(et)
print(f"empty_weight={empty_weight.__class__} param?={isinstance(empty_weight, torch.nn.Parameter)}")
</code></pre>
<p>This displays:</p>
<pre><code>empty_weight=&lt;class 'empty_tensor.EmptyTensor'&gt; param?=True
</code></pre>
<p>The expected class of <code>empty_weight</code> is <code>torch.nn.Parameter</code>. In addition, <code>isinstance</code> says it is an instance of <code>torch.nn.Parameter</code> though <code>EmptyTensor</code> is not a subclass of <code>torch.nn.Parameter</code> (it is a subclass of <code>Tensor</code>.</p>
<p>You can do the same with a normal tensor.</p>
<pre><code>rt = torch.randn(2, 2)
real_weight = torch.nn.Parameter(rt)
print(f"real_weight={real_weight.__class__}")
</code></pre>
<p>As expected, the result is</p>
<pre><code>real_weight=&lt;class 'torch.nn.parameter.Parameter'&gt;
</code></pre>
<p>Can anyone help me understand what is going here?</p>
Behavior of Decoder Transformers

jupyter: <p>Hi,</p>
<p>I’m using pretrained mT5 to do semantic parsing task. I dont know how the model can predict unseen token?</p>
<p>As I implemented RNN-based seq2seq, the model cannot predict unseen token (Though these token appear in the input, then the model have to learn this partern and copy to the output). the final layer is classification among all the tokens in the vocabulary. The model will be bias to all the token classes it have learned so it cannot predict unseen token.</p>
<p>However I have tried with pretrained transformers. And It worked? I dont understand this point. Like the last layer is classification layer right? Then why the model can predict the class It have never met before?</p>
<p>Thank you</p>

J_Johnson: <p>In transformer training for next token prediction, generally speaking, a triangular mask gets applied on the input and so every token in the input, except the first, gets trained on.</p>
<p>For instance, if we fed a model, “The fox jumped over the lazy dog.” It would get trained something like:</p>
<p>Input: The _ _ _ _ _ _. Target: fox<br>
Input: The fox _ _ _ _ _. Target: jumped<br>
…</p>
<p>And so on.</p>

jupyter: <p>Hi I understood how it was training</p>
<p>But in my downstream task. Its even copy some meaningless name from the input. For example:</p>
<p>I do NER problem just to detect entities without classify its type:<br>
A conference abc was held on 12-9-2000<br>
 → model’s output: conference abc | 12-9-2000. Which abc is meaningless token and unseen token (from the downstream task data), so the model somehow can learn and predict it via classification layer? the amazing thing is the class of token abc wasn’t trained.</p>
<p>I also trained a  RNN Sequence to Sequence with Attention, it’s not capable of doing that. My thought is that the embedding vectors of unseen token (in the test set) wasn’t trained not like pretrained model. So the model treats every token which hasn’t been learned the same (pytorch initialized the weight of embedding by normal distribution as I remember)  so it doesn’t know how to copy them.</p>
How can I see the source of guards?

dan-garvey: <p>How can I see the source of guards?<br>
I get stuff like:</p>
<pre><code class="lang-auto">[2024-02-09 20:55:17,752] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-02-09 20:55:17,780] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s0 = 2 for L['arg1_1'].size()[1] [2, 15] (_dynamo/variables/builder.py:1809 in &lt;lambda&gt;)                             
[2024-02-09 20:55:17,783] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s1 = 2 for L['arg2_1'].size()[1] [2, 32766] (_dynamo/variables/builder.py:1809 in &lt;lambda&gt;)                          
[2024-02-09 20:55:17,786] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s2 = 2 for L['arg3_1'].size()[1] [2, 32766] (_dynamo/variables/builder.py:1809 in &lt;lambda&gt;)                         
 [2024-02-09 20:55:17,790] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s3 = 2 for L['arg4_1'].size()[1] [2, 32766] (_dynamo/variables/builder.py:1809 in &lt;lambda&gt;)                          
[2024-
</code></pre>
<p>or something like:</p>
<pre><code class="lang-auto">[2024-02-09 20:54:31,191] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-02-09 20:54:31,215] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s0 = 2 for L['arg0_1'].size()[1] [2, 16] (_dynamo/variables/builder.py:1809 in &lt;lambda&gt;)                             
[2024-02-09 20:55:04,670] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards

</code></pre>
<p>but the source listed in parentheses is the dynamo code that generated the guard not the model code.</p>
How does the stride and kernel size is defined for ADAPTIVEAVGPOOL1D

seyeeet: <p>I am wondering how the stride and kernel sizes are decided for adaptiveavgpool-1d?<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool1d.html#adaptiveavgpool1d" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool1d.html#adaptiveavgpool1d</a></p>
Implement custom LayerNormalization layer for channel-wise normalization

H_ng_Le: <p>Hi, currently pytorch supports <a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html" rel="noopener nofollow ugc">LayerNorm</a> operation with <code>normalized_shape</code> in the form <code>[∗×normalized_shape[0]×normalized_shape[1]×…×normalized_shape[−1]]</code>. However, I want to implement a custom LayerNorm operations that only normalize across channels. For examples, given input size <code>[N, C, W, H]</code>, the custom layer would normalize across <code>C</code>, but not <code>W</code> or <code>H</code>. I tried to look for the source code of <code>LayerNorm</code> implementation and then tried to modify it but couldn’t find the source code. Can anybody point me to the location of the source code? Also, I’m looking at this tutorial <a href="https://pytorch.org/docs/master/notes/extending.html" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch master documentation</a> to try to write the custom layer. It’d also be great if someone can show me necessary steps to build the described custom layer, or some relevant tutorials. Thank you very much!</p>

ccaven: <p>You can use <code>permute</code> to apply LayerNorm to any dimensions you want. For your image example, this should do the trick:</p>
<pre data-code-wrap="py"><code class="lang-py">from torch.nn.functional import layer_norm

img = torch.rand((1, 3, 256, 256))
# Send the channel axis to the end
channels_last = img.permute(0, 2, 3, 1)
# Apply LayerNorm
normed = layer_norm(channels_first, normalized_shape=[3])
# Put channel axis back
normed_img = normed.permute(0, 3, 1, 2)
</code></pre>
Loss.backward() error when using custom autograd function

torch2.0: <p>Hi All!</p>
<p>I am facing the error:<br>
AttributeError: ‘CheckpointFunctionBackward’ object has no attribute ‘input_tensors’</p>
<p>Please find the implementation of my custom backward function.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b86d6164e273c4ea5402407896edece44c1f8ec9.png" data-download-href="https://discuss.pytorch.org/uploads/default/b86d6164e273c4ea5402407896edece44c1f8ec9" title="pytorch_community_doubt"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b86d6164e273c4ea5402407896edece44c1f8ec9.png" alt="pytorch_community_doubt" data-base62-sha1="qjwcsYyXt3ttNyLfIRW5a9H589j" width="527" height="500" data-dominant-color="15181F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_community_doubt</span><span class="informations">670×635 26.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/torch2.0">@torch2.0</a>,</p>
<p>When sharing code, please copy and paste the code and wrap it within 3 backticks ```, rather than sharing a screenshot.</p>
<p>Your issue is that you haven’t saved the <code>inputs_tensors</code> object within the forward pass with <code>ctx.save_for_backward(inputs_tensors)</code>, that’s why it doesn’t exist within the backward pass.</p>
<p>Also, I don’t believe you can invoke a <code>torch.autograd.grad</code> call within the <code>backward</code> method as it’s ran within a <code>torch.no_grad()</code> decorator (even if you enable gradient tracking via <code>torch.enable_grad()</code> yourself). You need to manually define the formula for the backward method yourself.</p>

torch2.0: <p>Hi <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>,</p>
<pre><code>                                      Thanks for your suggestion. I will make the corrections.
</code></pre>
Compile source code, code navigation

AidanGoldfarb: <p>My short question is where can I find the implementation of some of the compiler features described <a href="https://pytorch.org/docs/stable/_modules/torch.html" rel="noopener nofollow ugc">here</a>. Specifically, how the different modes (default, max-autotune, reduce-overhead, max-autotune-no-cudagraph) affect the behavior of the compiler.</p>
<p>My broader, and perhaps more important question is if there are any tips for navigation a code base like this. In my experience, the documentation has not been super clear about where to find the implementations of these features. For example, I struggle to find where <a href="https://pytorch.org/docs/stable/_modules/torch.html" rel="noopener nofollow ugc">this</a> code can be found, and how it fits into the structure of the compiler. I would very much appreciate any tips for navigating a codebase such as this, without needed to email the specific developer with every question.</p>
<p>Thanks!</p>

marksaroufim: <p>When you <code>import torch</code> for the first time it’ll run an <code>__init__.py</code> which exposes a function called <code>compile()</code> which just passes those calls to <code> torch._dynamo.optimize()</code></p>
<p>When you pass in <code>compile(</code>mode=“some mode”)` the mode will set some configs here <a href="https://github.com/pytorch/pytorch/blob/main/torch/_inductor/__init__.py#L98" class="inline-onebox">pytorch/torch/_inductor/__init__.py at main · pytorch/pytorch · GitHub</a></p>
<p>So for example here we learn that <code>reduce-overhead</code> sets a config variable called <code>triton.cudagraphs</code></p>
<p>grep for <code>triton.cudagraphs</code> and you’ll find when it’s set and not in a file called <code>_inductor/compile_fx.py</code></p>
<p>Hard to give general tips but what’s worked for me is alternating between reading the code like a book and then writing toy examples and stepping through them with a debugger, fixing random bugs also really helps because you’re more likely to get the attention of a maintainer</p>

AidanGoldfarb: <p>I recently found <a href="https://pytorch.org/blog/pytorch-2-paper-tutorial/" rel="noopener nofollow ugc">this</a>, which has been incredibly helpful. Linked as I think it could save people a lot of time “reading the code like a book” <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Questions about using torch.Distributions to create a negative bionomialo mixture distribution

ChineseBest: <p>Hi, all, I intend to mix two Negative bionomial distirbution with each parameter size (n<em>m), actually there are two 2</em>n*m distributions in total. I intend to implement this function based on Categorical and MixtureSameFamily.</p>
<p>However, it seems that I cannot directly create this mixture distirbutions based on these two functions. My codes are:</p>
<pre><code class="lang-auto">    def forward(
            self, u: torch.Tensor, v: torch.Tensor,
            b: torch.Tensor, l: torch.Tensor # l is sequencing depth
    ) -&gt; NBMixture:
        scale = F.softplus(self.scale_lin[b])
        logit_mu1 = scale * (u @ v.t()) + self.bias1[b]
        logit_mu2 = scale * (u @ v.t()) + self.bias2[b]

        mu1 = F.softmax(logit_mu1, dim=1)
        mu2 = F.softmax(logit_mu2, dim=1)

        log_theta = self.log_theta[b]

        total_count = torch.stack([log_theta.exp(), log_theta.exp()])
        total_logits = torch.stack([(mu1*l + EPS).log() - log_theta, (mu2*l + EPS).log() - log_theta])
        # Create Negative Binomial distributions
        nb1 = NegativeBinomial(total_count, logits=total_logits)
        # Define mixture weights
        mixture_weights = torch.distributions.Bernoulli(mu1).sample()
        mixture_logits = torch.stack([torch.distributions.Bernoulli(mu1).log_prob(mixture_weights),
                                    torch.distributions.Bernoulli(mu1).log_prob(1 - mixture_weights)])
        # mixture_logits = mixture_logits.reshape(mixture_logits.shape[1], mixture_logits.shape[2], 2)
        log_prob_mix = D.Categorical(logits = mixture_logits)
        
        # log_prob_mix = D.Categorical(torch.FloatTensor([0.6,0.4]))
        # Create the MixtureSameFamily distribution
        mixture_distribution = MixtureSameFamily(log_prob_mix, nb1)

        return mixture_distribution
</code></pre>
<p>And I received diminsions not matched errors. Coud anyone please help me? Thanks.</p>
NaN loss after few epochs of fine tuning ViT B16 on smaller datasets

pytorchuser101: <p>I am fine-tuning a pretrained ViT on CIFAR100 (resizing to 224), the training starts out well with decreasing loss and decent accuracy. But then suddenly the loss goes to NaN with the accuracy equaling random guess.</p>
<p>The learning rate I used was 0.0001 and Adam optimizer. With a learning rate of 0.001 a similar issue occurs, but a few epochs earlier.</p>
<p>This is my code:</p>
<pre><code class="lang-auto">model = torchvision.models.vit_b_16(weights='IMAGENET1K_V1')
model.to(device) 

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)

for epoch in range(total_epochs):
    model.train()
    train_loss = 0
    correct = 0
    for i, (x, y) in enumerate(trainloader):
        optimizer.zero_grad()
        x = x.to(device)
        y = y.to(device)
        outputs = model(x)
        loss = criterion(outputs, y)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()
            
        _, predicted_train = outputs.max(1)
        correct += predicted_train.eq(y).sum().item()
</code></pre>
<p>When learning rate is 0.0001, NaN values occur after 25~ epochs.<br>
When learning rate is 0.001, NaN values occur after 5~ epochs.</p>

alepistola: <p>Without any additional information, my guess is that applying L2 regularization through weight decay is leading to a division by zero problem that results in a propagation of NaN values. Have you tried to change its value?</p>

pytorchuser101: <p>Are you saying the current weight decay value could be high, causing the weights towards zero and cause the division by zero error ?</p>

alepistola: <p>exactly what I meant</p>

pytorchuser101: <p>Even with gradient clipping it goes to NaN after a few epochs</p>

Bibhabasu_Mohapatra: <p>can you check if you are normalizing your images with Imagenet mean and variance? <a class="mention" href="/u/pytorchuser101">@pytorchuser101</a> plus it generally does not occur if your inputs are CIFAR100</p>

pytorchuser101: <p>Yes, I am normalizing train and test sets by ImageNet mean and standard deviation:<br>
mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)</p>
<p>I tried (0.5, 0.5, 0.5) for mean and std dev as well. It gives the same issue.</p>
<p>I am resizing CIFARs to 224x224</p>
What are some common datasets for nlp equivalent to mnist or cifar for vision

kirk86: <p>We have torchvision for vision datasets but is there an equivalent libray for nlp tasks?<br>
Something that contains datasets that vary in complexity from simpler to harder ones, like for instance in vision MNIST → CIFAR → ImageNet?</p>

JamesTrick: <p>Hi Kirk - taking a look at the datasets withing <a href="https://pytorch.org/text/stable/datasets.html" rel="noopener nofollow ugc">torchtext</a> woul give you a good indication!</p>
<p>In my mind, things like AG News or IMDB datasets would be fairly popular like the MNIST datasets. But it really depends on the task you’re trying to achieve.</p>
Natural Language to SQL query

Sai1: <p>Hi everyone,<br>
I am trying to understand the task of generating SQL queries from natural language. I’m trying to read papers but unable to understand as this is my first time reading papers. Can anyone suggest an easy to read paper on this topic or any resources ?<br>
Thanks in advance.</p>

JamesTrick: <p>Papers with code is a good resource for initial research of models, they have a <a href="https://paperswithcode.com/task/text-to-sql" rel="noopener nofollow ugc">Text to SQL section here</a>.</p>
<p><a href="https://huggingface.co/models?other=text-to-sql" rel="noopener nofollow ugc">Huggingface</a> is another good starting point, they have some of the more lastest model such as CodeLlama that you could explore more.</p>
How much VRAM needed for Llama 2 70B model?

JasonJiang: <p>Hi,</p>
<p>I’m working on customizing the 70B llama 2 model for my specific needs. Can you provide information on the required GPU VRAM if I were to run it with a batch size of 128? I assumed 64 GB would be enough, but got confused after reading this post. <a href="https://discuss.pytorch.org/t/how-to-deal-with-excessive-memory-usages-of-pytorch/126098/2">How to deal with excessive memory usages of PyTorch? - vision - PyTorch Forums</a></p>
<p>Thanks</p>
How to implement deeply supervised network

ltnghia: <p>Can someone give me some advice to implement deeply supervised network, which has multiple loss? Thank you.</p>

greaber: <p>I don’t understand what you are confused about.  Don’t you just take a (weighted) sum of the different losses?</p>

ltnghia: <p>I want to implement the network like this. But I don’t how to implement multiple outputs and losses. I am just a newbie in PyTorch.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f3722aaedf882e62f17b316805f8170ddaada1e2.png" alt="Picture2" width="675" height="356"></p>

greaber: <p>If you look at a basic PyTorch tutorial like <a href="http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a>, you will see code like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>But there is nothing stopping you from having <code>forward</code> return outputs from several layers instead of just the last layer, and then you can use those other outputs to compute losses just like in the tutorial you use the last output to compute a loss.  And you can add the losses together to get your overall loss.</p>

Bibhabasu_Mohapatra: <p><a class="mention" href="/u/ltnghia">@ltnghia</a><br>
<a href="https://github.com/qubvel/segmentation_models.pytorch/issues/741" class="inline-onebox" rel="noopener nofollow ugc">Proposing option for Support for getting the Decoder outputs · Issue #741 · qubvel/segmentation_models.pytorch · GitHub</a>   I have made a tutorial solution for this.<br>
<a href="https://www.kaggle.com/code/bibhabasumohapatra/deep-supervised-in-segmentation-models-pytorch?scriptVersionId=124097237" class="inline-onebox" rel="noopener nofollow ugc">Deep Supervised in Segmentation Models PyTorch | Kaggle</a></p>