akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Use grad() graph assuming independence between inputs to compute an expression

JosueCom: <p>Assume I have 3 neural networks (linear layer for example):</p>
<pre data-code-wrap="python"><code class="lang-python">nn1 = nn.Linear(1, 1) # takes t, generates x
nn2 = nn.Linear(1+1, 1) # takes t and x, generates y
nn3 = nn.Linear(1+1+1, 1) # takes t, x, and y, generates z
</code></pre>
<p>And I have some unknown expression h that takes all outputs (x, y, and z) and yield some value.</p>
<pre data-code-wrap="python"><code class="lang-python">t = th.rand(10, 1)
x = nn1(t)
y = nn2(th.cat([t, x], dim=1))
z = nn3(th.cat([t, x, y], dim=1))

h = 3*x*y+z*y + 4 # Assume h is not known
</code></pre>
<p>Can I compute a function/expression/graph/etc for dh/dy (for example) that assumes the variables x, y, and z are independent, but that I can use to do back-propagation acknowledging their dependence?</p>

JosueCom: <p>Another perceptive:<br>
In the example above, can I use grad to compute hy = 3*x + z (assuming I don’t know h initally) with the information given above?</p>

zehonyi21: <p>idk but there could be two options that<br>
first one is just<br>
h.backward() and y.grad()= ah/ay<br>
if y is declared tensor like t</p>
<p>second one is<br>
h.backward() gives t.grad()= ah/at,<br>
zero_grad()<br>
y.backward() gives t_new.grad()= ay/at,</p>
<p>ah/ay=ah/at * at/ay = t.grad() / t_new.grad()</p>
<p>How about using these?</p>
Unusual CPU usage from slightly increasing batch size

dvijay: <p>I’m trying to train a simple LSTM model. The input size is about 11KB (same as target size). When I run the training loop with a batch size of 32, everything seems to work well. CPU usage is under 10% (even though that feels a bit excessive) and GPU usage is around 30%…</p>
<p>When I increase the batch size to 48, the CPU usage goes to 100% and everything slows down (training takes twice as long). GPU usage is virtually unchanged at around 30%. I checked with batch size of 40 and that seems fine. At 48 something seems to change and triggers the constant CPU usage. Everything else seems ok (plenty of RAM left, hardly any disk activity, etc).</p>
<p>Any suggestions or ideas on how to troubleshoot this? At the batch size of 48 the total input size should be about 528KB. Not sure if this triggers some kind of threshold somewhere. Any help is greatly appreciated.</p>

ptrblck: <p>You could check if the data loading itself shows a significant increase in the CPU processing when the batch size is increased.</p>
Pytorch with CUDA support installation offline

Swarna_Y.V: <p>Hello,<br>
I am using GeForce 1650 Graphics card. How to install CUDA 12.5 manually (without internet). I installed, CUDA 12.5, cudnn 9.2 and supporting drivers also. I installed torch from python using wheel files. Now my torch is not detecting GPU. Give me some advice.</p>

Ynjxsjmh: <p>Did you install the cuda version?</p>
<p>If you use pip, you need download the whl file that looks like <code>cu118/torch-2.3.0%2Bcu118-cp310-cp310-win_amd64.whl</code> from <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a> .</p>
<p>If you use conda, you can download file like tar.bz2 from anaconda site. Besides the PyTorch package, from <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>, you also need install <code>pytorch-cuda</code> or <code>cudatoolkit</code> package. (Personally speaking, I don’t know what does these two packages do and why don’t you need these packages for <code>pip</code>. )</p>
Optimizer load and update

DavidHr: <p>Hi all,</p>
<p>I have a Transformer model that is trained with diferent sets of data. In some of them could be new tokens that increase the vocab_size, and so the embeddings/possitions… These are under control. The problem arise when I load a previous optimizer because vocab_size could increase.</p>
<p>I know in Pytorch this can be achieved by copying the old info into new. and continue training but don’t know how to do it in Libtorch C++.</p>
<p>the pseudo code could be.</p>
<p>torch::load(optimizer, “.\IA\” + sEmpresa + “_optimizer.pt”);</p>
<p>model-&gt;IncreaseVocab(new_vocab_size);</p>
<p>torch::optim::Adam new_optimizer(model-&gt;parameters(), torch::optim::AdamOptions(0.001));</p>
<p>then I need that new_optimizer has all the previous info (new_optimizer = optimizer+new_size)</p>
<p>any help would be appreciated.</p>
What is the difference between the .t7 format and .pth format for saving a model, and more specifically, which is better?

songyuc: <p>Hi guys,<br>
I am training a model these days, and I find that there is another format of <code>.t7</code> for saving a model.<br>
So, I am wondering what is the difference between the <code>.t7</code> format and <code>.pth</code> format for saving a model, and more specifically, which would be a better choice?</p>
<p>Your answer and idea will be appreciated!</p>

ptrblck: <p><code>.t7</code> was used in Torch7 and is not used in PyTorch. If I’m not mistaken the file extension does not change the behavior of <code>torch.save</code>.</p>

songyuc: <p>Thanks sincerely for you explanation!</p>
Suggestion for a Unified Documentation Search

songyuc: <p>Hello PyTorch Team,</p>
<p>I hope this message finds you well. I would like to share some feedback regarding the organization of the PyTorch documentation.</p>
<p>Currently, the documentation is divided into “PyTorch” and “PyTorch Domains.” While this structure is informative, it can be confusing for new users who may struggle to find specific API functions, especially when they need to identify whether a function belongs to “PyTorch” or “PyTorch Domains.”</p>
<p>I suggest creating a unified search entry point that encompasses all documentation. This would allow users to quickly find the information they need without having to navigate between different sections. A centralized search function could significantly enhance the user experience, making it easier for both beginners and experienced users to access relevant information.</p>
<p>Thank you for considering this suggestion. I appreciate your efforts in making PyTorch an outstanding framework!</p>
<p>Best regards,<br>
Yucheng Song</p>
CrossEntropy Loss using One-hot encoded channels

Radu_Popovici: <p>Hi, I am developing an Unet model for bio-medical images. I was wondering if I could pass to the function the predictions as B x C x H x W and the target as B x C x H x W, where for the channels I preprocessed the target mask so that along the C dimension there is a 1 for where the respective class aka label is. Is this the correct way? I have seen people saying otherwise.</p>

ptrblck: <p>Yes, this is possible in newer PyTorch releases since “soft-targets” are now supported (unsure which PyTorch releases added this functionality).<br>
However, using the “classical” indices would save some memory since only the class index needs to be stored in the target.<br>
Here is a small example showing both approaches:</p>
<pre data-code-wrap="python"><code class="lang-python">criterion = nn.CrossEntropyLoss()

N, C, H, W = 2, 3, 4, 4
x = torch.randn(N, C, H, W, requires_grad=True)

# indices
target = torch.randint(0, C, (N, H, W))
loss = criterion(x, target)
print(loss)
# tensor(1.7541, grad_fn=&lt;NllLoss2DBackward0&gt;)

# soft-targets
soft_target = F.one_hot(target, num_classes=C).float()
print(soft_target.shape)
# torch.Size([2, 4, 4, 3])
soft_target = soft_target.permute(0, 3, 1, 2)
print(soft_target.shape)
# torch.Size([2, 3, 4, 4])
loss = criterion(x, soft_target)
print(loss)
# tensor(1.7541, grad_fn=&lt;DivBackward1&gt;)
</code></pre>

Radu_Popovici: <p>My dataset is heavily imbalanced, and I think my model is biased towards the background label. Would the ignore_index work for the one-hot encoded version? or should I just argmax and pass ignore_index=0, where 0 is the channel of the background?</p>
Multiple thread inference on one gpu can NOT fully utilise the gpu

dean_yang: <p>I’m doing reinforcement learning with c++ frontend on one rtx4090D GPU. I opend mutiple thread and assign a stream to each, and then do inference with single module. The functun work well, but has some performance issue.<br>
1  With very high thread number ,the throughput and gpu utilization decrease.  The best thread number on my env is 4. But still just ultilise 50% of gpu.<br>
2 If 2 processes with 4 thread each are lauched ,the gpu ultilization rate can be 100%, and throughput is doubled.</p>
<p>Seems the mutiple thread way is blocked some where?</p>

dean_yang: <p>I finally find it was block in lstm foward in get_dropout_state on a static varable:</p>
<p>static std::vector dropout_state_cache{<br>
static_cast&lt;size_t&gt;(cuda::getNumGPUs())};</p>
<p>And the related comments says:</p>
<p>// Every time we use a dropout state, we need to synchronize with its event,<br>
// to make sure all previous uses finish running before this one starts. Once<br>
// we’re done, we record the event to allow others to synchronize with this<br>
// kernel. Those events are really needed only for inter-stream sync on a<br>
// single GPU. I doubt anyone will want to run cuDNN RNNs in parallel on a<br>
// single GPU, so they should end up being complete no-ops.</p>
<p>In fact lstm forwad doesn’t need a DropoutState, but it still mutex lock on it.<br>
It turn out c++ mutiple thread on single device is not well supported here.</p>
Pytorch 1.1 and CUDA 12.5

ate254: <p>Hello! I am trying to run a program and these are the requirements:</p>
<ul>
<li>Python 3.6+</li>
<li>PyTorch 1.1 or higher (tested on PyTorch 1.1, 1,3, 1,5~1.10)</li>
</ul>
<p>I have CUDA 12.5 and I was trying to install pytorch 1.10. However, it is not supported by CUDA 12.5 so, can it be supported by CUDA 11.8 for example? If not, between those options which version of pytorch would be the best option?</p>
<p>Thanks in advance</p>

ptrblck: <aside class="quote no-group" data-username="ate254" data-post="1" data-topic="208753">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/65b543/48.png" class="avatar"> ate254:</div>
<blockquote>
<p>However, it is not supported by CUDA 12.5 so, can it be supported by CUDA 11.8 for example?</p>
</blockquote>
</aside>
<p>Your locally installed CUDA toolkit won’t be used as PyTorch ships with its own CUDA dependencies (unless you build PyTorch from source or a custom CUDA extension). Only a valid NVIDIA driver must be installed to execute PyTorch workloads on an NVIDIA GPU.</p>
Dp or ddp only for parallel computing

joohyunglee: <p>I would like to have different models, but I would also like them to share the data input.<br>
Also, I want to train those models in parallel (maybe 1 model per 1 GPU).</p>
<p>so I assume dp or ddp may work, but they all synchronize the weight, and for ddp in particular, it splits the dataset for each GPU through the sampler, and sync the gradient from each model.</p>
<p>How can I do this?<br>
Thank you ahead</p>

tianyu: <p>In that case, you can just kick off different jobs on different GPUs. They don’t have to be executed absolutely together in a synchronized way.</p>

joohyunglee: <p>thank you for the reply. yes right, but i would like them to share the data pipeline due to the large data memory overhead.<br>
Also those models are from the same graph but with different initial weight. Thank you.</p>

tianyu: <p>to share data loading, you can do</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp

ITERS = 10
def worker(qs, rank):
    for it in range(ITERS):
        print(f"rank {rank}, it {it}: {qs[rank].get()}")

def main():
    num_workers = 4
    qs = [mp.JoinableQueue() for _ in range(num_workers)]
    processes = []
    
    for rank in range(num_workers):
        p = mp.Process(target=worker, args=(qs, rank))
        p.start()
        processes.append(p)

    for it in range(ITERS):
        inp = torch.full((1,), it).share_memory_()
        for rank in range(num_workers):
            qs[rank].put(inp, block=False)
    
    for p in processes:
        p.join()

if __name__ == "__main__":
    mp.set_start_method('spawn')
    main()
</code></pre>
<p>NOTE: It depends on what kind of overhead you want to avoid - if the overhead is in producing and storing the input data in host memory, this approach would help, because the data is produced once and shared among all processes via shared memory. However, if the overhead lies in Host-to-Device copy and/or input data GPU memory consumption, this would not help, because you can’t really physically share memory across different devices.</p>
What does the `use_fast_accum` option do in `torch._scaled_mm`

veritas: <p>Recent PyTorch support for FP8 in the <code>torch._scaled_mm</code> has a <code>use_fast_accum</code> flag that I have found to increase throughput by a noticable amount.<br>
However, even after going through the CUDA code, I was unable to find out what this option does and what potential effects it may have on the matrix multiplication outputs.<br>
May I ask for help on where to find detailed documentation on the matter?<br>
If such documentation is not available, I would like to request that it be added, maybe in <code>torch.ao</code>.</p>
<p>Many thanks in advance!</p>

ptrblck: <p>This flag enables <code>CUBLASLT_MATMUL_DESC_FAST_ACCUM</code> <a href="https://github.com/pytorch/pytorch/blob/dde5974b1368d26bc2cade9722b8ef242e46e242/aten/src/ATen/cuda/CUDABlas.cpp#L1432">here</a> which is defined as:</p>
<blockquote>
<p>Flag for managing FP8 fast accumulation mode. When enabled, problem execution might be faster but at the cost of lower accuracy because intermediate results will not periodically be promoted to a higher precision. Default value: 0 - fast accumulation mode is disabled</p>
</blockquote>
<p>in the <a href="https://docs.nvidia.com/cuda/cublas/">cuBLAS docs</a>.</p>
Parallelizing many tiny MLP operations

Patrick_Darrow: <p>Objective:<br>
I’d like to take an input image, create dozens or hundreds of patches it, and run each patch through potentially dozens of small MLPs (&lt;1000 parameters). The patch selection is done deterministically, which I’m less worried about. The MLPs, however, currently execute serially in a for-loop which is tragically slow.</p>
<p>Question:<br>
Is there native support for parallelization of many (small) NNs?</p>
<p>TYIA!</p>

vmoens: <p>Try functorch!<br>
It’s in the latest nightly release of pytorch or installed independently with the latest stable.</p>
<p>You should retrieve the params of the modules using make_functional().<br>
Then you can stack your parameters across NNs and use vmap to dispatch the data and params to the functional module.</p>
<p>Have a look at vmap here</p>
<p><a href="https://pytorch.org/functorch/stable/generated/functorch.vmap.html#functorch.vmap" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/functorch/stable/generated/functorch.vmap.html#functorch.vmap</a></p>

SY_QQX: <p>Sorry that I’m late to the party. You could use torch.bmm to compute multiple mlps at the same time.</p>
<pre><code class="lang-auto">img = torch.randn([224, 224, 3])
patches = img.reshape([-1, 3]) # one pixel per patch
patch_size, num_channel = patches.shape

w1 = torch.randn([patch_size, num_channel, hidden_size])
w2 = torch.randn([patch_size, hidden_size, out_size])

# perform multi-mlp
out = torch.bmm(patches.unsqueeze(1), w1)
out = torch.relu(out)
out = torch.bmm(out, w2).squeeze(1)
</code></pre>
Pybind dataloader pickle singleton class conundrum

Robert_Lamont: <p>Good day fellow pytorch enthusiasts,</p>
<p>I have a multi file dataset that i use boost::iostreams::mapped_file to index into and then collate into tensors.<br>
pybind11 is used to create extension to be used in pytorch.<br>
I send these tensors to the gpu on the pytorch side.<br>
This works fine with num_workers = 0, but when I use more workers I have to pickle the class.<br>
I want the file handler class to be a singleton because it has the file handles used by boost::iostreams::mapped_file which improves caching and whatnot since the files are bigger than my RAM.</p>
<p>The thing is that I don’t know how to get all the instances of the OneMinuteDataset class to point to a single instance of file_holder.</p>
<p>here is my attempt using constructor overloading (only relevant parts included):</p>
<pre><code class="lang-auto">OneMinuteDataset::OneMinuteDataset()
{
    file_holder = std::make_shared&lt;FileHolder&gt;();
    std::cout &lt;&lt; "OneMinuteDataset ctor " &lt;&lt; &amp;file_holder &lt;&lt; "\n";
    file_holder-&gt;load_files();
}

OneMinuteDataset::OneMinuteDataset(std::shared_ptr&lt;FileHolder&gt; file_holder)
{
    this-&gt;file_holder = file_holder;
    std::cout &lt;&lt; "increase instance count" &lt;&lt; &amp;file_holder &lt;&lt; "\n";
}

#define PYBIND11_DETAILED_ERROR_MESSAGES
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)
{
    py::class_&lt;OneMinuteDataset&gt;(m, "OneMinuteDataset")
        .def(py::init&lt;&gt;())
        .def(py::init&lt;std::shared_ptr&lt;FileHolder&gt;&gt;())
        .def("size", &amp;OneMinuteDataset::size)
        .def("get_input", &amp;OneMinuteDataset::get_input)
        .def("get_target", &amp;OneMinuteDataset::get_target)
        .def(py::pickle(
            [](const OneMinuteDataset &amp;p) { // __getstate__
                return p.file_holder;
            },
            [](std::shared_ptr&lt;FileHolder&gt; s) { // __setstate__
                OneMinuteDataset p(s);

                return p;
            }
        ));


    py::class_&lt;FileHolder, std::shared_ptr&lt;FileHolder&gt;&gt;(m, "FileHolder")
        .def(py::init&lt;&gt;())
        .def("load_files", &amp;FileHolder::load_files);
/*
        .def(py::pickle(
            [](const std::shared_ptr&lt;FileHolder&gt; &amp;p) {
                return p;
            },
            [](const std::shared_ptr&lt;FileHolder&gt; &amp;p) {
                return p;
            }
        ));
*/
}

</code></pre>
<p>I have to include a pybind for FileHolder because its exposed in the pickle for OneMinuteDataset,<br>
but it wants that pickled too.</p>
<p>Is there a way to hide some class internals from pybind so i don’t have to pickle all the way down?<br>
Should I attempt some disjoint service to serve the data instead?</p>
<p>any hint or advice would be appreciated.</p>
<p>Regards,<br>
Rob</p>
Using AMP with QAT

Archer_Z: <p>I’m trying to speed up both training phase and inference time. While QAT gives promising results, but when I add AMP I get RuntimeError: expected scalar type Float but found Half. It seems fake quant only support float. Is this intended or is there work around?</p>

FactFraction: <p>This <a href="https://discuss.pytorch.org/t/mixed-precision-training-and-quantisation-aware-training-together/189758">post</a> seems to be what you are looking for.</p>
AssertionError: No inf checks were recorded for this optimizer. in custom diffusion model

Dave1: <p>I am trying to modify the UNet2DModel from the diffusers library by creating a new class and inheriting from it. I am trying to attach encoders and decoders onto the diffusion layers. However, I am not quite sure how to change the forward method. This is the code I’ve tried:</p>
<pre data-code-wrap="py"><code class="lang-py">class CustomUNet2DModel(UNet2DModel):
    def __init__(self, encoder, decoder, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.encoder = encoder
        self.decoder = decoder

    def forward(self, x, timesteps, *args, **kwargs):
        # Encode the input
        encoded_x = self.encoder(x)
        
        # Apply the diffusion process using the parent class
        diffusion_output = super().forward(encoded_x, timesteps, *args, **kwargs)
        
        # Decode the output back to the original size
        decoded_output = self.decoder(diffusion_output[0])
        
        return decoded_output, 0
</code></pre>
<p>But it’s giving me the error <code>AssertionError: No inf checks were recorded for this optimizer.</code>. I think this is because something about my logic of calculating the forward method is wrong, but I am not quite sure how to fix this.</p>

ptrblck: <p>Could you add the missing parts of the code showing how <code>amp</code> is used and how the optimizer is created?</p>

Dave1: <p>It turns out that the problem was because I did not change the model variable in the optimizer definition…<br>
<code>optimizer = torch.optim.AdamW(model.parameters(), lr=TrainingConfig.learning_rate)</code></p>
Feature extraction using pretrained Alexnet

gyanendar: <p>Hi Guys,</p>
<p>I am trying to use pytorch pretrained Alexnet model for feature extraction, which I will pass to the SVM classifier (scikit). I am doing the transfer learning as my dataset is small.</p>
<p>I got the model as<br>
alexnet_model = models.alexnet(pretrained=True)<br>
Then removed the fully connected layer<br>
alexnet_model.classifier = torch.nn.Sequential(*list(alexnet_model.classifier.children())[:-4])</p>
<p>I am unsure about how to pass the training image as input to the model and get the extracted features! Do I have to get the features of each input image sequentially through some loop? Is there a way I can do this all at once?</p>
<p>Thanks,</p>

ArminBaz: <p>As far as I’m aware, the images have to go through the model one by one. You can speed up the process through using a GPU and batching your inputs. Generally for <code>test_batch</code> and <code>val_batch</code> sizes people choose the largest batch size that their hardware is capable of supporting.</p>
<p>I would recommend using a <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">custom dataset</a> and dataloader to assist.</p>

sabillahm98: <p>I have this problem now. do you know how to solve this?</p>
Tensor Broadcasting problem

Iaan: <p>Hi, I’m trying to extract InternVideo2 model’s features and i’m stuck with this error. I hope I can get some help. :</p>
<p><strong>RuntimeError: The size of tensor a (9) must match the size of tensor b (96) at non-singleton dimension 1</strong></p>
<p>these are my input data :</p>
<pre><code class="lang-auto">input = {}
input['input_ids'] = input_ids
input['attention_mask'] = attention_mask
input['labels'] = labels
input['video'] = video_tensor

</code></pre>
<p>I expect that in this <a href="https://huggingface.co/OpenGVLab/InternVideo2-Chat-8B/blob/main/modeling_videochat2.py" rel="noopener nofollow ugc">page</a> ‘line 104’ cause this error.</p>
<pre><code class="lang-auto">text_embeds[video_idx == 1] = text_embeds[video_idx == 1] * 0 + prompt_video_embeds.to(text_embeds.device).to(text_embeds.dtype)
</code></pre>
<p>and this is the exact code of getting text embedding</p>
<pre><code class="lang-auto">text_embeds = self.lm.get_input_embeddings()(input_ids.long()).detach()
</code></pre>
<pre><code class="lang-auto">self.word_embeddings = nn.Embedding(
            config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id
        )
...

def get_input_embeddings(self):
        return self.embeddings.word_embeddings
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4e2dee4c4472ee9ed93db5f7b08209ff1a5a39e.png" data-download-href="https://discuss.pytorch.org/uploads/default/c4e2dee4c4472ee9ed93db5f7b08209ff1a5a39e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4e2dee4c4472ee9ed93db5f7b08209ff1a5a39e.png" alt="image" data-base62-sha1="s5JErNPZYIOcuoILV5K7j5mz1rE" width="690" height="79" data-dominant-color="252829"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">716×82 9.78 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
How can I run this code? <strong>My understanding is that two tensors (text_embeds, prompt_video_embeds) must be matched in size so that they can be added but I don’t know how.</strong></p>
<p>By the way, I also don’t understand why it’s called text embedding when initialize the text embedding values and replace them with video embedding.</p>
AssertionError: torch.float32 != torch.bfloat16

zhangruoxu: <p>Hi all,</p>
<p>I encountered an issue while trying to use <code>torch.compile</code> to compile a model with FSDP enabled. The assertion error arises from <a href="https://github.com/pytorch/pytorch/blob/main/torch/_subclasses/meta_utils.py#L74" rel="noopener nofollow ugc">this line</a> during the verification of fake tensors. The model I am compiling is from <a href="https://github.com/huggingface/transformers/blob/v4.34.0/src/transformers/models/llama/modeling_llama.py#L141" rel="noopener nofollow ugc">this file</a>.</p>
<p>After investigating the issue, I found some insights that might be helpful.</p>
<p>Let’s consider the tensor <code>cos_cached</code> in the model code referenced above.</p>
<ol>
<li><code>cos_cached</code> is a view, created when <code>emb.cos()</code> generates a tensor followed by a slicing operation.</li>
<li><code>cos_cached</code> is copied to the GPU <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/fsdp/_init_utils.py#L1039" rel="noopener nofollow ugc">here</a>. In this step, the <code>data</code> field of the tensor is directly modified. As a result, <code>cos_cached</code> remains a view, but it no longer shares data with the underlying tensor.</li>
<li><code>cos_cached</code> is cast according to automatic mixed precision <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/fsdp/_runtime_utils.py#L1638" rel="noopener nofollow ugc">here</a>. In my case, the dtype of <code>cos_cached</code> was originally <code>float32</code> and is now cast to <code>bf16</code>. Consequently, <code>cos_cached</code> is still a view, it does not share data with the underlying tensor, and their dtypes now differ.</li>
</ol>
<p>Finally, the type-checking assertion error occurs when creating a fake tensor for <code>cos_cached</code>.</p>
<p>I am wondering if this could be a bug and if directly modifying <code>tensor.data</code> is recommended in such cases.</p>
<p>Thanks,<br>
Yifei</p>
Call activation function from string

mpariente: <p>Hi,</p>
<p>Is there a way to call an activation function from a string?<br>
For example something like this :</p>
<pre><code class="lang-auto">activation_string = "relu"
activation_function = nn.activation(activation_string)
u = activation_function(v)
</code></pre>
<p>It would be really practical to have something like this, for example to define the activation function in a config file, instead of inside the classes.</p>
<p>Thanks in advance,<br>
Manu</p>

miguelvr: <p>you can create a dictionary with several activation function mappings:</p>
<pre><code class="lang-auto">activations = {
    'relu': nn.ReLU()
    'sigmoid': nn.Sigmoid(),
    'tanh': nn.Tanh()
}
</code></pre>
<p>and then call</p>
<pre><code class="lang-auto">activation_function = activations[activation_string]
u = activation_function(v)
</code></pre>

mpariente: <p>Thank you for your answer.<br>
I considered this option before asking the question but I just wanted to know if there was any built-in implementation before reinventing the wheel.<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

robloughnan: <p>You can dynamically access the method by using <code>getattr</code>:</p>
<pre><code class="lang-auto">activation_string = "relu"
activation_function = getattr(nn, activation_string)()  
u = activation_function(v)
</code></pre>

jr2804: <p>Hi, just in case that there is still a need for this functionality … I wrote the following helper function that parses out the export variable (<strong>all</strong>) in submodule torch.nn.modules.activation:</p>
<pre><code class="lang-auto">from torch.nn import Module
from typing import Callable
from torch.nn.modules import activation

Activation = Callable[..., Module]


def get_activation_fn(act: str) -&gt; Activation:
    # get list from activation submodule as lower-case
    activations_lc = [str(a).lower() for a in activation.__all__]
    if (act := str(act).lower()) in activations_lc:
        # match actual name from lower-case list, return function/factory
        idx = activations_lc.index(act)
        act_name = activation.__all__[idx]
        act_func = getattr(activation, act_name)
        return act_func
    else:
        raise ValueError(f"Cannot find activation function for string &lt;{act}&gt;")
</code></pre>
<p>Note that this function intentionally returns a <em>reference</em> to a torch.nn.Module, which still has to be instatiated. This allows to use the output of this function as a factory and with additional arguments, e.g.:</p>
<pre><code class="lang-auto">act1 = get_activation_fn('relu')
print(act1())
ReLU()

act2 = get_activation_fn('gelu')
print(act2())
GELU(approximate='none')

act3 = get_activation_fn('elu')
print(act3(alpha=1.2))
ELU(alpha=1.2)

</code></pre>
<p>Hope it helps!</p>
RuntimeError: Cannot find callable deit_tiny_patch16_224 in hubconf

neouyghur: <p>Hi,</p>
<p>Could you someone help me understand why I got this error “RuntimeError: Cannot find callable deit_tiny_patch16_224 in hubconf”, when running the following code:</p>
<pre><code class="lang-auto">    model = torch.hub.load('facebookresearch/deit:main',
                                   'deit_tiny_patch16_224', pretrained=True)
</code></pre>

ptrblck: <p>I cannot reproduce the issue using a recent nightly build:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.hub.load('facebookresearch/deit:main',
                       'deit_tiny_patch16_224', pretrained=True)

model
# VisionTransformer(
#   (patch_embed): PatchEmbed(
#     (proj): Conv2d(3, 192, kernel_size=(16, 16), stride=(16, 16))
#     (norm): Identity()
#   )
#   (pos_drop): Dropout(p=0.0, inplace=False)
#   (patch_drop): Identity()
#   (norm_pre): Identity()
#   (blocks): Sequential(
#     (0): Block(
# ...
</code></pre>

neouyghur: <p>Thanks for your reply. I have fixed it. It was caused by another custom module.</p>
Learn How to Leverage PyTorch 2.4 for Accelerating AI with this Workshop

sonyai: <p>Check out this workshop to learn how to leverage PyTorch 2.4 on a developer cloud to develop and enhance your AI workloads.</p>
<p>Through this workshop, you’ll:</p>
<ul>
<li>Experience seamless AI development on the Intel Tiber Developer Cloud</li>
<li>Try PyTorch 2.4 for fast and more dynamic AI models</li>
<li>Gain practical skills to take your AI projects to the next level</li>
</ul>
<p><a href="https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Accelerate-AI-Workloads-with-a-PyTorch-2-4-Workshop-on-the-Intel/post/1625501" rel="noopener nofollow ugc">https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Accelerate-AI-Workloads-with-a-PyTorch-2-4-Workshop-on-the-Intel/post/1625501</a></p>
Variable Sequnce Image and MetaData

Nipun_sandamal: <p>Hello Everyonve,<br>
Currently, I’m reading a paper and trying to implement it myself since the authors haven’t provided the code. This implementation is focused on melanoma recognition, and specifically, I’m working on building a classification system that can operate on a per-patient basis. For example, if my batch size is 4, each sample in the batch has a different sequence length. We are not only using images but also incorporating patient metadata. Therefore, when feeding the data into the encoder of the transformer, I have added padding and provided the mask accordingly.</p>
<p><strong>My question is: when it comes to classification, I want to perform classification based on the original sequence lengths, which vary within the batch. Is there any way to achieve this? Any help would be greatly appreciated</strong></p>
<p>Here is the github repository i had created:-<a href="https://github.com/rinnegann/Alive/blob/master/Paper_Implementation/PatientWiseClassification/encoder.ipynb" rel="noopener nofollow ugc">GitHub Repo</a></p>
Change dropout rate in jit script

Christopher_Manderfe: <p>Hi,<br>
is it possible to change the dropout rate of a pretrained model (jit script)?<br>
I have a working pretrained model for image classification, where the classification layer is trainable. Dropouts are implemented, but I would like to change the dropout rate in case that less training samples are used to avoid overfitting.<br>
Thanks<br>
Chris</p>

Soumya_Kundu: <pre><code class="lang-auto">for name, module in model.named_modules():
    if isinstance(module, torch.nn.Dropout):
        module.p = new_dropout_rate
</code></pre>
<p>Would something like this work?</p>

Christopher_Manderfe: <p>Thanks for your answer,<br>
actually this seems not to work. After reading all these jit generated files, I found that the dropout rate is a hardcoded argument. It’s not a parameter (like weights).</p>

Soumya_Kundu: <p>Thanks for the update</p>
Inference workflow in compile mode using transformers.pipeline()

Arunima_Ghosh: <p>Hi,</p>
<p>I am trying to run an inference workflow of a Llama model in compile mode using transformers.pipeline(). I am using the following line of codes to run the inference workflow in compile mode:</p>
<pre><code>model = LlamaForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-70B",use_cache=True,device_map='auto')
 tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-70B",use_cache=True,truncation=True,padding="max_length",max_length=64,return_tensors="pt")
 tokenizer.pad_token = tokenizer.eos_token
 tokenizer.padding_side = "left"
    model = torch.compile(model)
    pipeline = transformers.pipeline(
                         "text-generation",
                           model=model,
                           model_kwargs={"torch_dtype": torch.bfloat16},
                           tokenizer = tokenizer,
                           device_map="auto",
                            )		
    generation_config = {
                            "num_beams":1,
                            "max_new_tokens":32,
                            "do_sample":True,
                             "use_cache":True,
                              }
    outputs = pipeline(input_prompt,**generation_config)
</code></pre>
<p>It is expected that torch.compile() should compile the model, print some compilation messages but I am not getting any compilation messages.<br>
There is no error message but not getting any compilation messages.</p>
<p>Can you please guide me what might be wrong in code? How can I run an inference workflow of llama model in compile mode using transformers.pipeline()?</p>
<p>Thanks</p>
I can't seed with nn.DataParallel

Tahamustapha_Nehdi: <p>Hi i want to seed my env<br>
when i use</p>
<pre><code class="lang-auto">model = model.cuda() 
</code></pre>
<p>my model seeded correctly and i can reporduce the results<br>
but when i try to use :</p>
<pre><code class="lang-auto">model = nn.DataParallel(model,device_ids=[0,1]).cuda()
</code></pre>
<p>i have a small variability in my results</p>

ptrblck: <p>Did you follow all instructions mentioned in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility Docs</a>?</p>

Tahamustapha_Nehdi: <p>ofc i use this to seed model init and other numpy related funcs</p>
<pre><code class="lang-auto">random_seed = 42
random.seed(random_seed)
torch.manual_seed(random_seed)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(random_seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

</code></pre>
<p>for data loading processes seed i use this</p>
<pre><code class="lang-auto">import numpy 
import random
def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    numpy.random.seed(worker_seed)
    random.seed(worker_seed)

seed = 42
generator = torch.Generator()
generator.manual_seed(seed)

...... 

self.train_loader = torch.utils.data.DataLoader(
    trainset,
    sampler=build_train_sampler(
        trainset.train,
        train_sampler,
        batch_size=batch_size_train,
        num_instances=num_instances,
    ),
    batch_size=batch_size_train,
    shuffle=False,
    num_workers=workers,
    drop_last=True,
    generator=generator,
    worker_init_fn=seed_worker,
    
)
</code></pre>
<p>i think the problem is related to parallel preoceeses lunched useing nn.DataParallel</p>

ptrblck: <aside class="quote no-group" data-username="Tahamustapha_Nehdi" data-post="3" data-topic="208524">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tahamustapha_nehdi/48/55670_2.png" class="avatar"> Tahamustapha_Nehdi:</div>
<blockquote>
<p>ofc i use this to seed model init and other numpy related funcs</p>
</blockquote>
</aside>
<p>I might be missing it, but I don’t see e.g. <code>torch.use_deterministic_algorithms(True)</code> as mentioned in the docs.</p>
<aside class="quote no-group" data-username="Tahamustapha_Nehdi" data-post="3" data-topic="208524">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tahamustapha_nehdi/48/55670_2.png" class="avatar"> Tahamustapha_Nehdi:</div>
<blockquote>
<p>i think the problem is related to parallel preoceeses lunched useing nn.DataParallel</p>
</blockquote>
</aside>
<p>Why do you think that’s the case?</p>
Is torch.from_blob not available in python interface?

Stephan_D: <p>Hi, is it by design that torch.from_blob is not exposed in the python API? Or am I looking in the wrong places? I find plenty of docs with respect on how to use it in C++ interface but on python3.11, torch==2.1.2+cu121 (cuda 12.1), I cannot seem to find it. Context is I work in a python heavy shop and would like to be able to express/execute from_blob in python code.</p>

albanD: <p>Hey!</p>
<p>We indeed don’t expose that API to python for three main reasons: it is quite dangerous as the ownership system is not something that is “pythonic”, it requires some arguments/constructors to be c++ constructs to properly interface with the c++ backend and we never needed it.<br>
In particular, most systems using it like NumPy or DLPack interrop already work within c++ and so we never had the need.<br>
I would also mention that for testing, I’ve use <code>torch.util.cpp_extension.load_inline()</code> to write a small wrapper around that API and expose specific behavior to Python.</p>
<p>If you have a good use case for it and we can come up with a good pythonic API for it, we can definitely add that.<br>
If you’re interested, the first step would be to open an issue with details on the new python API so we can discuss that because adding the binding.</p>
Minimizing a custom function

badinmaths: <p>I have to approximate the maximum of the following function :</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/301ce5eadd4531d6ef3dec9e781da68a94ddcf7b.png" alt="image" data-base62-sha1="6RCPC9OTbCFsnJfEmcKmHn854Sv" width="617" height="104"></p>
<p>This problem seems unsolvable, but I thought we could approximate the solution and obtain good functions g1, g2 by using PyTorch. Additionally, it gives me practice with this framework, which I don’t yet fully master.</p>
<p>I suppose that g1(x) € {1,…max_value} and I use a simple model which gives almost integer values for gi with the smooth argmax.<br>
I approximate the double integration by a double summation.</p>
<p>With this method I obtain about 0.25 as a maximum. My colleagues said me I can obtain better with 0.35. Do you know how to improve the code?</p>
<pre data-code-wrap="py"><code class="lang-py">    import torch
    import torch.nn as nn
    import torch.optim as optim
    
    def smooth_floor(x, alpha=0.01):
        #if type(x) == float : x = torch.tensor([x])
        return torch.floor(x) + torch.sigmoid(alpha * (x - torch.floor(x))) - 0.5
    
    
    def smooth_argmax(x, dim=-1, temperature=0.01):
        x_softmax = torch.softmax(x / temperature, dim=dim)
        indices = torch.arange(x.size(dim), device=x.device).float()
        return torch.sum(x_softmax * indices, dim=dim)
    
    
    def eps(x) : 
        return  smooth_floor(x) - 2 * smooth_floor(x/2)
        
    class Function(nn.Module): 
        
        def __init__(self, max_value):
            super().__init__()
            self.layer = nn.Linear(1,100)
            self.layer2 = nn.Linear(100, max_value)
    
        def forward(self,x):
            x = self.layer(x)
            x = nn.ReLU()(x)
            x = self.layer2(x)
            return 1 + smooth_argmax(x)
        
    def loss(discretisation_step) : 
        T = torch.linspace(0,1,discretisation_step).reshape(-1,1)
        X = f(T)
        Y = g(T)
        T = T.reshape(-1)
        part1 = eps(torch.einsum("b,c-&gt;bc",T,2**X))
        part2 = eps(torch.einsum("b,c-&gt;bc",T,2**Y)).T
        return -(part1*part2).sum()/discretisation_step**2
    
    max_value  = 10
    discretisation_step = 500
    num_epochs = 10000
                
    f = Function(max_value)
    g = Function(max_value)
    
    optimizer = optim.AdamW(f.parameters(), lr=0.1)
    optimizer2 = optim.AdamW(g.parameters(), lr=0.1)
    
    
    for epoch in range(num_epochs):
        total_loss = 0
        f.train()
        g.train()
        optimizer.zero_grad()
        optimizer2.zero_grad()
        my_loss = loss(discretisation_step)
        my_loss.backward()
        optimizer.step()
        optimizer2.step()
        if epoch%10 == 0 : print(epoch, -my_loss)

</code></pre>
What is the source code of torch.Tensor.cpu()

kevinsouth: <p>Recently I have found that as the amount of communication increases in torch.Tensor.cpu(), the bus bandwidth (is calculated by amount of communication / latency) will increase accordingly, but when the amount of communication is 16MB, there will be a sudden drop in bus bandwidth (the latency increases largely), so I want to see the implementation. But where can I find the source code of torch.Tensor.cpu()</p>

Soumya_Kundu: <p><a href="https://pytorch.org/docs/stable/generated/torch.Tensor.cpu.html#torch.Tensor.cpu" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.Tensor.cpu.html#torch.Tensor.cpu</a></p>

kevinsouth: <p>The document only describe the function of torch.Tensor.cpu, but this document doesn’t give the source code of this method.</p>

Soumya_Kundu: <p>True, couldn’t find it either.</p>

kevinsouth: <p>Actually I think I find some source code related to this function:<a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/core/Tensor.cpp#L47C10-L47C41" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/core/Tensor.cpp at main · pytorch/pytorch · GitHub</a>. This function will be called when calling cpu(), but this function will call another function that may be generated by code generation. So we cannot view the source code.</p>
Pytorch.org link error

zehonyi21: <p>I cant see one of source code( torch.Tensor.backward)<br>
where the link is<br>
(<a href="https://pytorch.org/docs/stable/_modules/torch/_tensor.html#Tensor.backward" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/_tensor.html#Tensor.backward</a>)</p>
<p>I can find it through my directory but I think It would be good if I can see this on <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a><br>
Please tell me if it works</p>
Resnet inconsistency between train and eval mode

William-HYWu: <p>I’m trying to implement the Resnet in torch. But I found the output of the forward pass varies greatly between train and eval mode. Since the train and eval mode doesn’t affect anything besides batch norm and dropout, I don’t know if the results make sense.</p>
<p>Below is my test code:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torchvision import models

class resnet_lstm(torch.nn.Module):
    def __init__(self):
        super(resnet_lstm, self).__init__()
        resnet = models.resnet50(pretrained=True)
        self.share = torch.nn.Sequential()
        self.share.add_module("conv1", resnet.conv1)
        self.share.add_module("bn1", resnet.bn1)  # Use BatchNorm3d
        self.share.add_module("relu", resnet.relu)
        self.share.add_module("maxpool", resnet.maxpool)
        self.share.add_module("layer1", resnet.layer1)
        self.share.add_module("layer2", resnet.layer2)
        self.share.add_module("layer3", resnet.layer3)
        self.share.add_module("layer4", resnet.layer4)
        self.share.add_module("avgpool", resnet.avgpool)
        self.fc = nn.Sequential(nn.Linear(2048, 512),
                                nn.ReLU(),
                                nn.Linear(512, 7))

    def forward(self, x):
        x = x.view(-1, 3, 224, 224)
        x = self.share(x)
        return x
    
model = resnet_lstm()

input_ = torch.randn(1, 3, 224, 224)
model.train()
print("train mode output", model(input_))
model.eval()
print("eval mode output", model(input_))

</code></pre>
<p>Terminal output:</p>
<pre data-code-wrap="none"><code class="lang-none">train mode output tensor([[[[0.3603]],

         [[0.5518]],

         [[0.4599]],

         ...,

         [[0.3381]],

         [[0.4445]],

         [[0.3481]]]], grad_fn=&lt;MeanBackward1&gt;)
eval mode output tensor([[[[0.1582]],

         [[0.1822]],

         [[0.0000]],

         ...,

         [[0.0567]],

         [[0.0054]],

         [[0.3605]]]], grad_fn=&lt;MeanBackward1&gt;)
</code></pre>
<p>As you can see, the output of the train and eval mode is very different from each other. Would this damage the performance?</p>
Problem with WeightedRandomSampler

Ronald_Lauer: <p>Hi all, I’m a newbie to machine learning and just making my first steps.<br>
I have an imbalanced dataset, and I’m using WeightedRandomSampler to compensate for that.</p>
<p>I had the idea that there might be an issue with the WRS, so I had my dataset write debug data to a file so that I could analyse the sampling. It became clear that the WRS did compensate for the imbalance just fine, but at a price, namely a 6-fold reduction of the data diversity, meaning that it effectively only uses a 6th of the entire dataset. This factor 6 is no coincidence, because it is also the difference between the classes with the most data, and the classes with the least data.<br>
Could it be that this is some sort of bug in WeigthedRandomSampler? Like I said, I am just a newbie, and this is my first model, so it is possible that I made a mistake somewhere.</p>

Soumya_Kundu: <p>The diversity of the majority class is sacrificed, not the entire dataset (assuming you’re inversely weighed based on class size).</p>
<p>What you are noticing may be the result of your first batch which may not be your entire dataset.</p>
<p>Could you potentially show some code/prints to verify this?</p>

Ronald_Lauer: <p>Apologies, I made a silly error. There is no problem with Weighted Random Sampler after all.<br>
It does exactly what it should do.</p>
<p>So there must be a different reason why my model doesn’t work. I’ll check everything again and if necessary I will start over with a somewhat simplified version and less data.</p>
Finding all differences in two dataframes

WizardlyLuke: <p>I’m trying to train two identical models on two almost identical data sets. The problem is though the training starts out completely different and has a completely different outcome. My two datasets have the exact number of items and only differ slightly in some values since they are calculated at two different sources, but the differences are only on a dozen or so of about ~100 columns and differ by either more decimal places or by a very small amount. e.g. 1566.50125 vs 1566.5 and 1723.9 vs 1714.75.</p>
<p>Are there any good methods for finding all of the differences between two dataframes? Order, types, and data differences. I’m thinking there is something else I’m overlooking between the two.</p>
<p>Thanks</p>