akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to check what is amount of padding used when we use padding = "same"?

username: jaytimbadia
cooked: <p>Hello there,</p>
<p>I was wondering what is the amount of padding used when I put padding = “same” in conv2D!</p>
<p>I have an image.</p>
<pre><code class="lang-auto">import torch 

image = torch.randn((1, 3 ,224, 224))

conv2 = torch.nn.Conv2d(
    in_channels=3, 
    out_channels=5, 
    kernel_size=16,
    stride=1,
    padding='same'
)

conv2(image).shape  # 1, 5, 224, 224 
</code></pre>
<p>But how to know what amount of padding has been used?<br>
Since I have manually checked using padding = 4,5,6,7,8.<br>
None of the padding gives 224 as output shape.</p>
<p>So that means is it using fractional padding?</p>
<p>Jay</p>

username: Soumya_Kundu
cooked: <aside class="quote no-group" data-username="jaytimbadia" data-post="1" data-topic="210041">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/bcef8e/48.png" class="avatar"> jaytimbadia:</div>
<blockquote>
<p>So that means is it using fractional padding?</p>
</blockquote>
</aside>
<p>Yes!</p>
<aside class="quote no-group" data-username="jaytimbadia" data-post="1" data-topic="210041">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/bcef8e/48.png" class="avatar"> jaytimbadia:</div>
<blockquote>
<p>But how to know what amount of padding has been used?</p>
</blockquote>
</aside>
<p>Not sure unfortunately. Afaik, it’s been handled internally. There may be some way to print it.</p>
<p>I think this is a nice hardcoded way to check?</p>
<pre><code class="lang-auto">kernel_size = conv2.kernel_size
if isinstance(kernel_size, int):
    padding = kernel_size // 2
else:
    padding = (kernel_size[0] // 2, kernel_size[1] // 2)

print(f"Padding used: {padding}")

output = conv2(image)
print(f"Output shape: {output.shape}")
</code></pre>

username: jaytimbadia
cooked: <p>Thanks <a class="mention" href="/u/soumya_kundu">@Soumya_Kundu</a> for your reply!</p>
<p>With the code you provided, value of padding is coming as 8, but if we put 8 back, its nor preserving the image dimension.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c76a2f8fe066e58a2421cdbe571223da77d8bf7.png" alt="image" data-base62-sha1="1MfUg69ymov35Tahc28aL7oCD8X" width="522" height="436"></p>
<p>Jay</p>

username: jaytimbadia
cooked: <p>Can anyone please help here?</p>

username: ptrblck
cooked: <p>You could check <code>conv2._reversed_padding_repeated_twice</code> and use <code>F.pad</code> before applying the convolution to get the same output, but keep in mind it’s an internal attribute and could change in the future.</p>
IterableDataset - why is len required?

username: BradenC
cooked: <p>Hello,</p>
<p>I am having trouble with an IterableDataset. Pytorch wants me to tell it the <em>len</em> of my datasource. But as it’s an iterable datasource, why would that matter? If I can generate data at least as fast as I consume it, the datasource could be functionally infinite.</p>
<p>My situation is that I have data in many files. I have enough memory to hold any individual file, but not all of them at once. My current approach is below. In short: the iterable hands out data stored in memory. When the iterable runs out, it refreshes by loading data from the next file into memory.</p>
<p>If I set len to 0, it doesn’t load anything. So, it checks the len <em>before</em> loading? That seems fundamentally opposed to what Iterables are.</p>
<p>If I leave len as 1, things seem to work, but I get several annoying warnings:</p>
<p><code>UserWarning: Length of IterableDataset &lt;somestuff.MyDataSet object at 0x000002348AD5AF08&gt; was reported to be 1 (when accessing len(dataloader)), but 2 samples have been fetched.    warnings.warn(warn_msg)</code></p>
<p>If I set len to some very large number, things also seem to work and the warnings go away. I feel gross about it tho.</p>
<p>Why do I have to provide the len at all?</p>
<p>I found <a href="https://github.com/pytorch/pytorch/pull/23587" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/23587</a>, but that answer is unsatisfactory to me.</p>
<p>My code:</p>
<pre><code class="lang-auto">class MyDataSet(IterableDataset):
    def __init__(self, sample_dir):
        self.sample_dir = sample_dir
        self.len = 1

    def __len__(self):
        return self.len

    def get_sample(self):
        for f in [self.sample_dir + '/' + filename for filename in os.listdir(self.sample_dir)]:
            data = torch.load(f)
            for item in data:
                yield item

    def __iter__(self):
        return self.get_sample()

...

    dataset = MyDataSet('MyDir/samples')
    data_loader = DataLoader(
        dataset=dataset,
        batch_size=1,
        num_workers=0
    )
</code></pre>

username: StefanK
cooked: <p>We might set batch_size = None.<br>
Otherwise, the DataLoader wants to count the item to build a batch.</p>
Beginner tutorial on similar image classification

username: Vinicius_Carvalho
cooked: <p>Hi folks, I currently use a very primitive solution to match similar images. The goal is to find similar pages of scanned content and apply tags on it using previous known content. I compute the perceptual hashing of all my images using <a href="https://pypi.org/project/ImageHash/" rel="noopener nofollow ugc">ImageHash</a>, I then store that data in a structure and I compute the hammingDistance of the new image against all the stored hashes. It works, has a relative good precision but it can’t really scale.</p>
<p>I’m just starting with pytorch but I found tutorials on using resnet and extract features from the images. So I tried that:</p>
<pre><code class="lang-auto">class FeatureExtractor:
    def __init__(self):
        self.model = models.resnet50()
        self.model = torch.nn.Sequential(*(list(self.model.children())[:-1]))
        self.model.eval()
        self.preprocess = transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        ])

    def extract(self, image_path: str) -&gt; np.ndarray:
        # Load and preprocess the image
        image = Image.open(image_path).convert('RGB')  # Ensure image is in RGB format
        image = self.preprocess(image).unsqueeze(0)  # Add batch dimension

        img = Image.open(image_path).convert("RGB")
        img_t = self.preprocess(img).unsqueeze(0)  # Add batch dimension

        # Disable gradient computation for inference
        with torch.no_grad():
            features = self.model(img_t)  # Get features from the modified ResNet

        # Return the features as a list
        return features.numpy().flatten().tolist()

</code></pre>
<p>So I got all my vectors and stored them in a vector database. The problem however is that I noticed that the cosine similarity is really close to whatever image I present. For example<br>
considering:<br>
image1 : source<br>
image2: very similar<br>
image3: random image with almost no visual identical features</p>
<p>similarity[1,2]: 0.9998799064636614<br>
similarity[1,3]: 0.9987749858928928</p>
<p>Do you have any suggestions of other approaches I could use? Other models or other ways to extract features from those images that could help me find similar images to one presented?</p>
<p>Thank you</p>
Multi-label classification for model with positive outputs

username: marziehoghbaie
cooked: <p>Hi,<br>
I have a network (this is the model: <a href="https://github.com/M-Nauta/PIPNet/tree/main/pipnet" class="inline-onebox" rel="noopener nofollow ugc">PIPNet/pipnet at main · M-Nauta/PIPNet · GitHub</a>) whose output is always positive, but I like to use this model for multi-label task where I have to use BCEwithlogitloss. But with this always-positive output, the built-in sigmoid function does not work. I like to know if someone has any recommendations.<br>
My output is like this l = torch.tensor([[3.1847, 3.1812, 3.1875, 3.1981, 3.1937, 3.1888],<br>
[3.4225, 3.4238, 3.4206, 3.4222, 3.4128, 3.4266],<br>
[3.3188, 3.3171, 3.3155, 3.3196, 3.3170, 3.3163],<br>
[3.2350, 3.2372, 3.2413, 3.2451, 3.2469, 3.2372],<br>
[3.4386, 3.4308, 3.4364, 3.4415, 3.4254, 3.4318]])</p>
DDP Multi-node Training Hangs at TCPStore Initialization

username: Rishikesh_Dandekar
cooked: <p>I am trying to train a model using Distributed Data Parallel (DDP) across multiple nodes. However, the training process hangs at the TCPStore initialization in the <code>static_tcp_rendezvous.py</code> script inside next_rendezous method implementation (line 55), and nothing gets printed after that. It seems like the content of my training script (<code>train.py</code>) never gets executed when using multiple nodes.</p>
<p>When I run the same script with DDP on a single node, the training runs successfully without any issues.</p>
<p>Command I use to run the script:<br>
torchrun --nproc_per_node=1 --nnodes=2 --node_rank=0 --master_addr=“192.168.3.23” --master_port=1234 multinode.py</p>
<p>Pytorch version: 2.4.1</p>

username: H-Huang
cooked: <p>Since you are using a single node, try setting --master_addr=“localhost”, probably not all processes can communicate via “192.168.3.23”</p>
Unable to use libtorch with C++

username: DaiCC-Aurora
cooked: <p>F:/sigma/sigma/3rdparty/libtorch/include/c10/core/DynamicCast.h: In function ‘dest_t c10::fetch_and_cast(c10::ScalarType, const void*) [with dest_t = c10::qint8]’:<br>
F:/sigma/sigma/3rdparty/libtorch/include/c10/macros/Macros.h:401:5: error: ‘__assert_fail’ was not declared in this scope<br>
401 |     __assert_fail(                                                       <br>
|     ^~~~~~~~~~~~~</p>
<p>Codes:<br>
<span class="hashtag-raw">#include</span> <br>
<span class="hashtag-raw">#include</span> &lt;torch/script.h&gt;</p>
<p>using namespace std;</p>
<p>int main() {<br>
torch::Tensor tensor = torch::rand({3, 3});<br>
torch::Tensor tensor1 = torch::zeros({5, 5});<br>
cout &lt;&lt; tensor &lt;&lt; endl &lt;&lt; tensor1 &lt;&lt; endl;</p>
<pre><code>return 0;
</code></pre>
<p>}</p>
<p>Windows11<br>
libtorch-2.4.0-cpu<br>
How to compiler this file?</p>

username: Dirk10001
cooked: <p>I compiled it with VS 2019, on Win 11.</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;               // torch
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto"> 0.2519  0.6636  0.9867
 0.1741  0.1896  0.6346
 0.4020  0.5482  0.3991
[ CPUFloatType{3,3} ]
 0  0  0  0  0
 0  0  0  0  0
 0  0  0  0  0
 0  0  0  0  0
 0  0  0  0  0
[ CPUFloatType{5,5} ]
</code></pre>
nn.TransformerEncoder OOM with no_grad + eval + mask

username: 5had3z
cooked: <p>nn.TransfomerEncoder uses significantly more memory and OOM’s in a no_grad + eval scenario. I understand memory + speed trade offs are made some for the fast paths in the codebase, however I think this is exessive if it is to the point where OOM occurs in evaluation and not in training with autograd running. Once the no_grad + mask + eval trifecta occurs, 3GB usage ballons to OOM’ing an RTX 3090 (the next request is +8GiB at OOM). If you subtract any one of these conditions, the issue does not occur.</p>
<p>torch version: 2.4.0+cu124<br>
GPU: RTX 3090</p>
<pre data-code-wrap="python"><code class="lang-python"># Minimum reproduceable example
import torch
from torch import nn


def make_causal_mask(time: int, n_tokens: int, device: torch.device | None):
    """Make causal mask where multiple tokens are in each timestep"""
    mask = (
        torch.arange(time * n_tokens, device=device)
        .unsqueeze(0)
        .repeat(time * n_tokens, 1)
        // n_tokens
    )
    indices = torch.arange(time * n_tokens, device=device).unsqueeze(-1) // n_tokens
    final = mask &gt; indices  # True is not allowed to attend
    return final


mod = nn.TransformerEncoder(
    nn.TransformerEncoderLayer(128, 8, batch_first=True), 2
).cuda()
data = torch.randn(16, 4096, 128).cuda()
T = 512
N = 8
mask = make_causal_mask(T, N, device=data.device)

res = mod(data, mask=mask)
torch.cuda.synchronize()
print(f"training: {torch.cuda.memory_allocated() // 1e9} GB")

del res
torch.cuda.empty_cache()

with torch.no_grad():
    mod = mod.eval()
    res = mod(data, mask=mask)
    print(f"eval+no_grad+mask: {torch.cuda.memory_allocated() // 1e9} GB")
</code></pre>
Simplest Conv2D sample fail

username: Dirk10001
cooked: <p>Hi all,<br>
I made a simple sample. Conv an 8x8 image with a 4x4 kernel.</p>
<pre><code class="lang-auto">int main(int argc, const char *argv[]){

    // test
    torch::Tensor img = torch::ones({ 1,1,8,8 }, torch::TensorOptions(torch::kFloat).device(torch::kCPU));

    float* pimg = (float*)img.data_ptr();

    for (int i = 0; i &lt; 64; i++)
    {
        pimg[i] = (float)i;
    }

    torch::nn::Conv2d conv(torch::nn::Conv2dOptions(1, 1, 4).padding(1).stride(2));
    conv-&gt;to(torch::kCPU);

    torch::Tensor weights = conv-&gt;weight;
    float* data = (float*)weights.data_ptr();
    for (int i = 0; i &lt; 16; i++)
    {
        data[i] = 1.0;
    }

    torch::Tensor bias = conv-&gt;bias;
    data = (float*)bias.data_ptr();
    data[0] = 0.0;

    torch::Tensor output = conv-&gt;forward(img);
</code></pre>
<p>The sample fails at conv-&gt;forward with</p>
<pre><code class="lang-auto">Assertion failed: nthr_ == nthr, file C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\third_party\ideep\mkl-dnn\src\common/dnnl_thread.hpp, line 308
</code></pre>
<p>What is wrong with the simple sample?</p>
<p>Many thanks for your help.</p>
Support for NumPy-like structured array tensors?

username: rademacher-p
cooked: <p>Is there any interest in adding tensors with “fields” - that is, tensor objects like NumPy <a href="https://numpy.org/doc/stable/user/basics.rec.html" rel="noopener nofollow ugc">structured arrays</a>?</p>
<p>When combining multiple input sources, I find myself desiring a single array, rather than a <code>dict</code> of arrays; it can be more convenient for slicing, etc. Another common use case for me is having a single array for supervised learning data, with separate <code>x</code> and <code>y</code> fields.</p>
<p>Thank you in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

username: xinming_wang
cooked: <p>Hope this feature to be added into pytorch. Its meaningful for other tasks when interacting with tabular data, like pandas dataframe, structured arrays.</p>
How to add new Head by replacing the current Head of trained model?

username: mbkamran
cooked: <p>Since I have converted my model from onnx to PyTorch using the onnx2torch library, I do not have access to model definition and cannot hence modify it. In tf/keras, we are able to build a model by combining the input and output of functional layers, hence in my case, we can find out the name of any intermediate layer and build a model backbone and a new head with that strategy. How can I do the same in PyTorch?</p>
<p>I have tried using model.layer = CustomHead() as <a class="mention" href="/u/ptrblck">@ptrblck</a> has suggested, however, that will only replace one module (layer) with CustomHead. I want to break the entire trained model and attach a new head to it. For reference, following is an image of onnx graph depicting the where I wish to break the model and attach a custom head:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec4b3dd9cd8233909ec458f5a8661b7d2a8835e4.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec4b3dd9cd8233909ec458f5a8661b7d2a8835e4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec4b3dd9cd8233909ec458f5a8661b7d2a8835e4_2_369x500.png" alt="image" data-base62-sha1="xIlT1f9hjg6H4RQ0FXoqTneTaMk" width="369" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec4b3dd9cd8233909ec458f5a8661b7d2a8835e4_2_369x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec4b3dd9cd8233909ec458f5a8661b7d2a8835e4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec4b3dd9cd8233909ec458f5a8661b7d2a8835e4.png 2x" data-dominant-color="404041"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">500×677 28 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I wish to break the model at Clip (which is ReLU) and replace all the succeeding modules with my CustomHead().</p>
PPO and DDPG with Mujoco input frames

username: Gianluca_Maselli
cooked: <p>Hi,<br>
I’m trying to implement the PPO algorithm on a simple custom Mujoco environment where a Tiago robot should push a cube in a circular area. I want to implement the algorithm by relying on input frames given by env.render(). However, since I did not find any implementation on GitHub in continuous action spaces using frames I wanted to ask if the same strategy applied in the original DDPG paper works also for PPO.</p>
<p>In particular, the paper states:<br>
“…In all tasks, we ran experiments using both a low-dimensional state description (such as joint angles and positions) and high-dimensional renderings of the environment. As in DQN (Mnih et al., 2013; 2015), in order to make the problems approximately fully observable in the high dimensional environment we used action repeats. For each timestep of the agent, we step the simulation 3 timesteps, repeating the agent’s action and rendering each time. Thus the observation reported to the agent contains 9 feature maps (the RGB of each of the 3 renderings) which allows the agent to infer velocities using the differences between frames. The frames were downsampled to 64x64 pixels and the 8-bit RGB values were converted to floating point scaled to [0, 1]. See supplementary information for details of our network structure and hyperparameters…”</p>
<p>I also followed the same strategy for PPO because in Atari Games, the same methodology of stacking frames is used no matter the algorithm you use (e.g. A2C, A3C and DQN) and I thought that this assumption is true also for continuous action spaces.</p>
<p>Do you think it could work or is this only for DDPG?</p>
Jit,PytorchVideo,Deployment

username: Renuraj
cooked: <p>Problem: model_transmute_deploy inference speed is lesser than model_transmute.<br>
model_transmute_deploy size is 12.2 kb<br>
model_transmute size is 36.2 kb</p>
<p>Multiple input tensor size experimented with : [1,3,16,256,256], [1,3,4,256,256], [1,3,16,6,6]<br>
model_transmute_deploy inference time: 5.375,1.102, 0.0627sec<br>
model_transmute inference time:0.1049,0.0107,0.0245 sec</p>
<p>provide the reason for this is to why its this happening and also the optimal solutions to overcome this issue<br>
pytorch video accelerator script link: <a href="https://pytorchvideo.org/docs/tutorial_accelerator_use_model_transmuter" class="inline-onebox" rel="noopener nofollow ugc">Accelerate your model with model transmuter in PytorchVideo/Accelerator · PyTorchVideo</a></p>
How does one download dataset from gdrive using pytorch?

username: pinocchio
cooked: <p>I used <code>download_file_from_google_drive</code> before but now I get this error:</p>
<pre><code class="lang-auto">-&gt; starting Download
Traceback (most recent call last):
  File "download_miniImagent.py", line 53, in &lt;module&gt;
    download_and_extract_miniImagenet(root=root)
  File "download_miniImagent.py", line 35, in download_and_extract_miniImagenet
    from torchvision.datasets.utils import download_file_from_google_drive, extract_archive
ImportError: cannot import name 'download_file_from_google_drive' from 'torchvision.datasets.utils' (/Users/Me/anaconda3/envs/project/lib/python3.7/site-packages/torchvision/datasets/utils.py)
</code></pre>
<p>why?</p>
<p>I see it on the github:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/utils.py#L136" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/utils.py#L136" target="_blank" rel="nofollow noopener">pytorch/vision/blob/master/torchvision/datasets/utils.py#L136</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="126" style="counter-reset: li-counter 125 ;">
<li>            os.listdir(root)</li>
<li>        )</li>
<li>    )</li>
<li>
</li>
<li>    if prefix is True:</li>
<li>        files = [os.path.join(root, d) for d in files]</li>
<li>
</li>
<li>    return files</li>
<li>
</li>
<li>
</li>
<li class="selected">def download_file_from_google_drive(file_id, root, filename=None, md5=None):</li>
<li>    """Download a Google Drive file from  and place it in root.</li>
<li>
</li>
<li>    Args:</li>
<li>        file_id (str): id of file to be downloaded</li>
<li>        root (str): Directory to place downloaded file in</li>
<li>        filename (str, optional): Name to save the file under. If None, use the id of the file.</li>
<li>        md5 (str, optional): MD5 checksum of the download. If None, do not check</li>
<li>    """</li>
<li>    # Based on https://stackoverflow.com/questions/38511444/python-download-files-from-google-drive-using-url</li>
<li>    import requests</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>but it’s not on my downloaded version of torchvision, why?</p>

username: Ajith
cooked: <p>You can use below code with filename</p>
<pre><code class="lang-auto">download_and_extract_archive(
            url="https://drive.google.com/file/d/abc/view?usp=sharing",
            # url=f"https://drive.google.com/uc?id=1By9UUbUAyiTJ2LpX3jlsRXmYOEbMbuO8",
            download_root=self._dl_path,
            filename="archive.zip",
            remove_finished=True
        )
</code></pre>
<p>kindly replace your fileid</p>
Pytorch Installation SSH

username: Raminguyen12
cooked: <p>Hello everyone,</p>
<p>I am using my virtual computer connected through ssh. I want to install pytorch for my llava model but I am not sure which version is good with my spec now. Wishing the helps from all.</p>
<p>The problem is I am encountering now, cuda version 12.2, but the cuda compilation tool 10.1, and I have my own new cuda environment.</p>

username: ptrblck
cooked: <p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension. You need to install a compatible NVIDIA driver, the PyTorch binaries afterwards, and it should work.</p>
Grad is None for nn.Parameter that is used outside nn.module

username: verku
cooked: <p>Hi!</p>
<p>I have defined a nn.Parameter, but the value is not updating. I wonder if this in because some of the calculations done with this parameter are outside the forward loop of the module. I have classes that are not nn.Modules, which I call from the forward loop and give the parameter as variable. Can this be a cause for the gradient being none?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/verku">@verku</a>,</p>
<p>You need to share your code, and make sure to wrap it in 3 backticks ``` !</p>
Why adding boolen to a tensor adds 1 to all the elements of tensor?

username: shrebox
cooked: <p>I’m unable to understand why <code>torch.ones(2,2) + True</code> element-wise adds <code>+1</code> to all the values?</p>

username: AlphaBetaGamma96
cooked: <p>I’d assume it’s casting the boolean to an int, which in the case of <code>True</code> is 1 and <code>False</code> is 0. For example,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = True
&gt;&gt;&gt; int(x) #returns 1
</code></pre>
How can I load a sharded FSDP state dict to a single rank?

username: Vedant_Roy
cooked: <p>I understand how to load/save sharded FSDP state dicts from this tutorial: <a href="https://www.youtube.com/watch?v=uBLgprhJn_8" rel="noopener nofollow ugc">https://www.youtube.com/watch?v=uBLgprhJn_8</a>.</p>
<p>But, how can I load a sharded state dict onto a single rank?</p>
How can I avoid initializing parameters in Pytorch?

username: Vedant_Roy
cooked: <p>How do I avoid having my linear layers be initialized when I load my model? E.g, the following stacktrace shows that when creating a linear layer, <code>self.reset_parameters()</code> is called.</p>
<p>Given that I am immediately going to load weights from a checkpoint, I would like to avoid initializing the weights. Ideally, they’d just be filled with garbage memory.</p>
<pre><code class="lang-auto"> File "layers.py", line 91, in __init__
    self.w2 = nn.Linear(hidden_size, in_features, bias=False)
  File "/tmp/env/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 104, in __init__
    self.reset_parameters()
  File "/tmp/env/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 110, in reset_parameters
    init.kaiming_uniform_(self.weight, a=math.sqrt(5))
  File "/tmp/env/lib/python3.10/site-packages/torch/nn/init.py", line 460, in kaiming_uniform_
    return tensor.uniform_(-bound, bound, generator=generator)
</code></pre>

username: ptrblck
cooked: <p>You can use the <code>skip_init</code> method as described <a href="https://pytorch.org/tutorials/prototype/skip_param_init.html">here</a>.</p>
In place operation related to scaled_dot_product_attention

username: PeiZ
cooked: <p>Hi, we’ve been developing a large piece of code for a couple of months and recently we ran into in-place operations errors (<code>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 6, 3584, 64]], which is output 0 of ScaledDotProductEfficientAttentionBackward0, is at version 4; expected version 0 instead.</code>) when we’re trying to run the code on a difference device. This is the first time ever we saw this error. The same piece of code runs fine on the other devices. I did some research, it seems in-place operation errors are very basic and should not change with devices? Any help/input would be appreciated! It’s so puzzling.</p>
<p>If we replace the line <code>x = F.scaled_dot_product_attention(q, k, v)</code> with</p>
<blockquote>
<p>scale_factor = 1 / math.sqrt(q.size(-1))<br>
attn_weight = q @ k.transpose(-2, -1) * scale_factor<br>
attn_weight = torch.softmax(attn_weight, dim=-1)<br>
attn_weight = torch.dropout(attn_weight, 0.0, train=True)<br>
x = attn_weight @ v</p>
</blockquote>
<p>, the code runs fine too.</p>
<p>Update: the error is gone after we add clone (), <code>        x = F.scaled_dot_product_attention(q, k, v, attn_mask=mask2d_padding).clone()</code>. But still, any explanation of what’s going on would be appreciated</p>
Missing librt.so.1, libdl.so.2, libz.so.1 and libpthread.so.0 in PyTorch Source Build (aarch64)

username: vishwascm
cooked: <p>Hello,</p>
<p>I am building PyTorch from source on an aarch64 system and have noticed performance differences of Hugging face models(Bert, resnet etc) when running on 64 threads/cores, compared to the pip-installed version (2.4.0). Upon inspecting the shared libraries linked to <code>libtorch_cpu.so</code>, I found that the pip-installed version links to <code>librt.so.1</code>, <code>libdl.so.2</code>, libz.so.1 and <code>libpthread.so.0</code>, but these libraries are missing in the source build.<br>
I used following method to build:<br>
[<a href="https://github.com/pytorch/builder/blob/main/aarch64_linux/aarch64_wheel_ci_build.py" class="inline-onebox" rel="noopener nofollow ugc">builder/aarch64_linux/aarch64_wheel_ci_build.py at main · pytorch/builder · GitHub</a>].<br>
Please help me out to build pytorch similar to pip-installed version including all missing libraries.</p>
Simple Tutorial for TPU usage with pytorch

username: Schroter
cooked: <p>Hi All,<br>
I would like to know if there is any good tutorials on how to utilise TPUs for pytorch models please.<br>
Thanks &amp; Best Regards<br>
AMJS</p>

username: shauheen
cooked: <p>Hi,</p>
<p>There are a few tutorials <a href="https://cloud.google.com/tpu/docs/tutorials/transformer-pytorch" rel="noopener nofollow ugc">here</a>. Please let the team know if you run into <a href="https://github.com/pytorch/xla/issues/new/choose" rel="noopener nofollow ugc">any issues</a>, or if you are looking for a particular model.</p>
<p>Thanks,<br>
Shauheen,</p>

username: Bjorn_Lindqvist
cooked: <p>The link to the tutorials does not work anymore.</p>
`training` property of complied models is always `True`

username: kad
cooked: <p>I noticed that I can’t set the <code>training</code> property to <code>False</code> for modules that I have compiled, regardless of the state of the property when the module was compiled.</p>
<p>Is this expected behaviour?</p>
<p>A demo:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

print(torch.__version__)

class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(16, 1)

    def forward(self, x):
        return self.fc(x)

print("--------standard--------")
m = M()
print(f"m.training = {m.training}\t← default")
m.train(False)
print(f"m.training = {m.training}\t← m.train(False)")

print("--------compiled--------")
m = M()
m = torch.compile(m)
print(f"m.training = {m.training}\t← default")
m.train(False)
print(f"m.training = {m.training}\t← m.train(False)")

print("------compiled (eval)---")
m = M()
m.train(False)
m = torch.compile(m)
print(f"m.training = {m.training}\t← default")
m.train(False)
print(f"m.training = {m.training}\t← m.train(False)")
</code></pre>
<p>Output:</p>
<pre><code class="lang-plaintext">2.4.1+cu121
--------standard--------
m.training = True	← default
m.training = False	← m.train(False)
--------compiled--------
m.training = True	← default
m.training = True	← m.train(False)
------compiled (eval)---
m.training = True	← default
m.training = True	← m.train(False)
</code></pre>

username: ptrblck
cooked: <p>This is a known issue tracked <a href="https://github.com/pytorch/pytorch/issues/132986">here</a>.</p>

username: Chillee
cooked: <p>This appears to be fixed on master now: <a href="https://github.com/pytorch/pytorch/issues/132986#issuecomment-2372851399" class="inline-onebox" rel="noopener nofollow ugc">`.eval()` and `.train()` don't set value of `.training` properly on `torch.compile()` module · Issue #132986 · pytorch/pytorch · GitHub</a></p>
Transformer example: Position encoding function works only for even d_model?

username: Vandana_Rajan
cooked: <p>Hi,</p>
<p>I am using the transformer example from <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a><br>
I noticed that if ‘d_model’ in PositionalEncoding function is an odd number, then it throws an error.</p>
<p>‘*** RuntimeError: The expanded size of the tensor (31) must match the existing size (32) at non-singleton dimension 1.  Target sizes: [5000, 31].  Tensor sizes: [5000, 32]’</p>
<p>So, are we supposed to use only even numbered ‘d_dim’ or is it an issue with the function?</p>
<p>Thanks,</p>

username: Abhilash_Srivastava
cooked: <p>Interesting observation! The reason it’s happening is:</p>
<pre><code class="lang-auto">        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
</code></pre>
<p>Here, <code>torch.sin</code> is used for the even indexes and <code>torch.cos</code> is used for the odd indexes of the input dimension (size = d_model). So, if d_model is odd, there will be a dimension size mismatch (division by 2 leaves a remainder).</p>

username: Vandana_Rajan
cooked: <p>I changed the function like this and removed the error.</p>
<pre><code class="lang-auto">class PositionalEncoding(nn.Module):

    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        if d_model%2 != 0:
            pe[:, 1::2] = torch.cos(position * div_term)[:,0:-1]
        else:
            pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)
</code></pre>

username: Paul_Micaelli
cooked: <p>Still not fixed in Pytorch 1.9. If even d_model values are required then this should be an “assert …” line in the code. As it is the error message isn’t helpful</p>

username: yunhe_zhangyunhe
cooked: <aside class="quote no-group" data-username="Vandana_Rajan" data-post="3" data-topic="100986">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vandana_rajan/48/19834_2.png" class="avatar"> Vandana_Rajan:</div>
<blockquote>
<pre><code class="lang-auto">f d_model%2 != 0:
            pe[:, 1::2] = torch.cos(position * div_term)[:,0:-1]
        else:
            pe[:, 1::2] = torch.cos(position * div_term)
</code></pre>
</blockquote>
</aside>
<p>thank you,i fix the bug,but the last one didn`t have postional encoding ,i think it cant make a big difference</p>
How to change learning rate with Libtorch 1.5?

username: slaweks
cooked: <p>The old way<br>
trainer.options.learning_rate(new_lr_rate)<br>
does not work anymore.</p>

username: Hartbook
cooked: <p>The function name changed from learning_rate to lr ( <a href="https://github.com/pytorch/pytorch/releases" rel="nofollow noopener">https://github.com/pytorch/pytorch/releases</a> )</p>

username: slaweks
cooked: <p>Hi,<br>
But trainer (optimizer) does not have options anymore.</p>

username: Hartbook
cooked: <p>I think you have to use the optimizer constructor taking options as argument, that’s how I do it.<br>
I don’t know if you can change the options after the optimizer is constructed, maybe someone knows ?</p>

username: pascal.soveaux
cooked: <p>Hi,</p>
<p>Did you take a look into param_group option lr ?<br>
I have to implement a lr scheduler tomorrow, I will tell you.</p>
<p>Not so obvious because lr is not part of an options base class. I looked at the source code and here is what I’ve done:</p>
<pre><code class="lang-auto">template &lt;
    typename Optimizer = torch::optim::Adam,
    typename OptimizerOptions = torch::optim::AdamOptions&gt;
inline auto decay(
    Optimizer &amp;optimizer,
    double rate)
    -&gt; void
{
    for (auto &amp;group : optimizer.param_groups())
    {
        for (auto &amp;param : group.params())
        {
            if (!param.grad().defined())
                continue;

            auto &amp;options = static_cast&lt;OptimizerOptions &amp;&gt;(group.options());
            options.lr(options.lr() * (1.0 - rate));
        }
    }
}

</code></pre>
<p>I declared a spec for every of my optimizers. It works but I am not happy with this.</p>
<p>Pascal</p>

username: slaweks
cooked: <p>Thank you. Yes, it appears that the trick is:<br>
static_cast&lt;OptimizerOptions &amp;&gt;(group.options()).lr(new_lr_rate);<br>
Not obvious <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

username: pascal.soveaux
cooked: <p>I was wrong, I changed to:</p>
<pre><code class="lang-auto">    for (auto &amp;group : optimizer.param_groups())
    {
        if(group.has_options())
        {
            auto &amp;options = static_cast&lt;OptimizerOptions &amp;&gt;(group.options());
            options.lr(options.lr() * (1.0 - rate));
        }
    }
</code></pre>
<p>Pascal</p>

username: Bernhard_Jaeger
cooked: <p>It seems in libtorch 2.4 the function name need to be changed to:</p>
<p>options.set_lr(options.lr() * (1.0 - rate))</p>
How to perform backpropagation with noise

username: Toby-Kang
cooked: <p>Hi, I wonder if it is possible to backpropagate with noise. For example, I may have a network look like this:</p>
<pre><code class="lang-auto">class ThreeLayerNetwork(nn.Module):
    def __init__(self):
        super(ThreeLayerNetwork, self).__init__()
        self.fc1 = nn.Linear(1, 2)
        self.fc2 = nn.Linear(2, 1)
        
    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.fc2(x)
        return x
</code></pre>
<p>I believe the gradient of fc1 is calculated based on that of fc2 during backpropagation. Now I would like to add noise to the gradient of fc2, and let the gradient of fc1 calculated based on the noisy gradient of fc2. I wonder in which way can I achieve it.</p>
<p>Thanks a lot!</p>

username: ptrblck
cooked: <p>You could use <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook"><code>torch.nn.Module.register_full_backward_hook</code></a> to access the gradients and manipulate the <code>input_grad</code> before returning it.</p>
Improving Pytorch kernel performance

username: reikdas
cooked: <p>My code (for a somewhat degenerate matmul) looks like this:</p>
<pre><code class="lang-auto">def opt_foo(out, i1, i2, j1, j2, idx):
    out[i1*512:i2*512].view(i2-i1, 512).add_(torch.mm(A[idx:idx+(i2-i1)*(j2-j1)].view(j2-j1,i2-i1).t(), B[j1*512:j2*512].view(j2-j1, 512)))

opt_foo(out, 0, 200, 900, 1000, 0)
opt_foo(out, 200, 400, 1500, 1600, 20000)
opt_foo(out, 600, 800, 3100, 3200, 40000)
opt_foo(out, 1000, 1200, 1000, 1100, 60000)
opt_foo(out, 1000, 1200, 1500, 1600, 80000)
opt_foo(out, 1200, 1400, 400, 500, 100000)
opt_foo(out, 1400, 1600, 6400, 6500, 120000)
opt_foo(out, 1400, 1600, 7600, 7700, 140000)
opt_foo(out, 1600, 1800, 900, 1000, 160000)
opt_foo(out, 1600, 1800, 2400, 2500, 180000)
</code></pre>
<p><code>A</code>, <code>B</code> and <code>out</code> are all tensors on the GPU.<br>
The different views are to prevent transposing the tensor.<br>
<code>i1</code> and <code>i2</code> represent rows. Similarly, <code>j1</code> and <code>j2</code> represent columns. This is traversed in row major order, implying that the values of <code>i1</code> and <code>i2</code> are always either increasing or the same as the previous call. Also, <code>i2&gt;i1</code> and <code>j2&gt;j1</code> always because they represent the bounds.</p>
<p>Let’s say the current time taken by this is <code>x ms</code>.</p>
<p>I have tried inlining the kernel. Performance is still <code>~x ms</code>.</p>
<p>I have tried compiling <code>opt_foo</code> using <code>torch.compile</code> (outside of the part I time). It is much slower than <code>x ms</code>.</p>
<p>I tried storing the result in an intermediate list instead of updating <code>out</code> for each call. Note how there is a dependency between <code>out</code>s that have the same value of <code>i1</code>, so I did have to update <code>out</code> between calls having the same <code>i1</code>s. I also tried merging the update to <code>out</code> for calls where the <code>i2</code> of the previous call is the same as the <code>i1</code> of the new call. However, still the time taken was <code>~x ms</code>.</p>
<p>When profiling it using PyTorch, I saw the most time taken by CUDA was in <code>aten::mm</code> and <code>ampere_sgemm_64x32_sliced1x4_nt</code>.<br>
On the CPU, most time was taken in <code>aten::mm</code>, <code>aten::add_</code> and <code>cudaLaunchKernel</code>.</p>
<p>Does it mean anything that it seems that <code>aten::mm</code> seems to spend more time on the CPU than on CUDA?</p>
<p>How could I further optimize this?</p>
The Problem of Dimension Change for Additional Parameters in a Model due to DataParallel

username: Superlunary
cooked: <pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"] = '0,1,2'

class model1(nn.Module):
    def __init__(self, const, dim_in,feat_dim):
        super(model1, self).__init__()
        # encoder + head
        self.encoder = resnet50
        self.head = nn.Linear(dim_in, feat_dim)

        # additional learnable parameters
        self.additional_param = nn.Parameter(torch.randn(const, feat_dim)) # shape [const, feat_dim]
        nn.init.uniform_(self.additional_param, a=0.0, b=1.0)

    def forward(self, x):
        additional_param_out = self.additional_param

        x = self.encoder(x)
        x = self.head(x)
        x = F.normalize(x, dim=1)

        return x, additional_param_out 

class loss1(nn.Module):
    def __init__(self):
        super(loss1, self).__init__()
        pass
    def forward(self, results, additional_param):
        loss_val = (results*additional_param).sum()

        return loss_val

net = model1(const, dim_in, feat_dim)
net = torch.nn.DataParallel(net)
loss = loss1()

for batch_idx, (img, target) in enumerate(batch_iterator):

    result1, result2 = net(img) 
    # result1.shape = [batch size, feat_dim]
    # result2.shape = ?
    loss_val = loss(result1, result2 )
</code></pre>
<p>The code mentioned above is a somewhat simplified version of the original code. The key point is that, in addition to the encoder and projection head in the model, there is also a randomly initialized learnable parameter (<code>additional_param</code>), and this parameter is used in the loss calculation along with the encoded features.</p>
<p>When using a single GPU without <code>DataParallel</code>, there is no issue. However, when using multiple GPUs with <code>DataParallel</code> as in the above code, the shape of <code>result2</code> changes, causing problems.</p>
<p>For example, when <code>const = batch size</code>, <code>additional_param</code> would have the shape <code>[batch size, feat_dim]</code>. But when using 3 GPUs, <code>result2.shape</code> becomes <code>[3 * (batch size), feat_dim]</code> (instead of <code>[3, batch size, feat_dim]</code>), which causes issues during loss computation.</p>
<p>A simple solution could be to use <code>result2[:(batch size)]</code> in the loss calculation, but I am not sure if this is the correct approach. Could you suggest an appropriate way to resolve this issue?</p>

username: ptrblck
cooked: <p>Your explanation of the root cause for this issue sounds correct as the global batch size will be divided by the number of GPUs creating the local batch size inside the <code>forward</code> method. However, parameters etc. are usually not initialized using the batch size so could you explain why you are using the batch size in <code>dim0</code>? Are you computing a per-batch loss? If so, outside of the <code>forward</code> method your model output should have the global batch size again.</p>
Cuda 12.4 unavailable after install in conda environmnt (aarch64 nvidia GH)

username: dunlopj
cooked: <p>So I successfully installed pytorch with cuda on aarch64 in a conda environment using</p>
<p>“pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu124" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu124</a>”</p>
<p>After initially successfully verifying that pytorch cuda was installed and working, subsequent cuda checks returned false. Any idea what is happening? Do I need to change some path variable?</p>
Training gets slower when resuming from a checkpoint

username: Subhash
cooked: <p>I am training a UNet 3d segmentation model using DDP on a multi-node GPU cluster (8 single-gpu nodes). The network has approximately ~60M parameters, and I am training with FP16.</p>
<p>I am seeing that the training performance significantly slows down (by a factor of 2-3) when resuming from a checkpoint. When I start a new training it takes an average of 35 minutes per epoch. But, when I restart the training from a previous checkpoint it takes over 90 minutes per epoch. There are no other changes I make to the training configuration, except that I load from a prior checkpoint.<br>
I have tested this several times and the behavior is consistent.<br>
Would really appreciate any help with this.</p>

username: ptrblck
cooked: <p>That’s a strange issue. Could you profile your workload (a few training iterations, e.g. 10) using Nsight Systems for both use cases and check where the bottleneck in your code is (or which operations are slowing down)?</p>

username: Pranil_Patil
cooked: <p>It’s the same issue for me with Huggingface’s SFT Trainer. I am using 2 A100 GPUs. When I started from checkpoint, time per iteration increased almost five times.</p>
RuntimeError: Function 'MkldnnRnnLayerBackward0' returned nan values in its 1th output when using set_detect_anomaly True

username: OkabeSan
cooked: <p>Hi.</p>
<p>When I am running my RL project, it gives me nan (The Error below) after a few iterations while I clipped the gradient of my model using this:<br>
<code>torch.nn.utils.clip_grad_norm_(self.critic_local1.parameters(), max_norm =4)</code></p>
<p>The Error:</p>
<pre><code class="lang-auto">*ValueError: Expected parameter probs (Tensor of shape (1, 45)) of distribution Categorical(probs: torch.Size([1, 45])) to satisfy the constraint Simplex(), but found invalid values:*
*tensor([[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]], grad_fn=&lt;DivBackward0&gt;)*
</code></pre>
<p>So I used <strong><code>torch.autograd.set_detect_anomaly(True)</code></strong> to detect where is the anomaly and it says:<br>
<strong><code>Function 'MkldnnRnnLayerBackward0' returned nan values in its 1th output</code></strong><br>
I did not find it anywhere what is this error and <code>MkldnnRnn</code> and what is the root of the error <code>nan</code>? Because I thought that the error nan should be solved when we clip the gradients.</p>
<p>The issue is that the code runs without errors on my laptop, but it raises an error when executed on the server. I don’t believe this is related to package versions.</p>
Unable to import torch.distributed.HashStore

username: SwayStar123
cooked: <pre><code class="lang-auto">import torch
import torch.distributed as dist

print("Is dist available: ", dist.is_available())
print("Pytorch version: ", torch.__version__)

store = dist.HashStore()
# store can be used from other threads
# Use any of the store methods after initialization
store.set("first_key", "first_value")
</code></pre>
<p>Output</p>
<pre><code class="lang-auto">Is dist available:  True
Pytorch version:  2.4.1+cu124
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[12], line 7
      4 print("Is dist available: ", dist.is_available())
      5 print("Pytorch version: ", torch.__version__)
----&gt; 7 store = dist.HashStore()
      8 # store can be used from other threads
      9 # Use any of the store methods after initialization
     10 store.set("first_key", "first_value")

AttributeError: module 'torch.distributed' has no attribute 'HashStore'
</code></pre>

username: SwayStar123
cooked: <p>Looks like HashStore doesnt support windows</p>
<pre><code class="lang-auto">    if sys.platform != "win32":
        from torch._C._distributed_c10d import (
            HashStore,
            _round_robin_process_groups,
        )
</code></pre>
<p>RIP</p>
Unique function for float tensors

username: Ameen_Ali
cooked: <p>hello.<br>
I have a float tensor and i am welling to run torch.unique on it , however it does not work but if i convert it to int or long it does work, i know that the different float values in the tensor are finite because im quantizing to some bit, any idea how can i run torch.unique on float tensor?<br>
Thanks alot!</p>

username: googlebot
cooked: <pre><code class="lang-auto">a=torch.tensor([1., 1.+1e-7, 1.+1e-6, 1.+1e-5, 1.5])
explvl=torch.tensor([20]) #threshold 1/(2**20) ~= 1e-6
a.ldexp(explvl).round().unique().ldexp(-explvl).tolist()
: [1.0, 1.000_0009536743164, 1.000_009536743164, 1.5]
</code></pre>

username: april2111
cooked: <p>Thank you for this code snippet!</p>