akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch Acceleration GPT-Fast

khalid-sharma: <p>Hi,</p>
<p>I’m basically struggling to understand how to be able to use todays blogpost on accelerating LLM throughput.</p>
<p>I want to make sure I’m understanding or doing it properly but what I think I need to do is:</p>
<ol>
<li>clone the repo</li>
<li>choose a model checkpoint to convert into a faster version using</li>
</ol>
<pre><code class="lang-auto">export MODEL_REPO=meta-llama/Llama-2-7b-chat-hf
./scripts/prepare.sh $MODEL_REPO
</code></pre>
<ol start="3">
<li>Then run that model using:</li>
</ol>
<pre><code class="lang-auto">python generate.py --compile --checkpoint_path checkpoints/$MODEL_REPO/model.pth --prompt "Hello, my name is"
</code></pre>
<p>If this is wrong, I’d the proper way to do this because I get a long download for the weights and then an error on line 3 above regarding model.pth and the checkpoints</p>
<p>Furthermore, I don’t really understand if I can convert any open source llm in this way or just the ones tested etc; I saw some mention of mistral but I guess I dont really know the full process to be confident enough. Also might be a dumb question but I won’t see the results of this efficiency without an A100 too right? Appreciate any guidance.</p>
<p>If anyone could walk me through how I’m supposed to utilize this methodology the right way, I’d appreciate that immensely.</p>
<p>Documentation: <a href="https://pytorch.org/blog/accelerating-generative-ai-2/" rel="noopener nofollow ugc">Blog Post</a><br>
Github: <a href="https://github.com/pytorch-labs/gpt-fast" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch-labs/gpt-fast: Simple and efficient pytorch-native transformer text generation in &lt;1000 LOC of python.</a></p>

marksaroufim: <p>The implementation is meant to be a hacky copy and paste one so right now it mostly supports Llama but it should be easy to support other implementations</p>
<p>What’s the error you run into when you call generate? You didn’t paste any logs</p>

khalid-sharma: <p>The error I have currently is related to not yet being approved by Meta via hugging face to use the templates, so was asking specifically about the process for when I am approved.</p>
<p>I understand that these are quantization methods and speed up methods so is the idea that once I have access I can use these files to make small faster models that I can reupload publicly to Huggingface for example or LM Studio?</p>

marksaroufim: <p>As long as you use the same kind of state dict as the one for llama2 you should be good</p>
Pytorch website timing out

Sayan01: <p>What kind of junk servers are Pytorch websites hosted on? It is repeatedly timing out the download. And please don’t try to say that the issue is something else.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/3/638eb2ce8733bad810a421f1194ff505af263a24.png" data-download-href="https://discuss.pytorch.org/uploads/default/638eb2ce8733bad810a421f1194ff505af263a24" title="Screen Shot 2023-12-02 at 12.34.00 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/638eb2ce8733bad810a421f1194ff505af263a24_2_690x88.png" alt="Screen Shot 2023-12-02 at 12.34.00 PM" data-base62-sha1="ecIZYydwgFA6YxSBgYpm6yjnIjy" width="690" height="88" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/638eb2ce8733bad810a421f1194ff505af263a24_2_690x88.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/638eb2ce8733bad810a421f1194ff505af263a24_2_1035x132.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/3/638eb2ce8733bad810a421f1194ff505af263a24.png 2x" data-dominant-color="F7EAE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-12-02 at 12.34.00 PM</span><span class="informations">1098×141 27.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you please watch your language?<br>
Also, which install command are you using?</p>
Autoencoder output size is different from input size

H1dd3n_Squ1d: <p>I am trying to implement a CNN autoencoder that will take in Mel spectrogram as inputs but am currently running into an issue with the output size being different from the input size. The input has a shape of [1, 64, 302] (channels, n_mels, num_feats). The output has a shape of [1, 64, 304].</p>
<pre><code class="lang-auto">class CNN_AE(nn.Module):
    def __init__(self):
        super(CNN_AE, self).__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(32, 16, kernel_size=5, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(16, 8, kernel_size=5, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(8, 16, kernel_size=5, stride=2, padding=1, output_padding=1),
            nn.ReLU(),
            nn.ConvTranspose2d(16, 32, kernel_size=5, stride=2),
            nn.ReLU(),
            nn.ConvTranspose2d(32, 1, kernel_size=5, stride=2, padding=1, output_padding=1),
            nn.Sigmoid()
        )
         
    def forward(self, x):
        print(x.shape)
        x = self.encoder(x)
        x = self.decoder(x)
        print(x.shape)
        return x
</code></pre>
<p>Am I using the padding and output_padding wrong?</p>
New added data cause FileNotFound on Dataloader

XezXey: <p>I have a list of my data path but not all of them are in the directory. And i will copy my data from other place once my dataloader going to process this file (e.g. 1 batch before this). But this turn to be the OSError: file not found. Do you know how can i fix this or doing this way in another way around. The reason that i want to do this way is my very large dataset and i want to just partially copy them then remove after the training on that data is done.<br>
I also add some code snippets of my toy example that show the problem.</p>
<pre><code class="lang-auto">
import glob, os
import numpy as np
from torch.utils.data import DataLoader, TensorDataset, Dataset, RandomSampler, Sampler

class CustomDataset(Dataset):
    def __init__(self, data):
        self.data = data
    def __len__(self):
        return len(self.data)
    def __getitem__(self, index):
        return np.loadtxt(self.data[index])
    
if __name__ == '__main__':
    if os.path.exists('./3.txt'):
        os.system('rm 3.txt')
    file = glob.glob('./*.txt') + ['./3.txt']
    print(file)
    dataset = CustomDataset(file)
    dataloader = DataLoader(dataset=dataset, batch_size=1, shuffle=False, num_workers=1, drop_last=True, pin_memory=True, persistent_workers=True)

    for i, x in enumerate(dataloader):
        print("before : ", i, x)
        if i == 1:
            with open('./3.txt', 'w') as f:
                f.write('3')
        print("after : ", i, x)
        input(dataset.data)```</code></pre>
How to efficiently multiply a mask to hidden layers

alexgagnon: <p>Hi, wondering if this is possible. I’ve set up an MLP model with a variable number of hidden layers using ModuleList. I’d like to know how to efficiently apply a mask of the same dimension to these hidden layers. Here’s what I have, but I’d like to avoid iteration and stick to just matrix multiplication if possible.</p>
<pre data-code-wrap="py"><code class="lang-py">class MLP(nn.Module):
    def __init__(self, input_size, num_hidden_layers, hidden_size, output_size):
        super(MLP, self).__init__()
        assert num_hidden_layers &gt; 0
        self.input = nn.Linear(input_size, hidden_size)
        self.layers = nn.ModuleList()
        for i in range(num_hidden_layers):
            self.layers.append(nn.Linear(hidden_size, hidden_size))
        self.output = nn.Linear(hidden_size, output_size)

def apply_mask(model, mask):
    # apply mask to hidden layer weights only
    for i in range(len(model.layers)):
        model.layers[i].weight.data = model.layers[i].weight.data * mask[i]
    return model
</code></pre>

soulitzer: <p>Are you looking for something like this? <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.apply" class="inline-onebox" rel="noopener nofollow ugc">Module — PyTorch 2.1 documentation</a><br>
but iteration seems hard to avoid if you’d like to use a differently specified mask for each weight</p>
<p>Also btw, you shouldn’t be using .data. If you’d like to update your weight in-place without autograd tracking you can use <code>with torch.no_grad</code>.</p>
Forward-over-reverse HVP slower than reverse-over-reverse HVP with torch.func API

MatDag: <p>Hello,<br>
I am benchmarking the different ways to perform Hessian-vector products with the API <code>torch.func</code>. I am a bit surprised by the results I get since the <code>forward-over-reverse</code> fashion seems significantly longer than the <code>reverse-over-reverse</code> way. Am I doing something wrong?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torchvision.models import resnet18
from functorch.experimental import replace_all_batch_norm_modules_


from time import perf_counter

batch_size = 4
num_classes = 1000
model = replace_all_batch_norm_modules_(resnet18()).cuda()

batch = {
    'images': torch.randn(batch_size, 3, 224, 224).cuda(),
    'labels': torch.randint(0, num_classes, (batch_size,),).cuda()
}


def loss(params):
    """loss function used for training."""
    logits = torch.func.functional_call(model, params,
                                        (batch['images'], ))
    res = torch.nn.functional.cross_entropy(logits, batch['labels'])
    return res


def hvp_forward_over_reverse(x, v):
    return torch.func.jvp(torch.func.grad(loss), (x, ), (v, ))[1]


def hvp_reverse_over_forward(x, v):

    def jvp_fun(x, v):
        return torch.func.jvp(loss, (x, ), (v, ))[1]

    return torch.func.grad(jvp_fun)(x, v)


def hvp_reverse_over_reverse(x, v):

    grad_fun = torch.func.grad(loss)

    return torch.func.grad(lambda y: sum(
        torch.dot(a.ravel(), b.ravel())
        for a, b in zip(grad_fun(y).values(), v.values()))
    )(x)


if __name__ == "__main__":
    params = dict(model.named_parameters())
    v = torch.func.grad(loss)(params)

    print("Forward-over-reverse")
    start = perf_counter()
    hvp_forward_over_reverse(params, v)
    print(perf_counter() - start)

    print("Reverse-over-forward")
    start = perf_counter()
    hvp_reverse_over_forward(params, v)
    print(perf_counter() - start)

    print("Reverse-over-reverse")
    start = perf_counter()
    hvp_reverse_over_reverse(params, v)
    print(perf_counter() - start)

</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Forward-over-reverse
0.27017239201813936
Reverse-over-forward
0.043903919868171215
Reverse-over-reverse
0.09637519344687462
</code></pre>
<p>Best,</p>

soulitzer: <p>You probably want to use <code>torch.cuda.sychronize()</code> and average your measurements over many runs to reduce variance. Forward-over-reverse is also the first hvp you are computing, so maybe cuda is still warming up/mallocing for the first time.</p>
Unexpected (wrong?) result when calculating derivatives using autograd

JonathanBe: <p>Hello everyone,</p>
<p>I’ve encountered an interesting issue while working with the autograd system in PyTorch, which could potentially lead to unexpected or even unnoticed problems. Here’s a minimal example to illustrate the issue:</p>
<p>Let’s say we have <code>z = x + y</code> and <code>f = z + x + y</code>.</p>
<p>Then <code>df/dz</code> should be 2, as <code>f = z + z = 2z</code>.</p>
<p>However, when I calculate this in PyTorch using the following code:</p>
<pre><code class="lang-auto">import torch

x = torch.tensor([1.], requires_grad=True)
y = torch.tensor([2.], requires_grad=True)

z = x + y
f = z + x + y

df_dz = torch.autograd.grad(f, z, retain_graph=True)

print(df_dz)
</code></pre>
<p>The output I get is 1, not 2. Can anyone explain why this is the case?</p>

soulitzer: <p>If I allow substitutions, it becomes arbitrary as to what a derivative wrt a intermediate like z, e.g. df_dz could be.</p>
<p>Suppose instead I have z:= 2x + y, then there’s two ways of writing f:</p>
<ol>
<li>2z - x</li>
<li>z + x + y<br>
Would df_dz be 1 or 2 here? It isn’t well-defined.</li>
</ol>
<p>A different way to think about this that is less confusing might be:</p>
<p>We have two functions, z, f defined as follows:<br>
z(a, b) := a + b<br>
f(w, x, y) := w + x + y</p>
<p>In each of the expressions above, the variables w, x, y, a, b are only meaningful within the scope of the function definition.</p>
<p>In the original example, I can compose the two functions and define  g(a, b, x, y) := f(z(a, b), x, y)<br>
and then h(x, y) := g(x, y, x, y).</p>
<p>From the perspective of f, it does not know could be related and hence the answer is always 1 without any ambiguity.</p>
Multiprocessing in the test dataset of reinforcement learning

Christian: <p>Hi,<br>
I have a problem with running speed of test phase of reinforcement learning. I use DQN for localization. In my case, different test samples take different numbers of steps to localize the target. I can’t use batch here. But for loop takes very long time to sequentially go through all the samples in testset. I tried multiprocessing for for loop to parallel the running for test samples. But it always give me errors. Eg. errors usually happen at .jstart(), and show TypeError: can’t pickle generator objects. Does anyone have a solution for it? Or an example would be better. Thank you.</p>
Captum LayerGradCam for vit_b_16 model

UnitTest: <p>Has anyone succesfully used LayerGradCam on vit_b_16 model? I am using</p>
<blockquote>
<p>create_feature_extractor(base_model, return_nodes=[‘encoder’])<br>
to extract the encoder features. Interested to visualize the GradCam output.</p>
</blockquote>
<p>Also, visualizing the gradients of the model (vit_b_16), we observe jpeg artifacts (on both JPEG and non-JPEG). However, previous research infers that compression should not effect model training. Thoughts?</p>
Torchtext compatibility

eastchun: <p>Is there a torchtext release that supports the followings ?</p>
<ul>
<li>pytorch 2.1.0, cuda 11.8, numpy 1.24.3</li>
</ul>

ptrblck: <p><code>torchtext==0.16</code> was released on the same day together with <code>torch==2.1.0</code>, so should be compatible.</p>
nn.Module hook for optimizer.step

MuawizChaudhary: <p>Hi all,</p>
<p>In my code, the optimizer updates the parameters after a training step. These parameters are then used at the start of the forward pass to generate weights for my model to use. The generation is done every training step, and the first step of eval.</p>
<p><a href="https://discuss.pytorch.org/t/use-forward-pre-hook-to-modify-nn-module-parameters/108498">I understand</a> that I could use <code>forward_pre_hook</code> to accomplish what I am describing. However, updating during that first eval step does not look very nice to me, I use a flag in my model to keep track of that update.</p>
<p>I believe that having my nn.Module call a hook after each optimizer step would be more convenient, for me and everyone else that may want to optimize parameters for use in generation of weights or filters or what not.</p>
<p>Does such a <code>optimizer_step_post_hook</code> exist? Is it possible to implement such a hook?</p>

agu: <p><a href="https://github.com/pytorch/pytorch/pull/89176" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/89176</a> added <code>torch.optim.Optimizer.register_step_post_hook</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Converting Pytorch (.pt) to Darknet (.weights and .cfg)

ajdewolf: <p>Hi,</p>
<p>I have a Pytorch weights file (.PT) which I would like to convert to Darknet format (.weights and .cfg) files.<br>
The .PT file was trained using this repository: <a href="https://github.com/ultralytics/yolov5" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ultralytics/yolov5: YOLOv5 🚀 in PyTorch &gt; ONNX &gt; CoreML &gt; TFLite</a></p>
<p>I found a repository of scripts, seemingly to accomplish a similar task, here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/vgsatorras/pytorch-caffe-darknet-convert">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/vgsatorras/pytorch-caffe-darknet-convert" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/dd3d404ba3f45617779d86604db38e9d6441276913ae118df57c1a35346ae9cb/vgsatorras/pytorch-caffe-darknet-convert" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/vgsatorras/pytorch-caffe-darknet-convert" target="_blank" rel="noopener nofollow ugc">GitHub - vgsatorras/pytorch-caffe-darknet-convert: convert between pytorch,...</a></h3>

  <p>convert between pytorch, caffe prototxt/weights and darknet cfg/weights - GitHub - vgsatorras/pytorch-caffe-darknet-convert: convert between pytorch, caffe prototxt/weights and darknet cfg/weights</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Though, the documentation is not enough for me to understand how to convert my particular .PT file.</p>
<p>Has anyone done this particular conversion, and could help me to understand what is necessary to do it?</p>
<p>Thanks,<br>
-ajdewolf</p>

bibinwilson: <p>Did you find any solution?</p>

Shubham_Gupta: <p><a class="mention" href="/u/bibinwilson">@bibinwilson</a> <a class="mention" href="/u/ajdewolf">@ajdewolf</a>  Did you find any solution</p>
Building pytorch1.8 with cuda statically linked

kandarp_sharda: <p>I was running  this command in docker  for building pytorch  with source code.</p>
<p><code>cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXE_LINKER_FLAGS="-static"</code></p>
<p>Error I am getting is this</p>
<pre><code class="lang-auto">CMake Error at cmake/public/cuda.cmake:74 (message):
  Caffe2: Couldn't determine version from header: Change Dir:
  '/pytorch/CMakeFiles/CMakeTmp'



  Run Build Command(s): /opt/conda/bin/cmake -E env VERBOSE=1 /usr/bin/make
  -f Makefile cmTC_39818/fast

  /usr/bin/make -f CMakeFiles/cmTC_39818.dir/build.make
  CMakeFiles/cmTC_39818.dir/build

  make[1]: Entering directory '/pytorch/CMakeFiles/CMakeTmp'

  Building CXX object CMakeFiles/cmTC_39818.dir/detect_cuda_version.cc.o

  /usr/bin/c++ -I/usr/local/cuda/include -Wno-deprecated
  -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -o
  CMakeFiles/cmTC_39818.dir/detect_cuda_version.cc.o -c
  /pytorch/detect_cuda_version.cc

  Linking CXX executable cmTC_39818

  /opt/conda/bin/cmake -E cmake_link_script
  CMakeFiles/cmTC_39818.dir/link.txt --verbose=1

  /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL
  -fopenmp -static -rdynamic
  CMakeFiles/cmTC_39818.dir/detect_cuda_version.cc.o -o cmTC_39818
  -Wl,-rpath,/usr/local/cuda/lib64 /usr/local/cuda/lib64/libcudart.so

  /usr/bin/ld: attempted static link of dynamic object
  `/usr/local/cuda/lib64/libcudart.so'

  collect2: error: ld returned 1 exit status

  CMakeFiles/cmTC_39818.dir/build.make:99: recipe for target 'cmTC_39818'
  failed

  make[1]: *** [cmTC_39818] Error 1

  make[1]: Leaving directory '/pytorch/CMakeFiles/CMakeTmp'

  Makefile:127: recipe for target 'cmTC_39818/fast' failed

  make: *** [cmTC_39818/fast] Error 2



Call Stack (most recent call first):
  cmake/Dependencies.cmake:1148 (include)
  CMakeLists.txt:564 (include)


-- Configuring incomplete, errors occurred!
</code></pre>
Only for first batch - Was asked to gather along dimension 0, but all input tensors were scalars

Woosung: <p>Hi guys,</p>
<p>I know that there are similar issues, but it seems a little bit different from my case</p>
<p>I tried to train my model with the dataparallel with multi-GPUs<br>
but, I got this warning only for the first batch among total iterations.</p>
<p>Does anyone know how to suppress this warning? or is it okay to ignore this?</p>
<p>thx</p>
Slowdown in CPU-based Preprocessing After Loading Model Weights onto GPU

HenryPaik1: <p>I’m encountering an unexpected issue in my machine learning pipeline. Specifically, after loading the model weights onto the GPU, the CPU-based preprocessing function, which relies solely on NumPy and ndarray operations and is entirely independent of the GPU or the model, experiences a significant slowdown.</p>
<pre><code class="lang-auto"># super slow

def preprocess(arr):
    ...do some with numpy
    return arr

device = "cuda"
model = model.to(device)
ckpt = torch.load(saved_checkpoint_path, map_location=device)
model_state = ckpt['model']
model.load_state_dict(model_state)

src = []
for data in file_list:
    ndarray = read_data(data)
    output = preprocess(ndarray)
    src.append(output)
</code></pre>
<p>There is no data transfer between the CPU and GPU for the actual data used in the preprocessing. Strangely, performing preprocessing and then loading the model weights to the GPU separately is faster compared to the scenario where the weights are loaded onto the GPU before the preprocessing step. This suggests that the act of loading model weights onto the GPU is somehow affecting the CPU-based preprocessing, even though there should be no direct interaction between the two.</p>
<pre><code class="lang-auto"># super fast

src = []
for data in file_list:
    ndarray = read_data(data)
    output = preprocess(ndarray)
    src.append(output)

device = "cuda"
model = model.to(device)
ckpt = torch.load(saved_checkpoint_path, map_location=device)
model_state = ckpt['model']
model.load_state_dict(model_state)

</code></pre>
<p>I’ve ruled out data transfer issues and confirmed that the preprocessing itself is not GPU-dependent. Has anyone encountered a similar situation, and if so, what could be causing this unexpected slowdown? I would appreciate any insights or suggestions on how to troubleshoot and resolve this issue.</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>

HenryPaik1: <p>I discovered that the issue is related to the “subprocess” module, as discussed in <a href="https://discuss.pytorch.org/t/torch-models-on-gpu-slow-down-python-subprocess-module/126884" class="inline-onebox">Torch models on GPU slow down python subprocess module?</a>. In my preprocessing code, I have utilized the subprocess module, as shown below.</p>
<pre><code class="lang-auto">from subprocess import PIPE, Popen
input = sample.tobytes()
cmd = [
    "/usr/bin/sox", 
    ...
]
p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = p.communicate(input)
</code></pre>
<p>Actually I conducted a test by loading the model weight on the CPU and observed the same issue as described earlier. What is your perspective on how the model weight influences the functioning of the subprocess? Could you offer any advice or insights on this matter?</p>
<hr>
<p>+ version information</p>
<pre><code class="lang-auto">[pip3] numpy==1.26.2
[pip3] torch==1.12.1+cu113
[pip3] torchaudio==0.12.1+cu113
</code></pre>
Torch models on GPU slow down python subprocess module?

davidw0311: <p>I have been trying to use the python subprocess module while training a neural network in pytorch, but I noticed that subprocess runs many times slower if I have a network initialized on a gpu. Here is an example script I used, with a very simple linear network, profiling the times using line_profiler, and looping a simple subprocess call 100 times:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import subprocess
from line_profiler import LineProfiler

class TestNN(nn.Module):
    def __init__(self, device):
        super(TestNN, self).__init__()
        self.fc1 = nn.Linear(5,16)
        self.device=device
        self.to(self.device)

def test_subprocess():
    device = torch.device('cuda:0')
    testNet=TestNN(device)

    for i in range(100):
        subprocess.run(["ls",  "-l"], capture_output=True)

if __name__ == '__main__':

    lprofiler =LineProfiler()
    lp_wrapper = lprofiler(test_subprocess)
    
    lp_wrapper()
    lprofiler.print_stats()
</code></pre>
<p>Just by moving the small network to gpu results in more than 4x slower execution of subprocess.run().</p>
<p>My results from line_profiler when the network is on cpu:</p>
<pre><code class="lang-auto">Total time: 1.46088 s
File: test_subprocess_gpu.py
Function: test_subprocess at line 13

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    13                                           def test_subprocess():
    14         1        172.0    172.0      0.0      device = torch.device('cpu')
    15         1        806.0    806.0      0.1      testNet=TestNN(device)
    16                                           
    17       101       1235.0     12.2      0.1      for i in range(100):
    18       100    1458671.0  14586.7     99.8          subprocess.run(["ls",  "-l"], capture_output=True)
</code></pre>
<p>My results when the network is initialized on GPU:</p>
<pre><code class="lang-auto">Timer unit: 1e-06 s

Total time: 8.63406 s
File: test_subprocess_gpu.py
Function: test_subprocess at line 13

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    13                                           def test_subprocess():
    14         1        174.0    174.0      0.0      device = torch.device('cuda:0')
    15         1    2084937.0 2084937.0     24.1      testNet=TestNN(device)
    16                                           
    17       101       1163.0     11.5      0.0      for i in range(100):
    18       100    6547789.0  65477.9     75.8          subprocess.run(["ls",  "-l"], capture_output=True)
</code></pre>
<p>Does anyone know what causes this slow-down and how to increase the speed with the network initialized on GPU? I am very puzzled why initializing a neural network on GPU will have any affect on the speed of subprocess.run(). Any help is greatly appreciated!</p>

HenryPaik1: <p>Did you find the solution?</p>
Two outputs with same inputs for torch.sin()

tinted: <p>In the below code, y and y2 plot differently even though they get same input, why is that?</p>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt

torch.manual_seed(42)

# Data, inputs and outputs
x_rand = torch.randint(-1000, 5000, (10000, 1), dtype=torch.float)

# Fn 1
x = 0.1 * x_rand
y = torch.sin(0.5 * x)

# Fn 2
x2 = 0.5 * 0.1 * x_rand
y2 = torch.sin(x2)

# Sorting
x_plt, index = torch.sort(x, dim=0)
y_plt = torch.gather(y, dim=0, index=index)

x2_plt, index = torch.sort(x2, dim=0)
y2_plt = torch.gather(y2, dim=0, index=index)

# Ploting
plt.plot(x_plt.numpy(), y_plt.numpy())
plt.plot(x2_plt.numpy(), y2_plt.numpy())
plt.xlim(-20, 20)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72cbb3597a247ca5959515057f8932d4e4efd021.png" alt="image" data-base62-sha1="gnwRcBY1KwSaTwzolDEb0COFEZz" width="284" height="210"></p>

ptrblck: <p>You are using different <code>x</code> axis values and are thus moving the smaller sine wave. Use the same axis and you will see the outputs overlap:</p>
<pre data-code-wrap="python"><code class="lang-python">plt.plot(x_plt.numpy(), y_plt.numpy())
plt.plot(x_plt.numpy(), y2_plt.numpy())
</code></pre>
<p>You can also check if by computing the error e.g. via <code>(y - y2).abs().max()</code> which is <code>0</code>.</p>

tinted: <p>I feel dumb for looking at this problem for hours, Thank you for taking time to answer this.</p>
Custom loss class not working as expected

preshel: <p>I have following code</p>
<pre><code class="lang-auto">_epsilon = torch.finfo(torch.float32).eps

def WeightedF1LossFunc(y_pred, y_true,positiveThresh,betaSquared):
    global _epsilon 
    y_pred = torch.ceil(y_pred - positiveThresh)
    tp = (y_true * y_pred).sum()
    fp = ((1 - y_true) * y_pred).sum()
    fn = (y_true * (1 - y_pred)).sum()
    precision = tp / (tp + fp + _epsilon)
    recall = tp / (tp + fn + _epsilon)
    f1_score = (_betaSquared + 1) * (precision * recall) / (_betaSquared * precision + recall + _epsilon)
    return 1 - f1_score

class WeightedF1LossClass(nn.Module):
    def __init__(self, positiveThresh,betaSquared):
        super(WeightedF1LossClass, self).__init__()
        self._betaSquared = betaSquared
        self._positiveThresh = positiveThresh
    def forward(self, inputs, targets):
        return WeightedF1LossFunc(inputs,targets,self._positiveThresh,self._betaSquared)
        
xTrain = _runData['train']['x']
yTrain = _runData['train']['y']
epochs = 10
lossFN = WeightedF1LossClass(0.5,_betaSquared)
model = CreateModel(_runData['num_of_features'], _layerConfig).to(device) # create a sequencial model with _runData['num_of_features'] inputs
optimizer = optim.Adam(model.parameters(), lr=0.001, eps=1e-07)  # Use Adam optimizer
for epoch in range(epochs):
    model.train()
    predictedY = model(xTrain)
    loss = lossFN(predictedY, yTrain)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad() 
    
    print(f'loss={loss.item():.4f}')
</code></pre>
<p>However the loss doesn’t seem to change…what’s wrong?<br>
loss=0.7822<br>
loss=0.7822<br>
loss=0.7822<br>
loss=0.7822<br>
loss=0.7822<br>
loss=0.7822<br>
loss=0.7822<br>
loss=0.7822<br>
loss=0.7822<br>
loss=0.7822</p>

ptrblck: <p>The <code>torch.ceil</code> operation will create zero gradients:</p>
<pre data-code-wrap="python"><code class="lang-python">y_pred = torch.randn(10, requires_grad=True)

out = torch.ceil(y_pred)
out.mean().backward()
print(y_pred.grad)
# tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
</code></pre>
Different accuracy when I don't use Distributed Mode

External_happy: <p>I am currently running a code based on a paper I found on GitHub that corresponds to a paper published in ICLR 2023. I am using a single GPU (no distributed mode). However, the results are significantly different from what was reported in the paper. Is it expected for the results to differ? For instance, the result on a single GPU for the DTD dataset is 50.1%, whereas in the paper, it is reported as 54.1% using Vit-B/16.</p>
What is the Highest Version of Torch that can be Built from Source?

TechVillain: <p>As most of you people already know, the official binary of PyTorch will not work if you have an older GPU. If your GPU has a Compute Capability of 3.0, or 3.5, then you are screwed. In order to use PyTorch with CUDA, you have to build a Torch wheel from source.</p>
<p>However, the official site never tells, what the highest version can be built from source if you own a sm_30 or sm_35 graphic card.</p>
<p>I have an Nvidia GeForce GTX graphic card with a Compute Capability of 3.0 (Kepler sm_30), running on Python 3.8 and cuDNN 7.6.5 in Windows.</p>
<p><strong>After extensive building and testing process, here is my conclusion.</strong></p>
<blockquote>
<p>With a Compute Capability 3.0 GPU, the highest CUDA version you can install is CUDA 10.2, and the highest PyTorch version you can build from source is Torch 1.10.2.</p>
</blockquote>
<p>In other words, starting from PyTorch 1.11.0 and above, there will be new nccl operators added, and therefore causing nvcc errors during the build process.</p>
<p>Although I am confident that no human being on this planet can ever possibly build PyTorch from source on any versions &gt;=1.11.0, I’m still open to hear different opinions and criticism from an opposite ideology.</p>

tom: <p>You could disable distributed and nccl…<br>
And this is quite typical: PyTorch is going to adopt certain things (I remember when we switched to <code>__shfl_sync</code> instead fo <code>__shfl</code> so the CUDA requirement bumped to &gt;= CUDA 9. But it would have been thinkable to just fix a few instances to compile with CUDA 8.<br>
But at some point you likely run into PyTorch using new C++ features and CUDA not supporting compilers that use them or somesuch or the places that need to be fixed become too many.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: I think the title could be amended to “…with Compute Capability 3.0/3.5”</p>
Question Regarding Opacus Warning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions

kr-ramesh: <p>I had a question regarding the warning thrown out by Opacus (it’s previously been discussed in other issues, such as this one: <a href="https://github.com/pytorch/opacus/issues/328#issuecomment-1027361968" class="inline-onebox" rel="noopener nofollow ugc">deprecation message for non-full backward hook · Issue #328 · pytorch/opacus · GitHub</a>)</p>
<p>I’ve been testing an unmodified version of gpt-2 medium model with Opacus. I wanted to clarify that the warning doesn’t necessarily imply that the DP guarantee is affected by this (particularly as to my knowledge, the Opacus repo now supports the grad sample functions for all the layers in GPT-2, and I’ve validated the modules with the module validator). Any help regarding this or additional information would be great, thanks!</p>
Should we perform optimizer.zero_grad() before we do forward pass or after that?

Peter_Ham: <p>as titled. should I call zero grad before I do forward pass? Or I just need to make sure it is performed before I call loss.backward()?</p>

jpeg729: <p>The forward pass doesn’t care about the grads, nor does it modify them.<br>
You only need to zero the grads before calling <code>.backward()</code>.</p>

Peter_Ham: <p>Thanks! It helps a lot</p>

vb000: <p>Does calling zero_grad before or after forward pass have any memory implications, however?</p>
Missing Histograms for LayerNorm in Numeric Suite Analysis

kmitsunami: <p>When using the FX Graph mode quantization API for Static Post-Training Quantization for my model, the output seems incorrect. As shown on <a href="https://pytorch.org/tutorials/prototype/numeric_suite_tutorial.html" rel="noopener nofollow ugc">this page</a>, when displaying the weight differences before and after quantization using ns.compare_weights from Numeric Suite, the histogram for LayerNorm in the float model does not appear (see the attached image). The values of the weights do exist, but since they are not displayed on the graph, it’s difficult to determine if the quantization has been done correctly. Am I missing something?<br>
torch version: 2.1.0</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b4bfe56a3129f0179c3a2a74a2f5f51e699d9e7.png" alt="down_blocks.0.attentions.0.transformer_blocks.0.norm1.weight" data-base62-sha1="8syZ8z9KDkOijKNk18J6RbxpDpl" width="640" height="480"></p>

kmitsunami: <p>There is an update. After a detailed check, it was found that the weights of LayerNorm are exactly the same in both Float and Quant versions. Despite the layer name changing to QuantizedLayerNorm after quantization, is it correct to understand that the weights of the normalization layer do not get quantized?</p>

jcaip: <p><a class="mention" href="/u/kmitsunami">@kmitsunami</a></p>
<p>The QuantizedLayerNorm module calls into a quantized layer norm op (see <a href="https://github.com/pytorch/pytorch/blob/f93ea143097716dac0bf1a7c3d89b6a6e3a4e1d0/torch/ao/nn/quantized/modules/normalization.py#L25" rel="noopener nofollow ugc">here</a> ), which also takes in scale and zero point. So I would expect that the weights are the same for the two layers, but that QuantizedLayerNorm is a bit faster, since it calls into the quantized op instead of the normal one.</p>

kmitsunami: <p>Hi <a class="mention" href="/u/jcaip">@jcaip</a>, understood. Thanks for your help!</p>
How to use torchrl example buffer with multiprocessing?

earthbound_untilwhen: <p>There is multiple processes and i want them to use same example buffer. For example 5 of them produce and send data, 1 of them samples from data.</p>
<p>I can do this with built-in queue whcih is process safe but then i need to get all data, sample and put again.</p>
<p>I wanted to use example buffer from torchrl and tried multiple things with especially LazyMemmapStorage but always got<br>
RuntimeError: Cannot sample from an empty storage.</p>
<p>Is there a way to use example buffer from torchrl with python multiprocessing in a process safe way?</p>
<p><a class="mention" href="/u/vmoens">@vmoens</a> said: Regarding the <strong>distributed</strong> side of things, <code>LazyMemmapStorage</code> paves the way to shared storages across can be placed anywhere on the available partitions: if nodes have access to a common space with reasonably fast common storage, they will all be capable of writing items to the buffer and access it at low cost. Stay tuned to hear more about this! but i couldn’t find a way to make it work.</p>
<p>Source: <a href="https://discuss.pytorch.org/t/rfc-torchrl-replay-buffers-pre-allocated-and-memory-mapped-experience-replay/155335" class="inline-onebox">[RFC] TorchRL Replay buffers: Pre-allocated and memory-mapped experience replay</a></p>
<p>Thank you for reading!</p>

vmoens: <p>Hey! Do you have an example of what you’re doing to share?</p>

earthbound_untilwhen: <p>Hi thank you for your attention,</p>
<p>I will give a basic example: There is 6 cpu processes. Five of them create data and feed it to buffer. One of them learns from that data by using the buffer. I need a shared array which can hold a dictionary preferably. I want to sample from whole data so I have to access whole buffer at once. I thought using LazyMemmapStorage with disk location specified but I couldn’t make it work.</p>
<p>In the paper they used Reverb from Deepmind.<br>
Is there a way to do it with torchrl currently?</p>

vmoens: <p>I think I see where the problem might be:<br>
the data of the replay buffer will be accessible to all processes, but not the metadata of the buffer (cursor etc).<br>
So when each process feeds the buffer, it will overwrite the data written by other processes!</p>
<p>The solution we use is to rely on RPC, where each process calls the replay buffer <code>extend</code> method in a centralized way.</p>
<p>There is an example <a href="https://github.com/pytorch/rl/blob/main/benchmarks/storage/benchmark_sample_latency_over_rpc.py" rel="noopener nofollow ugc">here</a></p>
<p>That being said, we could think about a pure mp-based RB implementation if you think that would be useful!</p>

vmoens: <p><a href="https://github.com/pytorch/rl/pull/1724" rel="noopener nofollow ugc">This PR</a> or a version of it should solve your issue!<br>
If you could test the changes in test/test_rb.py and tell me if that is the usage you wanted to make it’d be awesome!</p>
How to add/del/replace/fuse operators in torch.nn.module?

oleotiger: <p>I want to customize the pre-trained LLM model downloaded from HuggingFace. There may be operations like adding, removing, replacing or fusing operators.</p>
<p>One approach is to use torch.fx. However, most HuggingFace models take input as condition and torch.fx does not yet support dynamic control flow.</p>
<p>Will TorchScript supersede torch.fx in the foreseeable future? I don’t think TorchScript is an ideal tool to fulfill my goal because I wish to retain the PyTorch modules in code.</p>
<p>Is there any way that I can tweak models from HuggingFace in a flexible fashion?</p>
Can't set batch_first=true in torch::nn::TransformerOptions?

Fitanium: <p>I want to use Transformer by LibTorch, and want to set batch_first=true, But</p>
<pre data-code-wrap="C++"><code class="lang-C++">torch::nn::TransformerOptions options(512, 8);
options.batch_first(true); // no such method in torch::nn::TransformerOptions
</code></pre>
<p>Can anyone help me solve this problem? Thanks in advance.</p>
Distributed broadcast didn't send tensor to other GPUs

ShuokaiPan: <p>Hi there,</p>
<p>I am trying to send a tensor from GPU0 to all the othre GPUs using the <code>broadcast</code> function, on a single machine with 2 GPUs, but it seems that this function was not working for me. You can find a toy example below. Can someone tell me what I did wrong here?</p>
<p>Thanks a lot for your help.</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist

def broadcast_tensor():
    dist.init_process_group(backend='nccl')

    # group = dist.new_group(list(range(2)))
    local_rank = dist.get_rank()

    if local_rank == 0: 
        t = torch.tensor([1,2,3]).to(local_rank)
    else:
        t = torch.empty(3).to(local_rank)

    dist.broadcast(t, src=0)
    print('local rank: ', local_rank, ' tensor', t)

    dist.destroy_process_group()

    return t

if __name__ == '__main__':
    t = broadcast_tensor()
</code></pre>
<p>The print out messages are the following,</p>
<p>local rank:  0  tensor tensor([1, 2, 3], device=‘cuda:0’)<br>
local rank:  1  tensor tensor([1.4013e-45, 0.0000e+00, 2.8026e-45], device=‘cuda:1’)</p>

ShuokaiPan: <p>To provide more info, I was running the script like this,</p>
<p><code>torchrun  --standalone --nnodes=1 --nproc-per-node=2  test_dist_broadcast.py</code></p>

ShuokaiPan: <p>I have found a version that would distribute the tensors to all GPU device, but I have to create non-empty tensors first on each GPU. Does anyone know why this is the case? I found many online examples that first create empty tensors on non-src rank devices. I am using the official torch GPU docker image with version of 2.1.0.</p>
<pre><code class="lang-auto">def broadcast_tensor():
    dist.init_process_group(backend="nccl")

    local_rank = dist.get_rank()
    t = torch.randint(0, 5, size=(3,), device=local_rank)
    print('local rank: ', dist.get_rank(), 'Before broadcast, tensor: ', t)

    dist.broadcast(t, 0)
    print('local rank: ', dist.get_rank(), 'After broadcast, tensor: ', t)
    dist.destroy_process_group()
    return t
</code></pre>
<p>The print out messages are</p>
<p>local rank:  0 Before broadcast, tensor:  tensor([0, 0, 4], device=‘cuda:0’)<br>
local rank:  1 Before broadcast, tensor:  tensor([3, 3, 1], device=‘cuda:1’)<br>
local rank:  0 After broadcast, tensor:  tensor([0, 0, 4], device=‘cuda:0’)<br>
local rank:  1 After broadcast, tensor:  tensor([0, 0, 4], device=‘cuda:1’)</p>
What is the proper arrangement for creating a convolutional block?

Jeev: <p>Hi, I want to learn how to create a custom convolutional block in PyTorch. I will make it like this</p>
<blockquote>
<p>nn.conv2d(),<br>
nn.ReLU(),<br>
nn.BatchNorm2d(),</p>
</blockquote>
<p>or like this</p>
<blockquote>
<p>nn.conv2d(),<br>
nn.BatchNorm2d(),<br>
nn.ReLU(),</p>
</blockquote>
<p>Which one is correct, and please explain why it should be arranged like that?</p>

tom: <p>ResNet uses the latter and while people have said things about why, it would seem that the main thing is that it works well in practice.<br>
That said, with bias in BN initialized to 0, you have that the inputs to relu are 50% &gt; 0 and 50% &lt; 0 if the distribution is somewhat symmetric, so the relu “does something” and the outputs don’t all vanish.<br>
Also note that quite often, this order is in the context of a residual block, so 0 means that the skip connection’s term is passed forward, which might not be a bad thing to have reasonably often.<br>
But again, if the other thing would work better, people would use it and invent stories why that way is the best.</p>
<p>Best regards</p>
<p>Thomas</p>

Jeev: <aside class="quote no-group" data-username="tom" data-post="2" data-topic="192802">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>Best regards</p>
<p>Thomas</p>
</blockquote>
</aside>
<p>Thank you for your response, but I’m confused about why when the bias in Batch Normalization (BN) is initialized to 0, half of the input to the ReLU function will be greater than 0, and the other half will be less than 0. Also, what is meant by bias here? Is it the same as weight parameters?</p>
<p>I apologize for asking too many questions, it’s because I don’t know anything yet and am just starting to learn about CNN.</p>

tom: <aside class="quote no-group" data-username="Jeev" data-post="3" data-topic="192802">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeev/48/65883_2.png" class="avatar"> Jeev:</div>
<blockquote>
<p>Thank you for your response, but I’m confused about why when the bias in Batch Normalization (BN) is initialized to 0, half of the input to the ReLU function will be greater than 0, and the other half will be less than 0. Also, what is meant by bias here? Is it the same as weight parameters?</p>
</blockquote>
</aside>
<p>Oh, that was sloppy. If you have an distribution with median matches approximately the mean and the batch norm sets the mean to 0, you have that half the inputs are &lt; 0. The bias is added to the batch norm at the end (<a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="noopener nofollow ugc">γ in the formula</a>).</p>
<p>Best regards</p>
<p>Thomas</p>
How LSTMCell implemented in C++?

Sun-Xiaohui1: <p>Hello, I am confused about LSTMCell’s parameters and variables. In the docs, it says <code>weight_ih - of shape (4*hidden_size, input_size)</code> and <code>input - of shape (batch, input_size)</code>. Obviously, they cannot be calculate like the equation <code>i = σ(Wii * x + bii + Whi * h + bhi </code>. So I guess some dimension transformation are completed in C++.<br>
I want to know, besides dimension transformation, are there any other modifycations on the parameters and variables?   Because when I implemented LSTMCell in JAX, and keep the same inputs, weights and bias as Pytorch’s, I got different outputs form Pytorch’s LSTMCell’s outputs.<br>
Here is my LSTMCell’s implement.   Note:<code>self.Wi - of shape (input_size, hidden_size*4), self.Wh - of shape (hidden_size, hidden_size*4), b - of shape (hidden_size*4,) </code></p>
<pre><code class="lang-auto">def __call__(self, carry: Tuple, x: jnp.ndarray):
    h, c = carry
    gated = x @ self.Wi + self.bi
    gated += (h @ self.Wh + self.bh)
    
    i, g, f, o = jnp.split(gated, indices_or_sections=4, axis=-1)
    c = jax.nn.sigmoid(f) * c + jax.nn.sigmoid(i) * jax.nn.tanh(g)
    h = jax.nn.sigmoid(o) * jax.nn.tanh(c)
    return (h, c), h
</code></pre>
<p>Thanks！</p>
Computing Jacobian/Hessian Vector Product

mohammedehab2002: <p>Hi!<br>
I’m working on a research problem where I have a GAN, and I need to compute the following: Let $A_{ij}=\frac{L}{\theta^G_i \theta^D_j}$ where $L$ is the loss function. I need to compute $Av$ for a vector $v$. How can I do that?<br>
One thing I tried is to compute the Hessian-vector product using autograd.functional.hvp on an auxiliary function that I feed (D.parameters(),G.parameters()), and it returns the loss. However, I get all zeros because the function dosen’t directly depend on its inputs. How can I fix this?<br>
Thanks in advance!</p>
How can I resize image tensors?

_lovepytorch: <p>I have a tensor - batch of images with shape [32, 3, 640, 640] and values in the range [0, 1] after diving by 255. How can I resize that tensor to [32, 3, 576, 576]?</p>
<p>I see the option <code>torch.nn.funtional.interporlate</code> and the documentation wrote <code>The input dimensions are interpreted in the form: mini-batch x channels x [optional depth] x [optional height] x width.</code> I am not clear this part about <code>[optional depth] x [optional height] x width</code> because in my case I have 576x576. Thanks.</p>

ptrblck: <p>This should work:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(32, 3, 640, 640)
y = F.interpolate(x, (576, 576))
print(y.shape)
# torch.Size([32, 3, 576, 576])
</code></pre>

_lovepytorch: <p>Thanks.<br>
It worked properly after comparing with OpenCV.</p>