akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cuda drivers error

username: Mbakop_Dimitri
cooked: <p>I am trying to run the Masked Siamese Networks for Label-Efficient Learning on a remote server but i go this error “RuntimeError: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: <a href="http://www.nvidia.com/Download/index.aspx" class="inline-onebox" rel="noopener nofollow ugc">Official Drivers | NVIDIA</a> Alternatively, go to: <a href="https://pytorch.org" rel="noopener nofollow ugc">https://pytorch.org</a> to install a PyTorch version that has been compiled with your version of the CUDA driver.”</p>
<p>the driver on the server are “NVIDIA-SMI 470.239.06   Driver Version: 470.239.06   CUDA Version: 11.4”<br>
and the torch i got in my env are "<br>
“# Name                    Version                   Build  Channel<br>
pytorch-cuda              12.1                 ha16c6d3_5    pytorch<br>
pytorch-mutex             1.0                        cuda    pytorch<br>
torch                     2.3.0                    pypi_0    pypi<br>
torchaudio                2.2.0                py38_cu121    pytorch<br>
torchtriton               2.3.0                      py38    pytorch<br>
torchvision               0.18.0                   pypi_0    pypi”</p>
<p>Do I need to downgrade the Pytorch version ? Since I cannot update the driver ?</p>

username: ptrblck
cooked: <p>Update your driver to use the PyTorch binaries with CUDA 12 or install the PyTorch binaries with CUDA 11.</p>
Training model using annotated image

username: Surekha_Gaikwad
cooked: <p>Hi, I am working on emotion recognition project where I am using EMOTIC dataset for training my model. All these images are annotated and all the info are stored in torch (.t7) file e.g “DiscreteContinuousAnnotations26_train.t7” for training images. In these images required person is annotated by bounding box whose emotions model needs to capture and learn gradually.<br>
But I am facing issue where I don’t know how to read this torch file and also I am not getting how to feed this annotation file to model so that it will capture required person in the bounding box and will learn its expression.</p>
<p>Please help as I am doing this end to end project first time and I am quite beginner in this area.</p>

username: Surekha_Gaikwad
cooked: <p>I just want to know how to train model with images having bounding box (as label) around particular person so that it can learn only features of that person instead of entire image containing other objects and persons.</p>

username: AMARNATH_REDDY
cooked: <p>Hi Surekha_Gaikwad,<br>
I am also try to train annotated model.</p>
<p>You got any results for how to train annotated images with torch?</p>
Libtorch build for ROCM error: "aten/src/THH" not exist

username: Fov
cooked: <p>I want to build libtorch for rocm from source and followed the instructions from here:<a href="https://github.com/pytorch/pytorch/blob/main/docs/libtorch.rst" rel="noopener nofollow ugc">Building libtorch using CMake</a>. It occured this error:</p>
<pre><code class="lang-auto">CMake Error at aten/CMakeLists.txt:82 (add_subdirectory):
  add_subdirectory given source "src/THH" which is not an existing directory.
ROCm is enabled.
</code></pre>
<p>Directory “pytorch/aten/src/THH” does’t exist but it should be. According to pytorch/aten/CMakeLists.txt:</p>
<pre><code class="lang-auto">if(USE_ROCM)
  # TODO: AT_HIP_ENABLED (change this once we represent HIP as HIP in
  # ATen proper)
  set(AT_CUDA_ENABLED 1)
  add_subdirectory(src/THH)
  message("ROCm is enabled.")
elseif(USE_CUDA)
  set(AT_CUDA_ENABLED 1)
  add_subdirectory(src/THC)
</code></pre>
<p>If build for cuda, there exists the directory “pytorch/aten/src/THC”, but for rocm, the corresponding directory “THH” does not exist. I‘ve searched several versions pytorch source code, none of them have this directory.<br>
So can someone tell me where can get this lost directory? Or have I missed something to install? I would appreciate for your help with this problem.</p>
How to convert a matrix to heatmap image in torch

username: Ahmad_Pouramini
cooked: <p>suppose a matrix like:</p>
<pre><code>&gt;&gt;&gt; a
tensor([[0.1000, 0.2000, 0.3000],
        [0.4000, 0.5000, 0.6000],
        [0.7000, 0.8000, 0.9000]])
</code></pre>
<p>I would like to visualize it as a heatmap image. I tried:</p>
<pre><code>img = PIL.Image.fromarray(a.numpy())
</code></pre>
<p>However this image is too small in dimensions (3x3). I would like to scale or expand it so that I can view it better. What is the solution?</p>

username: Ahmad_Pouramini
cooked: <p>I found this answer:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/33282368/plotting-a-2d-heatmap">
  <header class="source">

      <a href="https://stackoverflow.com/questions/33282368/plotting-a-2d-heatmap" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3320135/karnivaurus" target="_blank" rel="noopener nofollow ugc">
    <img alt="Karnivaurus" src="https://www.gravatar.com/avatar/?s=256&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/33282368/plotting-a-2d-heatmap" target="_blank" rel="noopener nofollow ugc">Plotting a 2D heatmap</a>
</h4>

<div class="tags">
  <strong>python, matplotlib, seaborn, heatmap</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/3320135/karnivaurus" target="_blank" rel="noopener nofollow ugc">
    Karnivaurus
  </a>
  on <a href="https://stackoverflow.com/questions/33282368/plotting-a-2d-heatmap" target="_blank" rel="noopener nofollow ugc">01:37PM - 22 Oct 15 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: FM_LYD
cooked: <p>Thank u, i find this is very useful</p>
Tensor Comprehensions and einpos how to use for made a rolling metrics between two vectors

username: nicolalandro
cooked: <p>I need to implement a cosine similarity between a smallest 1D tensor A and a longer 1D tensor B, I want to iterate the B tensor like a 1D conv and make cosine distance like the following pseudo code:</p>
<pre><code class="lang-auto">cosine_sim = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
def cosine_rolling_1(a, b):
    for i in range(len(b) - len(a) + 1):
        yield cosine_sim(a, b[i:i+len(a)])

def cosine_rolling_2(a, b):
    for i in range(len(b) - len(a) + 1):
        yield torch.dot(a, b[i:i+len(a)])/(torch.norm(a)*torch.norm(b[i:i+len(a)]))
</code></pre>
<p>I thin that <a href="https://einops.rocks/" rel="noopener nofollow ugc">einpos</a> is not food because I have two input but also with <a href="https://github.com/facebookresearch/TensorComprehensions" rel="noopener nofollow ugc">Tensor Comprehension</a> i dont think that I can write something like that:</p>
<pre><code class="lang-auto">def cosine_rolling_tc(float(R) A, float(C) b) -&gt; (o) {
    o(r) +=! dot(A, b(r-c:r-c+R))/(norm(A)*norm(b(r-c:r-c+R)))
}
</code></pre>
<p>What is the best way to implement in torch that operation? Do I need to use low level cuda?</p>
NaN loss in training loop

username: Danibahnani
cooked: <p>Hi!<br>
I’ve designe a network for a regressive task using LSTM.However, my loss becomes NaN when I try to train it and I don’t understand why.</p>
<p>This is my training loop:</p>
<pre><code class="lang-auto">def training_loop(dataloader,model,loss_fn,report_loss_fn,optimizer, batch_size, epoch,report_freq=10, writer=writer):
    size = len(dataloader.dataset)

    model.train()
    for batch_n, (X,y,x_mask,y_mask) in enumerate(train_dataloader):
        pred = model(X)

        #validate predictions by selecting non-padded values
        valid_pred = torch.masked_select(pred,y_mask)
        valid_targets = torch.masked_select(y,y_mask)

        #calculate optim loss
        loss = loss_fn(valid_pred,valid_targets)
        loss.backward()
        
        #adjust params
        optimizer.step()
        optimizer.zero_grad()
        
        #get loss for each pass
        running_loss = loss.item()

        if batch_n % report_freq == 0:
            current_values = batch_n*batch_size + len(X)
            writer.add_scalar('Train optim loss',running_loss,epoch+1)
            print(f"Train loss: {running_loss:&gt;7f} [{current_values:&gt;5d}/{size:&gt;5d}]")
</code></pre>
<p>The print statements print this, where it all nan after:</p>
<pre><code class="lang-auto">Train loss: 1.087582 [    4/ 1634]
Train loss: 0.840909 [  104/ 1634]
Train loss:     nan [  204/ 1634]
Train loss:     nan [  304/ 1634]
Train loss:     nan [  404/ 1634]
</code></pre>
<p>I’m using this optimizer:<br>
<code>optim = torch.optim.Adam(params=lstm_model.parameters(recurse=True),weight_decay=1e-05,lr=0.0001)</code></p>
<p>I’ve tried changing the learning rate, when I increased it to 0.5, the first 3-4 were not nan, so i think it could be something related to that.</p>
Which PyTorch version is compatible with H100 GPU and CUDA capability sm_90?

username: Daniel_Elias
cooked: <p>my CUDA Version: 12.4<br>
my PyTorch version: 1.13</p>
<p>Error: “NVIDIA H100 80GB HBM3 with CUDA capability sm_90 is not compatible with the current PyTorch installation”</p>
<p>Will Pytorch 2.2 work? PyTorch 1.13 appears to only support until sm_86</p>
<p>Or is there any other workaround?</p>

username: Daniel_Elias
cooked: <p>I see PyTorch 2.1 has sm_90:</p>
<ul>
<li>Torch: 2.1.0+cu121</li>
<li>Torch: [‘sm_50’, ‘sm_60’, ‘sm_70’, ‘sm_75’, ‘sm_80’, ‘sm_86’, ‘sm_90’]</li>
</ul>
<p>You can check on this with:</p>
<p>print(torch.<strong>version</strong>)<br>
print(torch.cuda.get_arch_list())</p>

username: ptrblck
cooked: <p>Every current stable and nightly release will support your H100.</p>
Memory bandwidth profiler?

username: hzzt
cooked: <p>Does pytorch’s profiler support memory bandwidth profiling? I only found memory “allocation” profiling in the docs. I want to determine whether or not my program is bounded by memory bandwidth. Are there any tools to do that?</p>

username: justairchem
cooked: <p>I’m commenting on this in the hopes that someone more informed than I am might respond, but I had the same question. I was wondering if it would be valid to determine the memory bandwidth of a particular operation by simply dividing the memory allocated by the average time reported by the profiler?</p>
Any way to improve device mismatch errors?

username: Soumya_Kundu
cooked: <pre><code class="lang-auto"> mask = (pred_cc_volume != 0) &amp; (~torch.isin(pred_cc_volume, tp))
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument test_elements in method wrapper_CUDA_isin_Tensor_Tensor)
</code></pre>
<p>Any way to gauge which is on what device from this error? Or is it worth putting in an effort to generate a pr/issue?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/soumya_kundu">@Soumya_Kundu</a>.</p>
<p>You can try calling the <code>.device</code> method for both of your tensors, <code>pred_cc_volume</code> and <code>tp</code> to see which one is on the GPU or CPU.</p>
<p>If you want to issue a pull request, and modify the error message, the link is here <a href="https://github.com/pytorch/pytorch/pulls" class="inline-onebox" rel="noopener nofollow ugc">Pull requests · pytorch/pytorch · GitHub</a></p>
PyTorch 0.3.1 Code to Latest Version of Ubuntu and GPU?

username: jamesjang26
cooked: <p>Hello PyTorch Community,</p>
<p>I am currently working on a research project that involves an old model developed using PyTorch 0.3.1. Given the advancements in hardware and software, I need to update this model to run on my modern computing environment. Below are the details of my current setup and the libraries involved:</p>
<h3><a name="current-hardware-and-software-setup-1" class="anchor" href="#current-hardware-and-software-setup-1"></a>Current Hardware and Software Setup</h3>
<ul>
<li><strong>Operating System</strong>: Ubuntu 20.04</li>
<li><strong>GPU</strong>: 2x NVIDIA 3090 GPUs</li>
<li><strong>CUDA Version</strong>: 12.3 (Base installation)</li>
</ul>
<h3><a name="original-models-environment-2" class="anchor" href="#original-models-environment-2"></a>Original Model’s Environment</h3>
<ul>
<li>six</li>
<li>tqdm</li>
<li>torch==0.3.1</li>
<li>torchtext==0.2.3</li>
<li>future</li>
<li>pyxdameraulevenshtein</li>
</ul>
<h3><a name="situation-3" class="anchor" href="#situation-3"></a>Situation</h3>
<p>The code was initially designed for an environment with much older hardware and software versions. I am facing compatibility issues, particularly with the CUDA and PyTorch versions. Specifically, PyTorch 0.3.1 does not support CUDA 12.3, and there are substantial API changes between PyTorch 0.3.1 and the current version.</p>
<h3><a name="request-4" class="anchor" href="#request-4"></a>Request</h3>
<p>Could anyone provide guidance or best practices on how to effectively update my old PyTorch 0.3.1 code to be compatible with the latest version of PyTorch? I am looking for concrete steps or examples to facilitate this transition.</p>

username: ptrblck
cooked: <p>I would suggest the pragmatic way of executing the code and fixing issues based on warnings and errors. E.g. I would assume <code>Variable</code>s were used, which should be removed (you can still fine their usage in current code even though they were deprecated since <code>torch==0.4</code>).</p>
Simplest collate for returning a list in each batch

username: mtanti
cooked: <p>If I want to include a list of sentences in a data set item, like this:</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
	def __init__(self):
		self.items = [
			{'number': 1, 'sents': ['sent1_1', 'sent1_2']},
			{'number': 2, 'sents': ['sent2_1', 'sent2_2']},
			{'number': 3, 'sents': ['sent3_1', 'sent3_2']},
		]
	def __len__(self):
		return len(self.items)
	def __getitem__(self, idx):
		return self.items[idx]
</code></pre>
<p>And then use a data loader with a default collate function, like this:</p>
<p><code>next(iter(torch.utils.data.DataLoader(Dataset(), batch_size=2)))</code></p>
<p>The default collate function will group the sentences across items like this:</p>
<p><code>{'number': tensor([1, 2]), 'sents': [('sent1_1', 'sent2_1'), ('sent1_2', 'sent2_2')]}</code></p>
<p>When what I want is for the lists of sentences within each item to be kept together like this:</p>
<p><code>{'number': tensor([1, 2]), 'sents': [('sent1_1', 'sent1_2'), ('sent2_1', 'sent2_2')]}</code></p>
<p>What’s the simplest collate function that will do this for me?</p>

username: Ruben_Band
cooked: <p>It seems like this is a python related question and not so much a torch related question, since you are working with lists, dictionaries and strings. Getting the numbers in a torch tensor however can be done like this:</p>
<pre><code class="lang-auto">torch.tensor([x.number for x in self.items])
</code></pre>
<p>Hope this helps a bit.</p>
How to `pack_sequence` and `PackedSequence`

username: KiwiKraze
cooked: <p>Hi! I can’t find a up to date example that uses <code>pack_sequence</code> and its output <code>PackedSequence</code> in the context of a RNN-like network. That is, I have a model that processes a sequence timestamp by timestamp.</p>
<p>As far as I understand, earlier functions <code>pad_sequence</code>, <code>pack_padded_sequence</code>, <code>pad_packed_sequence</code> can be replaced by the newer <code>pack_sequence</code> (at least for the common use case).</p>
<p>I think I got my head around on how to use <code>pack_sequence</code> and I wonder if you could provide some feedback on whether I am right. A minimal working example is:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn
from torch.nn.utils.rnn import pack_sequence

BATCH_SIZE = 5
N_SAMPLES = 7


class Dataset(torch.utils.data.Dataset):
    def __init__(self, n=25):
        super().__init__()
        self.n = n

    def __len__(self):
        return self.n

    def __getitem__(self, idx):
        l = torch.randint(low=5, high=10, size=(1,))
        x = torch.arange(l.item(), dtype=torch.float)
        y = x * 2.0
        return x, y


def collate_pack(batch):
    xx, yy = zip(*batch)

    return (
        pack_sequence(xx, enforce_sorted=False),
        pack_sequence(yy, enforce_sorted=False),
    )


dataset = Dataset(n=N_SAMPLES)

dataloader = torch.utils.data.DataLoader(
    dataset, batch_size=BATCH_SIZE, collate_fn=collate_pack
)

model = nn.Linear(in_features=1, out_features=1)

for i_batch, batch in enumerate(dataloader):
    print("Batch=", i_batch)
    print("-" * 20)

    x_pkd, y_pkd = batch

    idx_start = 0
    for t, batch_size in enumerate(x_pkd.batch_sizes):
        print("Processing time t=", t)

        x, y = (
            x_pkd.data[idx_start : idx_start + batch_size],
            y_pkd.data[idx_start : idx_start + batch_size],
        )
        x = x.unsqueeze(1)  # (batch) -&gt; (batch, seq_length = 1)
        y = y.unsqueeze(1)  # (batch) -&gt; (batch, seq_length = 1)
        print("\tInput:", x.shape, y.shape)

        y_hat = model(x)
        print("\tOutput:", y_hat.shape)

        idx_start += batch_size
        print()

print("Done!")
</code></pre>
<p>The idea is that:</p>
<ul>
<li>The dataset contains sequences of (x, y) of different length</li>
<li>Because the sequence length is different, the dataloader need a different collate function. Here is where I use <code>pack_sequence</code></li>
<li>When I use dataloader, the batch will be a <code>PackedSequence</code>. As I loop through each element in the sequence, the <em>effective</em> batch size will change (i.e., later timestamps will contain only the values of the longer sequences)</li>
</ul>
<p>It seems to work, but I don’t like very much the pattern</p>
<pre><code class="lang-auto">x_pkd.data[idx_start : idx_start + batch_size],
</code></pre>
<p>Do you know if <code>PackedSequence</code> has already a method to get the right values directly?</p>
Computer vision OCR

username: saurabh_yadav
cooked: <p>I am working on ocr for Identity cards, my ocr( using YOLO) is quite good, but facing one challenge, when id’s are rotated (0-360), my model is not doing good. I want to train a model which can detect angle of rotation( same like google lens, which automatically detects the angle of rotation first then it performs the ocr.<br>
Is there any model for detecting the angle of rotation ( normal methods like HoughLines) is not consistent with my requirements.<br>
here is an example of few of the ids, I am working on</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/eaacae0fb7186c3b7e905635c3384106bcae139e.png" data-download-href="https://discuss.pytorch.org/uploads/default/eaacae0fb7186c3b7e905635c3384106bcae139e" title="download"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaacae0fb7186c3b7e905635c3384106bcae139e_2_517x500.png" alt="download" data-base62-sha1="xu1GYmD2JyARmMlxPG6nSBlxRlc" width="517" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaacae0fb7186c3b7e905635c3384106bcae139e_2_517x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaacae0fb7186c3b7e905635c3384106bcae139e_2_775x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/a/eaacae0fb7186c3b7e905635c3384106bcae139e.png 2x" data-dominant-color="999390"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">download</span><span class="informations">821×793 458 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
CUDA Path Not Correctly Configured

username: Suzaku_Kururugi
cooked: <p>Hello,<br>
I am trying to configure Pytorch with CUDA support. However, torch.cuda.is_available() keeps on returning false.</p>
<p>Checking nvidia-smi, I am using CUDA 10.0. I modified my bash_profile to set a path to CUDA.</p>
<pre><code class="lang-auto">export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64\{LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
</code></pre>
<p>When attempting to use CUDA, I received this error.<br>
<code>THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=50 error=30 : unknown error</code></p>
<p>Is there some solution?</p>

username: LeviViana
cooked: <p>You can always try to set the environment variable <code>CUDA_HOME</code>. By the way, one easy way to check if torch is pointing to the right path is</p>
<pre><code class="lang-python">from torch.utils.cpp_extension import CUDA_HOME
print(CUDA_HOME) # by default it is set to /usr/local/cuda/
</code></pre>

username: Suzaku_Kururugi
cooked: <aside class="quote no-group" data-username="LeviViana" data-post="2" data-topic="63695">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>from torch.utils.cpp_extension import CUDA_HOME</p>
</blockquote>
</aside>
<p>Interestingly, I got no CUDA runtime found despite assigning it the CUDA path. <strong>nvcc</strong> did verify the CUDA version.</p>

username: LeviViana
cooked: <p>That’s weird.</p>
<p><code> CUDA_HOME=a/b/c python -c "from torch.utils.cpp_extension import CUDA_HOME; print(CUDA_HOME)"</code></p>
<p>This prints <code>a/b/c</code> for me, showing that torch has correctly set the CUDA_HOME env variable to the value assigned.</p>

username: Suzaku_Kururugi
cooked: <p>It detected the path, but it said it can’t find a cuda runtime.</p>
<p><code>No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-10.0'</code></p>

username: samm
cooked: <p>excuse me did you solve it ?</p>

username: Ensiyeh_Raoufi
cooked: <aside class="quote no-group quote-modified" data-username="LeviViana" data-post="4" data-topic="63695">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>CUDA_HOME=a/b/c python -c “from torch.utils.cpp_extension import CUDA_HOME; print(CUDA_HOME)”</p>
</blockquote>
</aside>
<p>Did you find the solution to this issue? I have the same problem. Thanks</p>
Connected Components

username: shounak_cy
cooked: <p>I would like to know if there a Pytorch Implementation or library of Connected Components? I am doing cv2.contours on a binary mask on CPU, instead I want to move this to GPU and use connected components.</p>
<p>Any suggestions ?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p><code>Kornia</code> seems to have implemented the connected components labeling algorithm in <a href="https://github.com/kornia/kornia/pull/1184">this PR</a>. CC <a class="mention" href="/u/edgarriba">@edgarriba</a></p>
Compiled function 4000x slower than non-compiled one

username: Arthur_B
cooked: <p>Consider</p>
<pre><code class="lang-auto">import torch
import time

torch.set_float32_matmul_precision('high')

@torch.compile
def product_loop_compiled(A, B, iter):
    Y = torch.zeros_like(A, device='cuda')
    for _ in range(iter):
        X = A @ B
        Y += X
    return Y

def product_loop_normal(A, B, iter):
    Y = torch.zeros_like(A, device='cuda')
    for _ in range(iter):
        X = A @ B
        Y += X
    return Y

def testitest(iter, compile=False, n=50):
    A = torch.randn(n, n, device='cuda')
    B = torch.randn(n, n, device='cuda')
    if compile:
        product_loop_compiled(A, B, 100) # burn in
    else:
        Y = product_loop_normal(A, B, 100) # burn in
    torch.cuda.synchronize()
    start = time.time()
    if compile:
        Y = product_loop_compiled(A, B, iter)
    else:
        Y = product_loop_normal(A, B, iter)
    torch.cuda.synchronize()
    duration = time.time() - start
    print(duration / iter * 1000, 'ms')
    return Y

testitest(5000, compile=True)
testitest(5000, compile=False)
</code></pre>
<p>Running it gives</p>
<pre><code class="lang-auto">40.336214542388916 ms
0.013336086273193359 ms
</code></pre>
<p>It looks like the compiled code is 4000x slower. How is that possible?<br>
I’ve heard of compiled code sometimes being slower in torch, but <em>4000x</em> make me think I’m the one doing something wrong.</p>
<p>This is with a NVIDIA GeForce RTX 3060, 12GB, python 3.11.6 and torch 2.2.0+cu121</p>
<p>What am I missing?</p>
<p>(edit: 2.3.0 gives the same type of time discrepancy)</p>
Using PyTorch DataLoader for data loading and TF as the training backend

username: Rajveer_B
cooked: <p>Is it possible to use PyTorch DataLoader for data loading and TF as the training backend?</p>

username: ptrblck
cooked: <p>This should theoretically be possible unless the multiprocessing usage in the <code>DataLoader</code> conflicts somehow with TF (I would assume it should show a warning).</p>
How do I specify torch::nn::Upsample scale_factor?

username: Roger_Frye
cooked: <p>I am tryting to specify an Upsample module in a <code>torch::nn::Sequential</code> argument list. The documentation in<br>
<code>moose/framework/contrib/libtorch/include/torch/csrc/api/include/torch/nn/options/upsampling.h</code><br>
gives the example<br>
<code>/// model(UpsampleOptions().scale_factor({3}).mode(torch::kLinear).align_corners(false));</code><br>
but I don’t know how to apply that information.<br>
I have tried this line<br>
<code> torch::nn::Upsample(),</code><br>
and it compiles but gets the runtime error “either size or scale_factor should be defined”.<br>
I want to specify a scale_factor of 2. These are some of the ways that I have tried to implement this:</p>
<pre><code class="lang-auto">      torch::nn::Upsample(nullptr, 2.0),
      torch::nn::Upsample(void, 2.0),
      torch::nn::Upsample(2.0),
      torch::nn::Upsample(scale_factor{2.0}),
      torch::nn::Upsample(scale_factor={2}),
      torch::nn::Upsample(scale_factor({2})),
      torch::nn::Upsample(torch::nn::UpsampleOptions().scale_factor({2})),
      torch::nn::Upsample(torch::nn::UpsampleOptions(scale_factor({2}))),

</code></pre>
<p>but none of them compile.</p>

username: Roger_Frye
cooked: <p>I found the answer in <a href="https://pytorch.org/cppdocs/api/structtorch_1_1nn_1_1_upsample_options.html#structtorch_1_1nn_1_1_upsample_options" rel="noopener nofollow ugc">pytorch cppdocs</a></p>
<p><code>torch::nn::Upsample(torch::nn::UpsampleOptions().scale_factor(std::vector&lt;double&gt;({2}))),</code></p>
Faster way to do multiple embeddings?

username: AlliedToasters
cooked: <p>Hello everybody!</p>
<p>I’m working on a <a href="https://github.com/AlliedToasters/dfencoder" rel="nofollow noopener">torch-based library</a> for building autoencoders with tabular datasets.</p>
<p>One big feature is learning embeddings for categorical features.</p>
<p>In practice, however, training many embedding layers simultaneously is costing me some slowdowns. I am using for-loops to do this and running the for-loop on each iteration is (I think) what’s causing the slowdowns.</p>
<p>When building the model, I associate embedding layers with each categorical feature in the user’s dataset:</p>
<pre><code class="lang-auto">        for ft in self.categorical_fts:
            feature = self.categorical_fts[ft]
            n_cats = len(feature['cats']) + 1
            embed_dim = compute_embedding_size(n_cats)
            embed_layer = torch.nn.Embedding(n_cats, embed_dim)
            feature['embedding'] = embed_layer

</code></pre>
<p>Then, when I run .forward():</p>
<pre><code class="lang-auto">        embeddings = []
        for i, ft in enumerate(self.categorical_fts):
            feature = self.categorical_fts[ft]
            emb = feature['embedding'](codes[i])
            embeddings.append(emb)

       #num and bin are numeric and binary features
        x = torch.cat(num + bin + embeddings, dim=1)
</code></pre>
<p>Then <code>x</code> goes into dense layers.</p>
<p>This gets the job done but running this for loop during each forward pass really slows down training, especially when a dataset has tens or hundreds of categorical columns.</p>
<p>Does anybody know of a way of vectorizing something like this? Thanks!</p>

username: Kotzi
cooked: <p>Try to sum up the embedded vectors. And init the embedding layers near zero. I think this could potentially work like skip connections. Works for me in my autoregressive transformer which needs more features when 1 vocabulary and 1 timeembedding. It actually works for 50 different tokenizers in parallel.</p>
Use of the addition assignment operator (+=) on a tensor leads to a RuntimeError

username: RalfG
cooked: <p>This question is similar to the thread <a href="https://discuss.pytorch.org/t/runtime-error-in-backpropagation-of-the-graph-second-time/125301" class="inline-onebox">Runtime Error in backpropagation of the graph second time</a>, but I wanted to ask about the implementation logic behind the issue.</p>
<p>Using the addition assignment operator on a tensor with itself (<code>t += t</code>) leads to <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</code> during backprop. Replacing this operation with an individual assignment of the addition (e.g.: <code>t = t + t</code>) works without issues.</p>
<p>A minimal example to replicate the error:</p>
<pre><code class="lang-auto">import torch

# Define the model
class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear1 = torch.nn.Linear(3, 1)
        self.linear2 = torch.nn.Linear(1, 3)
        self.activation = torch.nn.ReLU()

    def forward(self, x):
        x = self.linear1(x)
        x = self.activation(x)
        x = self.linear2(x)
        x = self.activation(x)
        return x

model = Model()

# Define the optimizer and loss function
optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
loss_fn = torch.nn.MSELoss()

# Define the input and target tensors
x = torch.tensor([[1, 2, 3], [1, 2, 3]], dtype=torch.float32)
y = torch.tensor([[4, 5, 6], [7, 8, 9]], dtype=torch.float32)

# Perform a forward pass, compute the loss, perform a backward pass, and update the weights
model.train()
optimizer.zero_grad()
outputs = model(x)
outputs += outputs
loss = loss_fn(outputs, y)
loss.backward()
optimizer.step()
</code></pre>
<p>Is this behavior expected? I was always under the impression that <code>t += t</code> is simply syntactic sugar for <code>t = t + t</code>.</p>

username: JuanFMontesinos
cooked: <p>Yes, it’s expected.<br>
+= is an inplace operation that is defined in pythons <code>__iadd__</code><br>
which differs from a=a+x that relies on <code>__add__</code></p>
<p>The former is, as the error explains, in-place, and modifies the data directly in memory without using the mechanisms that pytorch needs for trazability of your ops.</p>
WeightedRandomSampler has low data loading efficiency

username: CYYJL
cooked: <p>Dear groupers,<br>
I work on an unbalanced dataset. I’ve encountered a strange phenomenon regarding WeightedRandomSampler. In my Docker environment with two RTX 3090 GPUs and 256 CPU cores, when I run the code using WeightedRandomSampler for sampling individually, it runs at a normal speed. However, when I run the program using both GPUs simultaneously, the data loading rate is very low, and the GPUs are constantly waiting for data input. I’m not sure how to solve this issue. I tried distributed training, but it seems that WeightedRandomSampler cannot be used for sampling in distributed training.<br>
Can anyone help me to check my problems? Thanks a lot!</p>
<p>The following is the configuration I used for WeightedRandomSampler.</p>
<pre><code class="lang-auto">sampler = WeightedRandomSampler(
                weights=weights,
                num_samples=len(train_dataset),
                replacement=True,
                generator=sampling_generator, 
            )
</code></pre>
<p>Best！<br>
CYYJL</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/weightedrandomsampler-distributedsampler/52817/2">This topic</a> might be helpful asking about a weighted and distributed sampler.</p>
Setting Pearson Correlation Coefficient as a Loss really doesn't work well

username: Camille_Bernheim
cooked: <p>Here is a Minimal Reproducible Example: <a href="https://colab.research.google.com/drive/175Hre7JqLYEYt5y_bBRZlcGCuXjbco7N?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a>.</p>
<p>I’m trying to set up a Custom Loss, the Pearson Correlation Coefficient Loss (i.e. 1-PearsonCorrelationCoefficient obviously).</p>
<p>I created this class, thinking this would work easily:</p>
<pre><code class="lang-auto">class PearsonLoss(nn.Module):
    def __init__(self):
        super(PearsonLoss, self).__init__()

    @staticmethod
    def forward(x: torch.Tensor, y: torch.Tensor):
        # Flatten the tensors
        x = x.view(x.shape[0], -1)
        y = y.view(y.shape[0], -1)

        # Mean centering
        x_mean = torch.mean(x, dim=1, keepdim=True)
        y_mean = torch.mean(y, dim=1, keepdim=True)
        x = x - x_mean
        y = y - y_mean

        # Numerator
        cov = torch.sum(x * y, dim=1)

        # Denominator
        x_std = torch.sqrt(torch.sum(x**2, dim=1))
        y_std = torch.sqrt(torch.sum(y**2, dim=1))

        # Pearson Correlation Coefficient
        pcc = cov / ((x_std * y_std) + 1e-8)

        # Pearson **Loss**
        loss = 1 - pcc

        # Sum loss across the batch
        return loss.sum()
</code></pre>
<p>But it has a lot of trouble converging (cf. my minimal reproducible example which is trying to map an image to literally the same image with a super small UNet like architecture).</p>
<p>Obviously it would work really well with a MSE loss for example, so I’m wondering. Did I implement something wrong for my Pearson Correlation Coefficient Loss?</p>
<p>I have to use it for a “bigger project” that doesn’t fit in a MRE and, similarly, nothing converges. Since my NN cannot even do the identity function, I guess there is indeed a problem with my Class but I can’t seem to find it.</p>
<p>Any help is appreciated!</p>
<p>EDIT: sometimes it does converge but most of the times it looks like it “gets stuck” in a local minimum:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de5e159ff0e3f815ec0197a364e2282dda2cd2bf.png" alt="image" data-base62-sha1="vJ9zyuUl8wdgPnIRfFvu7KnSQpx" width="599" height="239"></p>

username: linminhtoo
cooked: <p>have you tried <code>torch.corrcoef()</code> ?</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.corrcoef.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.corrcoef.html</a></p>
<p><code>torch.sqrt(x)</code> is not a stable function. if your <code>x</code> is close to 0, the gradient will be very high or NaN, because for a function y = sqrt(x), gradient dy/dx = -1/(2*sqrt(x))</p>
<p>also I’m not understanding why you want to use a Pearson correlation function to optimize “an image”. Something like MAE or MSE should work fine for images.</p>
Understanding Pytorch Profiler for GPU

username: Mayur_Peshve
cooked: <p>Hello!</p>
<p>I wanted to understand how the PyTorch profiler relates CPU-bound tasks with GPU-bound tasks in a trace. I understand that timestamps are used for this purpose, but how are the timestamps generated for the GPU-bound tasks? I do see that the CUPTI from NVIDIA is used but I wanted to understand the exact APIs used to generate timestamps or to correlate the two. If any can point me in the right direction, that would be really helpful!</p>
finBERT on large dataset

username: aolValentin
cooked: <p>Hi. I have a quite large dataset with three column: date, stock name and headline of a news related to the stock. I would like to apply finBERT to the headline in order to obtain a sentimental score, which will be used to evaluate a sentimental analysis.<br>
The problem is that the dataset has ~1 million rows, which makes this task impossible. Hence, my question is: does anyone know a possible way to approach to this problem? What can I do in order to run this all model on my dataset?</p>
Can I replicate a module for more than once?

username: Lynx_Commando
cooked: <p>Hi community,<br>
Is it safe to call nn.parallel.replicate on one module for more than once?</p>
<p>E.g</p>
<pre><code class="lang-auto">import torch
a=torch.nn.Linear(3,3);
a.cuda();
ar=torch.nn.parallel.replicate(a,[0,0]);
ar1=torch.nn.parallel.replicate(a,[0])[0];
</code></pre>
<p>Which seems to be fine in this toy example…</p>
<pre><code class="lang-auto">import torch
a=torch.nn.Linear(3,3);
a.cuda();
ar=torch.nn.parallel.replicate(a,[0,0]);
ar1=torch.nn.parallel.replicate(a,[0])[0];
d1=torch.rand(3,dtype=torch.float32,device="cuda");
d2=torch.rand(3,dtype=torch.float32,device="cuda");
o=torch.optim.SGD(a.parameters());
ar[0](d1).sum().backward();
ar[1](d2).sum().backward();
a.weight.grad;
o.zero_grad();
ar1(d1).sum().backward();
ar[0](d2).sum().backward();
a.weight.grad;
</code></pre>
<p>However I don’t know if the replicate function interactive with something global, esp in the origin module (a in this case), which would ruin my day if I use it in some more complex situations<br>
(e.g. puting ar[0], ar[1], ar1 on different devices or different streams)</p>
<p>Thanks in advance.</p>
<p>Regards,<br>
Cat</p>
Is there a way to pin a shared memory?

username: dem123456789
cooked: <p>I found out that if I pin the memory after I use share_memory_() inside a process, the memory would not be shared anymore because pin_memory() makes a copy. However, if I pin the memory in advance, it associates only one specific cuda device (default to be 0). In the mean time, you have to share_memory_ before using mp.spawn, so it seems like a dead end?</p>
<p>Is there way to make a share memory and pin simultaneously for multiple cuda devices?</p>
How to install pytorch cuda121 for aarm64 ec2 instance

username: Mohtashim
cooked: <p>I want to use a cuda121 pytorch version on my g5g-2xlarge an arm-based instance , following are my versions</p>
<p>CUDA: 12.2<br>
Driver Version: 535.161.08<br>
Ubuntu: 22.04 LTS aarm64</p>
<p>How to install pytorch version: 2.2.1+cu121 ?</p>
LoRA trainable parameters

username: Mees.elizabeth
cooked: <p>Hi,<br>
When I print my model’s parameters after adding LoRA to the decoder, I get:</p>
<pre><code class="lang-auto">...
up_tr256.up_conv.weight
up_tr256.up_conv.bias
up_tr256.ops.0.conv1.lora_A
up_tr256.ops.0.conv1.lora_B
up_tr256.ops.0.conv1.conv.weight
up_tr256.ops.0.conv1.conv.bias
up_tr256.ops.0.bn1.weight
up_tr256.ops.0.bn1.bias
up_tr256.ops.1.conv1.lora_A
up_tr256.ops.1.conv1.lora_B
up_tr256.ops.1.conv1.conv.weight
up_tr256.ops.1.conv1.conv.bias
up_tr256.ops.1.bn1.weight
up_tr256.ops.1.bn1.bias
up_tr128.up_conv.weight
up_tr128.up_conv.bias
...
</code></pre>
<p>Can I now freeze my model and only train ‘lora_’ params? Cause they don’t have .weight in their names.</p>
Use DDP to train a single model, on a single GPU, multiple processes

username: copythatpasta
cooked: <p>I am at a loss here. I would like to maximize the number of steps my model takes within a specific time block, and so instead of filling the batch size to max out the VRAM, I want to run multiple processes for this single model and aggregate the results as if it was being on on multiple devices, but is only running on a single GPU. I have more then enough VRAM to run 3-4 processes for this specific model, potentially giving me a 3x speed boost in traversing steps. Like I said, a larger batch size is out of the question, as it only reduces the number of steps for a given time frame.</p>
<p>How can I do this? I saw something about using ddp, gloo backend, and setting the device twice as seen here <a href="https://github.com/Lightning-AI/pytorch-lightning/discussions/8630" rel="noopener nofollow ugc">Emulating multiple devices with a single GPU · Lightning-AI/pytorch-lightning · Discussion #8630 · GitHub</a> but lightning complains when I add the same device twice. I really want to avoid writing anything from scatch to aggregate the gradients between processes, so out of the box solutions are preffered. <a class="mention" href="/u/ptrblck">@ptrblck</a>  please help</p>
Build pytorch-rocm without AVX2

username: zanmax
cooked: <p>Hello, I am trying to build PyTorch-Rocm without AVX2.</p>
<blockquote>
<h1>Clone PyTorch with submodules</h1>
<p>git clone --recursive <a href="https://github.com/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
cd pytorch</p>
<h1>Set environment variables to disable AVX2 and enable ROCm</h1>
<p>export NO_AVX2=1<br>
export USE_ROCM=1<br>
export ROCM_PATH=/opt/rocm<br>
export PATH=$PATH:/opt/rocm/bin:/opt/rocm/hip/bin</p>
<h1>Install Python dependencies</h1>
<p>pip install -r requirements.txt</p>
<h1>Run AMD build script for ROCm</h1>
<p>python tools/amd_build/build_amd.py</p>
<h1>Build PyTorch</h1>
<p>python setup.py install</p>
</blockquote>
<p>But every time I receive an error:</p>
<blockquote>
<p>1 error generated when compiling for gfx900.<br>
make: *** [Makefile.shim:27: flash/autotune.attn_fwd/FONLY__^fp16@16,1,16,False,True,True___MI200.o] Error 1<br>
make: *** Waiting for unfinished jobs…<br>
1 error generated when compiling for gfx900.<br>
1 error generated when compiling for gfx900.<br>
make: *** [Makefile.shim:23: flash/autotune.attn_fwd/FONLY__^fp16@16,1,16,True,False,False___MI200.o] Error 1<br>
make: *** [Makefile.shim:11: flash/autotune.attn_fwd/FONLY__^fp16@16,1,16,True,True,True___MI200.o] Error 1<br>
1 error generated when compiling for gfx900.<br>
1 error generated when compiling for gfx900.<br>
make: *** [Makefile.shim:15: flash/autotune.attn_fwd/FONLY__^fp16@16,1,16,True,True,False___MI200.o] Error 1<br>
make: *** [Makefile.shim:19: flash/autotune.attn_fwd/FONLY__^fp16@16,1,16,True,False,True___MI200.o] Error 1<br>
1 error generated when compiling for gfx900.<br>
make: *** [Makefile.shim:1290: flash/shim.attn_fwd.o] Error 1<br>
ninja: build stopped: subcommand failed.<br>
ninja: build stopped: subcommand failed.</p>
</blockquote>
<p><strong>My hardware:</strong><br>
CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz<br>
GPU: 5 x WX9100</p>
<p>Could you help me with this problem?</p>

username: waheed
cooked: <p>Hello Max, Which rocm version is that, what OS, Kernel, torch, python versions, also what does <code>env</code> output?</p>
How to build pytorch without Cuda statistical linkage

username: orangemaru
cooked: <p>I am intercepting some CUDA APIs during training, but certain kernels (e.g., cudaLaunchKernel) are not being intercepted by my code. I suspect this is because some kernels are compiled with static linkage, especially those from third-party libraries. Is there a way to force dynamic linkage when building PyTorch to address this issue?</p>

username: ptrblck
cooked: <p>You could try to set <code>CUDA_USE_STATIC_CUDA_RUNTIME=0</code> and rebuild PyTorch.</p>
Torch Version of Hyp2f1

username: NudnikShpilkis
cooked: <p>Is there a <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.hyp2f1.html" rel="noopener nofollow ugc">Gauss Hypergeometric Function 2F1</a> in Pytorch? I saw this <a href="https://github.com/google-deepmind/torch-cephes/blob/master/cephes/bessel/hyp2f1.c" rel="noopener nofollow ugc">code</a>, but I wasn’t sure if it’s applicable or still usable.</p>
Loading Weights for GloVe Model

username: SecretAsianMan92
cooked: <p>I am trying to load the weights from the GloVe file that I have downloaded and I am confused on some details. There is one example of loading a glove model on the debugger cafe website that has the line</p>
<pre><code class="lang-auto">self.linear1 = nn.Linear(embed_dim, num_classes)
</code></pre>
<p>The embed_dim variable make sense but I am not sure about the num_classes variable. The example that this is pulled from has 7 classes because it is trying to classify emotions. However, since I am not classifying emotions would that mean I use the number of words in the GloVe in this case 400000? or am I wrong in using a linear layer all together and I am actually supposed to use a sparse layer like nn.Embedding?</p>

username: SecretAsianMan92
cooked: <p>So I have an update to the problem. I tried a linear layer and used that in my code and it errored with the message: “The model was not an embedding” so it seems like I need to use the Embedding sparse layer. Now my issue seems to be tracing this model because I am getting the error:</p>
<p>RuntimeError: Expected tensor for argument <span class="hashtag-raw">#1</span> ‘indices’ to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)</p>
<p>I was successful in tracing the linear model with the line<br>
<code>traced_model = torch.jit.trace(embedding_model, torch.rand(50))</code><br>
but that doesn’t seem to be the case with this embedding layer</p>
Torch.cuda.max_memory_reserved example please

username: clam004
cooked: <p>I just want to know how many of a certain type of GPU I need to rent out that my training job will fit into based on the per GPU vRAM, ie the value that says <code>494MiB / 46068MiB</code> under <code>Memory-Usage</code> when you do <code>nvidia-smi -l</code>, so that I dont get a surprise CUDA OOM error during my training and need to decrease batch_size.</p>
<p>i think im probably misunderstanding the meaning of the word <code>reset</code> here</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html#torch.cuda.max_memory_allocated" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html#torch.cuda.max_memory_allocated</a></p>
<p>because I am not getting this to return to zero memory</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

def max_memory_allocated_reserved(device="cuda:0"):
    
    gpu_memory_allo_bytes = torch.cuda.max_memory_allocated(device="cuda:0")
    gpu_memory_allo_gb = gpu_memory_allo_bytes/1024 ** 3

    gpu_memory_res_bytes = torch.cuda.max_memory_reserved(device="cuda:0")
    gpu_memory_res_gb = gpu_memory_res_bytes/1024 ** 3

    print(f"{device} reserved {gpu_memory_res_gb} GB, allocated {gpu_memory_allo_gb} GB")

    torch.cuda.empty_cache()
    torch.cuda.reset_peak_memory_stats(device)

    return gpu_memory_res_gb, gpu_memory_allo_gb
    


import torchvision.models as models

device="cuda:0"

model = models.resnet18().to(device)

inputs = torch.randn(5, 3, 224, 224).to(device)

max_memory_allocated_reserved(device)

# zero memory ?

max_memory_allocated_reserved(device)

inputs = torch.randn(100, 3, 224, 224).to(device)

max_memory_allocated_reserved(device)

outputs = model(inputs)

max_memory_allocated_reserved(device)
</code></pre>
<p>I want to reset the memory “watching” interval and expect the 2nd set of print statements to be “0” as well as to clear “cuda:0” of the tensors, but even when inside <code>max_memory_allocated_reserved</code> i do</p>
<pre data-code-wrap="python"><code class="lang-python">del model
del inputs
</code></pre>
<p>i still see  <code>494MiB / 46068MiB</code> under <code>Memory-Usage</code> when I do <code>nvidia-smi -l</code>, further more, without deleting the tensors i still see 0.0464468 GB allocated and 0.0625 GB reserved.</p>
<p>thank you!</p>
Extracting model weight matrices and doing the inference just by matrix multiplication

username: payman
cooked: <p>Is it possible to extract weights of a trained model, and after that, do the inference just by matrix multiplication in any language or system, without the need for torchscript or exporting model to other formats?</p>
<p>(My problem is that I have a model trained in python (Tacotron 2 for TTS), now I want to use this model in C++ and also in mobile devices, for this it is required to first use torchscript to compile the model, but it seems that this model is not scriptable! I’ve already posted about this <a href="https://discuss.pytorch.org/t/can-not-port-checkpoint-to-mobile-using-torchscritp-cant-redefine-method-forward-on-class-torch-model-encoder/164343">here</a>)</p>