akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Assertion Error with torch.compile

duguyikinsa: <pre><code>model_compiled = torch.compile(model)
</code></pre>
<p>File “/home/anaconda3/envs/python3.9/site-packages/torch/<strong>init</strong>.py”, line 1441, in compile<br>
return torch._dynamo.optimize(backend=backend, nopython=fullgraph, dynamic=dynamic, disable=disable)(model)<br>
File “/home/anaconda3/envs/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py”, line 182, in __call__assert callable(fn)<br>
AssertionError.</p>
<p>And my version is as follows :<br>
nvidia-driver-515, cuda=11.7,python=3.9,torch=2.0</p>

marksaroufim: <p>Do you have a full repro?</p>

duguyikinsa: <p>Traceback (most recent call last):<br>
File “/home/ikenaga/student-data/liuyiyu/Spy,without_training1234P1.1+C2.1+C2.2HDRVideo-HRWeightNet/main.py”, line 35, in <br>
main(args)<br>
File “/home/ikenaga/student-data/liuyiyu/Spy,without_training1234P1.1+C2.1+C2.2HDRVideo-HRWeightNet/main.py”, line 14, in main<br>
model_compile = torch.compile(model)<br>
File “/home/ikenaga/anaconda3/envs/lyy-2.0/lib/python3.9/site-packages/torch/<strong>init</strong>.py”, line 1441, in compile<br>
return torch._dynamo.optimize(backend=backend, nopython=fullgraph, dynamic=dynamic, disable=disable)(model)<br>
File “/home/ikenaga/anaconda3/envs/lyy-2.0/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py”, line 182, in <strong>call</strong><br>
assert callable(fn)<br>
AssertionError</p>
<p>And here is the code:<br>
import torch<br>
from options import train_opts<br>
from utils import logger, recorders, train_utils, test_utils<br>
from datasets import custom_dataloader<br>
from models import build_model</p>
<p>args = train_opts.TrainOpts().parse()<br>
log  = logger.Logger(args)</p>
<p>def main(args):</p>
<pre><code>model = build_model(args, log)
model_compile = torch.compile(model)
recorder = recorders.Records(records=None)

train_loader, val_loader = custom_dataloader(args, log)

for epoch in range(args.start_epoch, args.epochs+1):
    model_compile.update_learning_rate()
    recorder.insert_record('train', 'lr', epoch, model_compile.get_learning_rate())
    
    train_utils.train(args, log, train_loader, model_compile, epoch, recorder)
    if epoch == 1 or (epoch % args.save_intv == 0): 
        model_compile.save_checkpoint(epoch, recorder.records)
    log.plot_all_curves(recorder, 'train')

    if epoch % args.val_intv == 0:
        test_utils.test(args, log, 'val', val_loader, model_compile, epoch, recorder)
        log.plot_all_curves(recorder, 'val')
</code></pre>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
torch.manual_seed(args.seed)<br>
main(args)</p>

marksaroufim: <p>I’m sorry I still need a fully functional minimal repro, as in something I can copy paste in google colab and reproduce the same error you’re seeing</p>
<p>I can guess that the problem is <code>save_checkpoint()</code> you would need to checkpoint the original model but I’m just guessing without a repro</p>
Computing nn.Module inverse or Backward pass

zshn25: <p>I think of an nn.Module to be a function f that transforms an input x into some output y as y = f(x). Assuming all my layers are invertible, is there a direct way to compute the backward pass for x = f^-1(y)?</p>

KFrank: <p>Hi Zeeshan!</p>
<aside class="quote no-group" data-username="zshn25" data-post="1" data-topic="179199" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zshn25/48/57310_2.png" class="avatar"> zshn25:</div>
<blockquote>
<p>is there a direct way to compute the backward pass for x = f^-1(y)?</p>
</blockquote>
</aside>
<p>If your layers are indeed invertible and I interpret “compute the backward pass”<br>
to mean compute the Jacobian of f^-1, then yes, this Jacobian is the matrix<br>
inverse of the Jacobean of f, J[f^-1] = (J[f])^-1.</p>
<p>This statement, together with important relevant context, can be found in<br>
Wikipedia’s <a href="https://en.wikipedia.org/wiki/Inverse_function_theorem" rel="noopener nofollow ugc">inverse-function-theorem</a> entry.</p>
<p>If instead you are asking how to compute f^-1 itself, that, in general, can be<br>
difficult.  You could either compute the inverses of each of your layers separately<br>
and then chain those inverses together, or you would compute  the inverse of f,<br>
x = f^-1 (y), by numerically solving for x in f (x) = y.</p>
<p>Best.</p>
<p>K. Frank</p>

zshn25: <p>Thank you for the reply. I was asking about the backward pass f^-1, but thanks for pointing the inverse-function-theorem. Is there a way to use Pytorch’s autodiff for such a computation?</p>

KFrank: <p>Hi Zeeshan!</p>
<aside class="quote no-group" data-username="zshn25" data-post="3" data-topic="179199" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zshn25/48/57310_2.png" class="avatar"> zshn25:</div>
<blockquote>
<p>thanks for pointing the inverse-function-theorem. Is there a way to use Pytorch’s autodiff for such a computation?</p>
</blockquote>
</aside>
<p>You may use <a href="https://pytorch.org/docs/2.0/generated/torch.autograd.functional.jacobian.html" rel="noopener nofollow ugc">jacobian()</a> or <a href="https://pytorch.org/docs/2.0/generated/torch.func.jacrev.html" rel="noopener nofollow ugc">jacrev()</a> to compute the Jacobian of your <code>Module</code> and<br>
<a href="https://pytorch.org/docs/2.0/generated/torch.linalg.inv.html" rel="noopener nofollow ugc">inv()</a> to compute its inverse.  Note the Jacobian will be returned as a tuple of tensors<br>
and you will have to reorganize it into a two-dimensional matrix in order to pass it<br>
to <code>inv()</code>.</p>
<p>Bear in mind that the mapping from input to output for a typical neural network<br>
will not be invertible.  Unless your neural network / <code>Module</code> is in fact invertible<br>
you won’t be able to invert the Jacobian.</p>
<p>Best.</p>
<p>K. Frank</p>
How to write a C++ extension operator that returns the same view

Inspiros: <p>I want to write a custom operator in C++/Cuda that takes the input tensor’s values and puts them into the output tensor with a complex rule. The output tensor is usually a lot larger with many duplications, so creating an empty tensor and copying the values is not memory-efficient.</p>
<p>The desired behavior is somewhat similar to <code>torch.broadcast_to</code>:</p>
<pre><code class="lang-python">x = torch.ones(3, 1)
y = torch.ops.aten.broadcast_to(x, (3, 5))
y[0, 0] = 0.5
print('y:', y)
print('x:', x)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">y: tensor([[0.5000, 0.5000, 0.5000, 0.5000, 0.5000],
           [1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
           [1.0000, 1.0000, 1.0000, 1.0000, 1.0000]])
x: tensor([[0.5000],
           [1.0000],
           [1.0000]])
</code></pre>
<p>Can someone please provide a tutorial on writing a C++/Cuda extension that returns the same memory view? Or is there a Pythonic solution?</p>
How to properly concat a Tensor according to different dimensions?

Thibault712: <p>Imagine I have a tensor of shape x = (Batch,…, Heads, Sequence, Dim)<br>
And let’s say I want to do a concat on the last dim but according to the heads, basically output a tensor of shape (Batch, …, Sequence, Heads * Dim), doing a</p>
<p><code>x.view(*x.shape[:-3], x.shape[-2], -1)</code></p>
<p>would return something that is not what I want.<br>
I found a trick but it’s bad coding:</p>
<p><code>x.transpose(-2, -3).reshape(*x.shape[:-2], -1)</code></p>
<p>Do you have a suggestion ?</p>
<p>EDIT the best I found so far:</p>
<pre><code class="lang-auto">x = torch.cat(x.unbind(dim=-3), dim=-1)
</code></pre>

Khumbaba: <p>I would probably have done it this way:</p>
<p>Assuming x’s shape is (Batch, Heads, Sequence, Dim):</p>
<p>x.permute(0,2,1,3).flatten(start_dim=2,end_dim=3) <span class="hashtag">#not</span> tested</p>
<p>Not sure which is the “best” way though.</p>
<p>EDIT: I might have understood badly. You want is to switch dimensions or what? if so it can be done with permute : x.permute(0,2,1,3).</p>
How to properly use mypy with PyTorch?

pedropgusmao: <p>Hi there,<br>
Could anyone please provide an example on how to setup mypy to properly work with PyTorch?<br>
I’m currently using :</p>
<pre><code class="lang-auto">[mypy-torch.*]
# https://github.com/pytorch/pytorch/issues/42787#issuecomment-672419289  
implicit_reexport = True
</code></pre>
<p>I am aware that typing with PyTorch is still a work in progress. So it would be very much appreciated if additional suggested flags could come with some explanation as to why we currently need them instead of pure <strong>strict</strong>.<br>
If you also feel this could be useful in your project, please comment or like this  post.<br>
Thanks!</p>

albanD: <p>Hi,</p>
<p>You can check our own mypy ini file here: <a href="https://github.com/pytorch/pytorch/blob/master/mypy.ini">https://github.com/pytorch/pytorch/blob/master/mypy.ini</a><br>
It is updated every time we add typing to new part of our code base.</p>

rqtz: <p>Is pytorch + mypy working out for you? I’ve got a very simple example that gives me trouble.</p>
<pre><code class="lang-auto">import torch.nn as nn

class Foo(nn.Module):
    pass
</code></pre>
<p>mypy complains:</p>
<pre><code class="lang-auto">error: Class cannot subclass "Module" (has type "Any")
</code></pre>

tchittesh: <p>I have the same issue. Is the only solution to ignore the mypy error using something like <code># type: ignore</code>?</p>
Error while creating an object of TorchDispatchMode

Vinayaka_Hegde: <p>Hello,</p>
<p>I’m importing torch dispatch mode like so: <code>from torch.utils._python_dispatch import TorchDispatchMode</code>.</p>
<p>I’m getting the below error when I create an object of torchdispatch mode. It would be great if someone could help me figure out the right way of creating the object for my version of PyTorch.</p>
<p>I’m using PyTorch <code>v1.12.0a0+02fb0b0f.nv22.06</code> on a Jetson Orin device</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf68b011c961c8e50896c1a7877a8b97555d8afe.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf68b011c961c8e50896c1a7877a8b97555d8afe" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf68b011c961c8e50896c1a7877a8b97555d8afe.png" alt="image" data-base62-sha1="tAP7ZubmsCSbDasHNW1d2PnbCUu" width="690" height="64" data-dominant-color="141414"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1540×143 6.56 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Code :</p>
<pre><code class="lang-auto">
class FlopCounterMode(TorchDispatchMode):
    def __init__(self, module = None):
        self.flop_counts = defaultdict(lambda: defaultdict(int))
        self.parents = ['Global']
        if module is not None:
            for name, module in dict(mod.named_children()).items():
                module.register_forward_pre_hook(self.enter_module(name))
                module.register_forward_hook(self.exit_module(name))


flop_counter = FlopCounterMode(mod)    # this line is causing the error
with flop_counter:
  optmizer.zero_grad()
  outputs = mod(inp)
</code></pre>
Decoder in U-Net with Pretrained ResNet34 Encoder

Anastasia_Berlianna: <p>i’m still trying to understand my decoder part. my input image size for image is 512x512 RGB, when i check the encoder output for each resnet block it gives<br>
layer1 = (64, 256, 256)<br>
layer2 = (128, 128, 128)<br>
layer3 = (256, 64, 64)<br>
layer4 = (512, 32, 32)<br>
i use bottleneck and the output is (1024,32,32)</p>
<p>my question is, when i use decoder part like the basic unet, it can’t return the image resolution to 512x512 in the end but instead 256x256, so am i right if i use convtranspose2d as upsampling and then use F.interpolate in the forward part?<br>
here’s my decoder code</p>
<pre><code class="lang-auto">class Dec_Block(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()

        # Up-Convolution
        self.upconv = nn.ConvTranspose2d(in_channels, in_channels // 2, kernel_size=2, stride=2)
        
        # LeakyReLU
        self.relu = nn.LeakyReLU()
        
        # Batch normalization
        self.bn = nn.BatchNorm2d(out_channels)

        # Basic Block
        self.conv = BasicBlock(in_channels // 2 + out_channels, out_channels)

    def forward(self, inputs, skip):
        up_x = self.upconv(inputs)
        up_x = self.relu(up_x)
        up_x = self.bn(up_x)  
        skip = nn.functional.interpolate(skip, size=up_x.size()[2:], mode='bilinear', align_corners=True)
        x = torch.cat([up_x, skip], dim=1)
        x = self.conv(x)

        return x
</code></pre>
Problem when connecting two machines using distributed pytorch

Jesus_Tovar_Trujillo: <p>I’m new using Pytorch distributed and I want to connect two machines in the same network using TCP, but nothing happens when try to connect, it just hangs up.</p>
<p>The code of master machine with IP 192.168.43.51.</p>
<pre><code class="lang-auto">import os
import torch
import torch.distributed as dist

def setup():
    os.environ['MASTER_ADDR'] = '192.168.43.51'
    os.environ['MASTER_PORT'] = '8000'
    os.environ['RANK'] = '0'
    os.environ['WORLD_SIZE'] = '2'

    # initialize the process group
    dist.init_process_group(backend='gloo', init_method='tcp://192.168.43.51:8000',rank=0,world_size=2)
    print("Connected ...")

if __name__ == "__main__":
	setup()
</code></pre>
<p>The execution command is:</p>
<pre><code class="lang-auto">python3 distributed_master.py
</code></pre>
<p>The code of the machine that try to connect with the master (IP: 192.168.43.240):</p>
<pre><code class="lang-auto">import os
import torch
import torch.distributed as dist

def setup():
    os.environ['MASTER_ADDR'] = '192.168.43.51'
    os.environ['MASTER_PORT'] = '8000'
    os.environ['RANK'] = '1'
    os.environ['WORLD_SIZE'] = '2'

    # initialize the process group
    dist.init_process_group(backend='gloo', init_method='tcp://192.168.43.51:8000',rank=1,world_size=2)
    print("Connected ...")

if __name__ == "__main__":
	setup()
</code></pre>
<p>The execution command is:</p>
<pre><code class="lang-auto">python3 distributed_client.py
</code></pre>
<p>First I run the master and then the client but there is no response, it seems to hang on the init_process_group method.</p>
<p>Can someone help me?<br>
Thanks!</p>

Jamie_Donnelly: <p>Try setting your <code>init_method</code> to <code>env://</code> which just tells the processes to look in the environment variables to find other processes.</p>

Jesus_Tovar_Trujillo: <p>I tried your answer but the result is the same.</p>

Jesus_Tovar_Trujillo: <p>Hi, I solved my problem by specifying in both the client and master code the variables:</p>
<p><strong>Master node:</strong></p>
<pre><code class="lang-auto">- GLOO_SOCKET_IFNAME = &lt;master network interface name&gt;
- TP_SOCKET_FINAME = &lt;master network interface name&gt;
</code></pre>
<p>The following rule must be introduced to the firewall: <code>sudo ufw allow from &lt;client IP&gt;</code></p>
<p><strong>Client node:</strong></p>
<pre><code class="lang-auto">- GLOO_SOCKET_IFNAME = &lt;client network interface name&gt;
- TP_SOCKET_FINAME = &lt;client network interface name&gt;
</code></pre>
<p>The following rule must be introduced to the firewall: <code>sudo ufw allow from &lt;master IP&gt;</code></p>
Best way to temporarily disable dropout

Ricu: <p>Hello everyone,</p>
<p>I want to set dropout to zero during one forward pass and re-enable it for the next foward pass. Forboth that, I still want to use components like batch norms in their training mode (i.e. NOT use running stats like exp. moving average).</p>
<p>My current approach is to use <code>model.eval()</code>. The problem with this is that this also changes other components like batch norms (see list <a href="https://stackoverflow.com/questions/66534762/which-pytorch-modules-are-affected-by-model-eval-and-model-train" rel="noopener nofollow ugc">here</a>) to use running stats instead of batch information.</p>
<p>Are there any convenient ways to temporarily disable dropout without affecting the other components?</p>
<p>As a bonus: If possible it would be nice to still have the flag <code>model.training</code> be set to <code>False</code> when disabling dropout as I could use this in another if statement.</p>
<p>Cheers</p>

ptrblck: <p>You could call <code>.eval()</code> on the dropout layer only or change its internal <code>.training</code> attribute by directly accessing it in the model.</p>

Ricu: <p>Thanks, I do the following for now:</p>
<pre><code class="lang-auto">dropout_modules = [module for module in model.modules() if isinstance(module,torch.nn.Dropout)]
[module.eval() for module in dropout_modules] # disable dropout
...
[module.train() for module in dropout_modules] # enable dropout
...

</code></pre>
<p>If I run into any further problems I will come back to this.</p>
<p>Cheers</p>
Use of retain_graph = True

Harsh_Choudhary: <p>I have my loss function as loss = loss1 + loss2.<br>
After each forward pass, i calculate the gradients using loss.backward() and update my weights. Where loss1(w,b) loss2(w,b) and loss(w,b) are functions of network parameters.</p>
<p>Now in every iteration of gradient descent, I need the gradients of loss1 and loss2 wrt the network parameters as well.So i use loss1.backward(retain_graph=True) and loss2.backward(retain_graph=True) . Is it the right approach?</p>
<p>Also, if you can explain what is actually happening while doing retain_graph= True and with False I would be grateful</p>

srishti-git1110: <p>Hi,<br>
<code>retain_graph=True</code> causes autograd NOT to aggressively free up the saved tensors required for grad computation after the backward call.</p>
<p>But for this, you will not be able to call . backward  on a tensor more than once as the intermediate tensors required for the backward pass shall already be freed.</p>
<p>For your case, using <code>retain_graph=True</code> should help if you aren’t running into any errors and everything is working as expected. Otherwise, feel free to post the error along with an executable code snippet.</p>

Harsh_Choudhary: <p>Thanks for your answer,</p>
<p>I do not get any error, but as you mentioned : “i cannot call .backward more than once …”, That explains the error that I was recieving when I was not setting the retain_graph = True, parameter while calculating the gradients of loss1 and loss2 in each step of my gradient descent as they use the same tensors for computation.</p>
<p>Again, many thanks for clarification</p>

ptrblck: <p>A small addition: make sure to use <code>retain_graph=False</code> (or just drop this argument as it’s the default) in the last <code>backward</code> call to allow PyTorch to free the intermediates.</p>
Get output size of a model

Yimeng_Zhu: <p>Hi there,</p>
<p>is there any way one can figure out the output dimension of a model without passing a sample to it?</p>
<p>For example, I have two network net1 and net2. I want to use another network net3, which maps the concatenation of net1 and net2 as the feature to some label. By defining the net3, I have to specify the input dimension which equals net1.output_size + net2.output_size.</p>
<p>My idea would be net1.modules[-1].out_features + net2.modules[-1].out_features. Is there a more elegant solution？</p>
<p>Thanks!</p>

ptrblck: <p>I think your approach is valid, if the last layer is an instance of <code>nn.Linear</code> and I’m nor sure, if there is a more elegant approach.</p>

Yimeng_Zhu: <p>Thanks for replying.</p>
<p>The last layer is indeed a nn.Linear. Concretely, I want to implement the RNN Transducer here.</p>

ndvbd: <p>If you’re looking for the total activiation size of a network, or activation/output per layer, you can use this function that I wrote:</p>
<pre><code class="lang-auto">
total_output_elements = 0
def calc_total_activation_size(model, call_the_network_function):
    global total_output_elements
    total_output_elements = 0

    def hook(module, input, output):
        global total_output_elements
        total_output_elements += output.numel()
        
    handle = torch.nn.modules.module.register_module_forward_hook(hook)
    result = call_the_network_function()
    handle.remove()
    return result, total_output_elements
</code></pre>
Backward propagation in many-to-one LSTM forward operation

Eugene_Alexeev: <p>Hello! I am learning about LSTM networks and now I am training in many-to-one approach. Here is how my forward function looks like:</p>
<pre><code class="lang-auto">def forward(self, x):

        out0 = None
        for i in range(x.shape[1]):
            inp = x[:, i, :]

            out0 = torch.relu(self.inp_fc(inp))
            out0 = self.lstm(out0)

        out = torch.tanh(self.dense(out0))

        return out
</code></pre>
<p>x is <code>[1, samples_count, features_count]</code>. So I am processing each sample one at a time sequentially, using <code>for</code> loop. Once forward functions is done, I am doing a backward propagation using a loss function.</p>
<p>So my question is - Does the fact that I do not store <code>out0</code> in for loop impact backward propagation in any way? I think I might’ve asked a stupid question, but I am still struggling to understand where exactly gradients are stored</p>
Launching with mpirun instead of torchrun

santurini: <p>I am trying to launch a DDP training on 2 GPUs that are located on 2 different nodes and interconnected via LAN. I would like to use mpirun instead of torchrun, what changes should I do?</p>
<p>I already changed the env variables like this:</p>
<pre><code class="lang-auto"># from mpirun
print("Launching with mpirun")
rank = int(os.environ["OMPI_COMM_WORLD_RANK"])
local_rank = int(os.environ["OMPI_COMM_WORLD_LOCAL_RANK"])
world_size = int(os.environ["OMPI_COMM_WORLD_SIZE"])

dist.init_process_group(backend="nccl", rank=rank, world_size=world_size)
</code></pre>
<p>I used to launch the training with no problems with the following command from both nodes just changing the rank from 0 to 1 in the slave:</p>
<pre><code class="lang-auto">torchrun --nproc_per_node=1 --nnodes=2 --node_rank=0 --master_addr=xxx.xxx.xxx.xxx --master_port=1234 train_torch.py --train_args
</code></pre>
<p>But when I try to launch from the master with mpirun like this:</p>
<pre><code class="lang-auto">export MASTER_ADDR=xxx.xxx.xxx.xxx
export MASTER_PORT=1234
mpirun -np 2 -host xxx.xxx.xxx.xxx,xxx.xxx.xxx.xxy --map-by node python train_torch.py
</code></pre>
<p>It just hangs on and never starts to parse the script.</p>

kumpera: <p>Knowing where it’s hanging will help figuring out the solution.</p>
<p>Try running with the env var <code>TORCH_DISTRIBUTED_DEBUG</code> set to INFO or DETAIL.</p>

santurini: <p>It gives no information as it gets stuck after printing “No Protocol specified”.<br>
It seems like it is waiting for the second node to do something but I don’t know how to solve, it should work by just launching on the master, and I’m facing the same exact problem when trying to use DeepSpeed</p>
Custom build on Android,build is ok,

cc5b4cb40e77805f1eec: <p>custom build from source about my model, build is ok. but using libpytorch_jni.so on Android device will failed.</p>
<p>First, I build libpytorch_jni.so  without custom build. Build is ok, and model can inference on android<br>
conmmand is :  BUILD_LITE_INTERPRETER=0 scripts/build_pytorch_android.sh arm64-v8a.</p>
<p>Second.  I build libpytorch_jni.so  with custom build. Build is ok, but model failed in  inferencing on android<br>
command is : SELECTED_OP_LIST=mymodel.yaml BUILD_LITE_INTERPRETER=0 scripts/build_pytorch_android.sh arm64-v8a</p>
<p>Error message :<br>
ld.lld: error: undefined symbol: torch::jit::Module::train(bool)<br>
ld.lld: error: undefined symbol: c10::GradMode::set_enabled(bool)<br>
ld.lld: error: undefined symbol: at::Tensor::to(c10::TensorOptions, bool, bool, c10::optionalc10::MemoryFormat) const<br>
ld.lld: error: undefined symbol: c10::UndefinedTensorImpl::_singleton<br>
ld.lld: error: undefined symbol: c10::IValue::reportToTensorTypeError() const</p>
torch.nn.Module.forward() contracts

gandolfin69420: <p>if i understand correctly, if you want your module to be able to evaluate batches as well as single samples, your forward() method must accommodate that by handling BOTH correctly, rather than only single samples, since torch can’t reasonably infer whether a batch or a single sample was passed based on input_size as the input_size often even goes unspecified.<br>
is this understanding correct?</p>
<p>(i came to this conclusion, as making a OneNormalizationLayer with</p>
<pre><code class="lang-py">def forward(s,x):
    return x / x.abs().sum()
</code></pre>
<p>yields normalization across all entries of all samples if called with a batch)</p>

eqy: <p>It depends on the domain, but usually there is a batch dimension ‘N’ to the data (I assume <code>x</code> in this case), and the convention for a single sample is to simply have a batch dimension of size 1. It’s generally less common that a model is expected to handle data that has differing dimensions depending on the batch size.</p>

gandolfin69420: <p>though the Modules in torch’s standard library, at least those where you specify a specific input size or just some quantity that relates to the input size, it seems to be capable to infer any positive number of batch dimensions. (so you can call them with a sample, or an additional n dimensions of multiple samples)</p>
How to input text for T5 summarization?

randusr: <p>I am playing around with summarization, following this <a href="https://pytorch.org/text/main/tutorials/t5_demo.html" rel="noopener nofollow ugc">tutorial</a>. Instead of using the cnndm dataset, I am just copying the text of a news article from the internet, so how should I format this string to be interpreted by the T5 model. This is what I have tried so far:</p>
<pre><code class="lang-auto">torch.set_default_tensor_type('torch.cuda.FloatTensor')
device = torch.device("cuda")
padding_idx = 0
eos_idx = 1
max_seq_len = 512
t5_sp_model_path = "https://download.pytorch.org/models/text/t5_tokenizer_base.model"

transform = T5Transform(
    sp_model_path=t5_sp_model_path,
    max_seq_len=max_seq_len,
    eos_idx=eos_idx,
    padding_idx=padding_idx,
)

t5_base = T5_BASE_GENERATION
transform = t5_base.transform()
transform = transform.to(device)
model = t5_base.get_model()
model.eval()
model = model.to(device)
sequence_generator = GenerationUtils(model)
sequence_generator.device = device

beam_size = 1

model_input = transform("summarize: " + text_of_article)
model_output = sequence_generator.generate(model_input, eos_idx=eos_idx, num_beams=beam_size)
output_text = transform.decode(model_output.tolist())
</code></pre>
<p>When I do this, I get the error:</p>
<pre><code class="lang-auto">AssertionError: For batched (3-D) `query`, expected `key` and `value` to be 3-D but found 2-D and 2-D tensors respectively
</code></pre>
<p>which I assume is because I haven’t inputted the data correctly? The tutorial mentions that the T5 model requires data to be batched, but I don’t see how my input is any different than what is in the tutorial:</p>
<pre><code class="lang-auto">batch = next(iter(cnndm_dataloader))
input_text = batch["article"]
target = batch["abstract"]
beam_size = 1

model_input = transform(input_text)
</code></pre>
<p>It looks like they are just getting the next article (prefixed with the task) from the dataloader. I am new to pytorch and nlp in general, so any help would be greatly appreciated!</p>

randusr: <p>For others who find this, transform() expects a list. Solved by doing:</p>
<pre><code class="lang-auto">model_input = transform(["summarize: " + text_of_article])
</code></pre>
Have two generators produce the same indices

Surya_Narayanan: <p>How can I create two data loaders that sample from two datasets in the same order?</p>
<p>That is, I want to create <code>Dataloader1</code> that samples from <code>dataset1</code> in some order (returns indices 1, 3, 4, 6, 7 for example) and <code>Dataloader2</code> that samples from <code>dataset2</code> in the same order.</p>
<p>Please let me know if I can add more clarity.</p>

nivek: <p>If you pass in a deterministic <code>sampler</code> into both <code>DataLoader</code>, (or make the two samplers share the same initial seed), the output order should match between the two.</p>
How to copy weights from torch.nn.sequential weights from source class to target class

Htut_Lynn_Aung: <p>Hi,</p>
<p>I’ve been trying to convert torch model to TensorRT model using <code>torch_tensorrt</code> tool. The following code is used to convert model to TensorRT model:</p>
<pre><code class="lang-auto">trt_model = torch_tensorrt.compile(torch_model,
                                           inputs = [
                                                    torch_tensorrt.Input((2048, 1, 32, 32),
                                                    dtype=torch.float32)
                                                    ],
                                            enabled_precisions = precision,
                                            workspace_size = 1 &lt;&lt; 33
                                          )
</code></pre>
<p>However, in <code>forward</code> function of the model uses a global variable and it is causing problem in TensorRT conversion. Here is the error prompt:</p>
<pre><code class="lang-auto">python value of type 'float' cannot be used as a value. Perhaps it is a closed over global variable? If so, please consider passing it in as an argument or use a local varible instead.:
  File "/codebase/Net/model.py", line 65
    def forward(self, patch):
        descr = self.desc_norm(self.layers(patch) + eps_l2_norm)
                                                    ~~~~~~~~~~~ &lt;--- HERE
        descr = descr.view(descr.size(0), -1)
        return descr
</code></pre>
<p>My initial solution is to add <code>eps_12_norm</code> as class attribute in network class. I tried creating a wrapper <code>torch.nn.module</code> class which loads weights from original model and included <code>eps_12_norm</code> new attribute and saved it as a new model but error still remains the same in TensorRT model conversion. I tried loading the original weights into original class which has new class attribute <code>eps_12_norm</code>, I cannot load the pretrained weights either.</p>
<p>My other solution is that I have to copy the weights from old model to new model with new class attribute since the only change here is new attribute addition. The model has two <code>torch.nn.sequential</code> module, I used the following code to copy the weights from old model to new model:</p>
<pre><code class="lang-auto"># copy weights from old model layers module to new model layers module
source_layers = old_model.layers
target_layers = new_model.layers
for source_param, target_param in zip(source_layers.parameters(), target_layers.parameters()):
    target_param.data.copy_(source_param.data)

# copy weights from old model desc_norm module to new model desc_norm module
source_desc_norm = old_model.desc_norm
target_desc_norm = new_model.desc_norm

for source_param, target_param in zip(source_desc_norm.parameters(), target_desc_norm.parameters()):
    target_param.data.copy_(source_param.data)
</code></pre>
<p>However, the inference results from two models over same input data is not the same. How can I solve this situation?</p>
Could use some clarification on jacrev() etc

shubbey: <p>Consider the example below.  I’m trying to understand why the two calls aren’t identical (the second call returns a zero tensor instead of the proper gradients).  I feel like I’m missing something very fundamental here.  Thanks!</p>
<pre><code class="lang-auto">import torch
from torch.func import jacrev

def comp(x,y):
    return x*x*y;

def comp2(x,y):
    return torch.sum(comp(x,y))

def comp3(x,y):
    return torch.sum(y)

R = torch.tensor([1.0,2.0,3.0],requires_grad=True)
z = torch.tensor([5.0])

print(jacrev(comp2)(R,z))
print(jacrev(comp3)(R,comp(R,z)))
</code></pre>

shubbey: <p>Just FYI, I was in fact missing something fundamental.  d/dx f(y) will always be 0 even if y is computed as a function of x (since y is the output, not a function).  My intention here was to efficiently compute the hessian for a known jacobian output (aka the gradient).  Currently I am just calling torch.autograd() on each row of the jacobian and thought I could leverage jacrev() to do it faster.</p>
LSTM ‘tuple’ object has no attribute ‘size’

Purna_Mallepaddi: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/lstm-tuple-object-has-no-attribute-size/100397">LSTM 'tuple' object has no attribute 'size'</a>:</p>
<p>Hey thanks for posting this solution I also tried with the <code>torchsummary</code> from <code>torchinfo</code>. But still continued getting the error regarding the <code>AttributeError: 'tuple' object has no attribute 'size'</code>. Complete code of the model is here.<br>
import torch.nn as nn</p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(LSTMModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm1 = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, dropout=0.2)
        self.bn1 = nn.BatchNorm1d(hidden_size)
#         self.lstm2 = nn.LSTM(hidden_size, 32, batch_first=True, dropout=0.2)
        self.fc = nn.Linear(hidden_size, output_size)
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        
        # Initialize hidden state with zeros
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device=x.device)
        
        # Initialize cell state with zeros
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device=x.device)
        
        #Xaviers initialization
        torch.nn.init.xavier_normal_(h0)
        torch.nn.init.xavier_normal_(c0)
        
        # Forward propagate LSTM1
        out, (h_final, c_final) = self.lstm1(x, (h0, c0))
#         out = self.bn1(out)
        
        # Forward propagate LSTM2
#         out, _ = self.lstm2(out)
        
        # Decode the hidden state of the last time step
        out = self.fc(out[:, -1, :])
        out = self.softmax(out)
        return out
</code></pre>
Multiprocessing with custom class and tensors on GPU

Andrew_Edmondson: <p>I am writing a DDPG reinforcement learning agent where I will offload the training to another process. To enable the worker to place experiences in the replay buffer and have the optimiser use those experiences for training I am looking to share the ReplayBuffer object. In Python multiprocessing, this is done using <code>class MyManager(BaseManager)</code> and I’m using it as follows:</p>
<pre><code class="lang-auto">MyManager.register("ReplayBuffer", ReplayBuffer)
myManager = MyManager()
myManager.start()
replay_proxy = myManager.ReplayBuffer(size=100_000)
</code></pre>
<p>After which I can <code>replay_proxy.add(....</code> However, while this passes numpy arrays and whatever else totally fine, it won’t pass a tensor. torch.multiprocessing extends the standard multiprocessing, but it doesn’t include BaseManager so I’m not sure how to get around this.</p>
<p>If anyone has a clever trick I’d appreciate it because I hate doing the whole ‘move to CPU and convert to numpy’ to write the tensor to the replay buffer only to convert it back to a tensor and move it to GPU as soon as I need it.</p>
Libtorch is not available for low version _GLIBC

scott-5: <p>I wanna compile a program required libtorch, so download libtorch (cxx11 ABI for cpu on linux) provided by <a href="https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.12.1%2Bcpu.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.12.1%2Bcpu.zip</a>, but it turns:<br>
…<br>
<em>libtorch/lib/libtorch_cpu.so: undefined reference to ‘powf@GLIBC_2.27’</em><br>
<em>libtorch/lib/libtorch_cpu.so: undefined reference to ‘expf@GLIBC_2.27’</em><br>
…</p>
<p>it seems this libtorch library requires GLIBC version &gt;= 2.27 (my version GLIBC is 2.17)<br>
so, how can I get libtorch for lower version GLIBC except from source?<br>
Could pytorch officially provide libtorch for GLIBC 2.17 ?</p>

fiona: <p>Hi，Have you solved this? I met this problem too. And my glibc version is 2.27</p>
Multiply a tensor of 2D matrices with a (batched) tensor of vectors

grupot3: <p>Given these tensors:</p>
<pre><code class="lang-auto">batch_size = 5
k=3
F = torch.randn(k+1, 3, 3)
B = torch.randn(batch_size, k, 1)
</code></pre>
<p>I’m trying to perform the following operation but without using the for loop (with is pretty slow when A and B are huge tensors):</p>
<pre><code class="lang-auto">S=F[0,:,:]*B[:,0:1,0:1]
for i in range(1,F.shape[0]-1):
	S += F[i,:,:]*B[:,i:(i+1),0:1]
</code></pre>
<p>Is it possible to do this without using the for loop? I am aware of the broadcasting capabilities of Pytorch, but I am not sure how I could apply it to this example.</p>
<p>Thanks!</p>

grupot3: <p>I just solved it using <code>einsum</code> (see tutorial <a href="https://rockt.github.io/2018/04/30/einsum" rel="noopener nofollow ugc">here</a>), it is way faster than the loop</p>
<pre><code class="lang-auto">S=torch.einsum('ajk,ial-&gt;ijk', [F[0:-1,:,:], B])
</code></pre>
Pytorch cannot detect cuda in script

richardatpytorch: <p>I am facing a situation when I try <code>torch.cuda.is_available()</code> on the terminal, the result is True but then when I run the same in the script the result is False, leading to the error <code>No cuda devices is available</code>. Have anyone faced the same situation before?</p>

ptrblck: <p>Could you describe what “in the terminal” and “in the script” exactly means? Typically these issues are caused by different virtual environments being selected e.g. in Jupyter vs. your terminal so something similar could also be the case here.</p>

richardatpytorch: <p>Sorry for the unclear context. <code>In the script</code> means the python code written on file could not detect cuda device, however, if I try the command <code>python -c "import torch; print(torch.cuda.is_available())"</code> then I got the <code>True</code> return as you can see on this screenshot<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/914c564d2aab9a286138ed5c10b20eae3adddc98.png" data-download-href="https://discuss.pytorch.org/uploads/default/914c564d2aab9a286138ed5c10b20eae3adddc98" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914c564d2aab9a286138ed5c10b20eae3adddc98_2_690x38.png" alt="image" data-base62-sha1="kJmMEv2TGGk0aXxHviuI6DSmsbK" width="690" height="38" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914c564d2aab9a286138ed5c10b20eae3adddc98_2_690x38.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914c564d2aab9a286138ed5c10b20eae3adddc98_2_1035x57.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914c564d2aab9a286138ed5c10b20eae3adddc98_2_1380x76.png 2x" data-dominant-color="391831"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1666×92 26.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

richardatpytorch: <p>I have tried to reinstall my environment with python 3.9 and everything seems OK now. The error mentioned above is with python 3.11.</p>
How to make torchaudio select the customized compiled ffmpeg

xuehy: <p>I installed pytorch using miniconda by</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
</code></pre>
<p>Now I want to use the <a href="https://pytorch.org/audio/stable/build.ffmpeg.html" rel="noopener nofollow ugc">GPU video decoder/encoder</a> of torchaudio.</p>
<p>I compiled and installed a ffmpeg with cuda support following the guide in <a href="https://pytorch.org/audio/stable/build.ffmpeg.html" rel="noopener nofollow ugc">Enabling GPU video decoder/encoder — Torchaudio 2.0.1 documentation (pytorch.org)</a></p>
<p>However, after installation, torchaudio cannot pick the correct ffmpeg outside miniconda since ffmpeg is installed by conda as a dependency while installing pytorch.</p>
<p>How can I fix this problem?</p>

ptrblck: <p>You might need to build <code>torchaudio</code> from source and let it pick your custom library during the build.</p>

xuehy: <p>Ended up with compiling and installing everything (including<code>ffmpeg</code> and <code>cuda</code>) from source into conda env. <code>torchaudio</code> does not need re-compilation and it picks the correct ffmpeg then.</p>