akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is it possible to Process Tensors of Varrying Size on GPU

username: Jaideep_Valani
cooked: <p>Can Pytorch data loader process Tensors of Varrying size on GPU,without padding ?</p>
<p>I think we will have collate issues .Only work around i got get is to process an image one by one on GPU  using parallel cpu workers for Loading the images.</p>

username: tom
cooked: <p>You can have your own collate function in the data loader that cats and returns sizes if you want. If that is preferable over padding/cropping is might be dubious.</p>

username: Jaideep_Valani
cooked: <p>Cat will throw exception .<br>
I have not made customization to collate before  .<br>
If you are familiar could you an example pls .</p>
How to mask tensor with boolean using c++ api? how to achieve this python code with c++ api?

username: hyman
cooked: <pre><code class="lang-python">arr_t = [
            [0,0,1,1,1,0,0],
            [0,0,1,0,1,0,0],
            [0,0,1,0,1,0,3],
            [0,0,1,0,1,0,3],
            [0,0,1,1,1,0,3]
        ]

mask = [
            [0,0,1,1,1,0,0],
            [0,0,1,0,1,0,0],
            [0,0,1,1,1,0,0],
            [0,0,1,0,1,0,0],
            [0,0,1,1,1,0,0]
        ]
t1 = torch.IntTensor(arr_t).to(torch.int8)
t2 = torch.IntTensor(mask).to(torch.int8)
t3 = t2 &gt;0
t1[t3] = t2[t3]   # when using c++ this line can't be success
print(t1)
</code></pre>
<pre><code class="lang-auto">tensor([[0, 0, 1, 1, 1, 0, 0],
        [0, 0, 1, 0, 1, 0, 0],
        [0, 0, 1, 1, 1, 0, 3],
        [0, 0, 1, 0, 1, 0, 3],
        [0, 0, 1, 1, 1, 0, 3]], dtype=torch.int8)
</code></pre>

username: hyman
cooked: <p>use masked_scatter function can do that</p>
Error in using apex

username: 887574002
cooked: <p>Hi, I am going to use Apex to train my model since I am working on 3D medical images and I am frequently going OOM. I read the documentation of Apex API and I wanted to try using apex on a simple model first to get better intuition and then try it for my main model. Below is my code, but I do not know why I get the error<br>
<code>CUDA error: an illegal memory access was encountered</code></p>
<p>I will be thankful if you can help me.</p>
<pre><code class="lang-auto">mnist = torchvision.datasets.MNIST('./data',train= True,download=True,transform =transforms.ToTensor())
data_loader = DataLoader(mnist,batch_size=20,num_workers=2,shuffle=True)
class Model(nn.Module):
    # Our model

    def __init__(self):
        super(Model, self).__init__()
        
        self.fc1 = nn.Conv2d(1,10,3)
        self.bn1 = nn.BatchNorm2d(10)
        self.fc2= nn.Conv2d(10,20,3)
        self.bn2 = nn.BatchNorm2d(20)
        self.fc3= nn.Linear(11520,10)
        
    def forward(self,x):
        x = F.relu(self.fc1(x))
        x = self.bn1(x)
        x = F.relu(self.fc2(x))
        x = self.bn2(x)
        x = x.view(x.size(0),-1)
        x = self.fc3(x)
        return(x)

device = torch.device('cuda:6' if torch.cuda.is_available() else 'cpu')
model = Model().to(device)
optimizer = optim.Adam(model.parameters(),lr=0.1)
lr_sch = lr_scheduler.StepLR(optimizer,step_size=2,gamma=0.1)
criterion = nn.CrossEntropyLoss()
model,optimizer = amp.initialize(model,optimizer,opt_level="O2",keep_batchnorm_fp32=True,loss_scale="dynamic")

def train(epoch):
    model.train()
    t_loss = 0
    for X,y in data_loader:   
        X= X.to(device)
        y = y.long().to(device)
        pred = model(X)     
        loss = criterion(pred,y)  
        t_loss+= loss.item()     
        optimizer.zero_grad()   
        with amp.scale_loss(loss, optimizer) as scaled_loss: 
            scaled_loss.backward()    
        optimizer.step()
        return(t_loss/len(data_loader.dataset))
num_epochs = 20
train_loss = []
cudnn.benchmark = True
for epoch in range(num_epochs):
    t_loss = train(epoch)
    train_loss.append(t_loss)
</code></pre>

username: ptrblck
cooked: <p>Thanks for reporting this issue!<br>
It seems that <code>multi_tensor_apply</code> is triggering an illegal memory access (and so far the first debugging seems to point to a launch of this kernel on the wrong device). We’ll look into it.</p>
<p>As a workaround, you could try to use <code>cuda:0</code> or (even better) the native automatic mixed precision implementation as described <a href="https://pytorch.org/docs/master/amp.html">here</a>.<br>
To use native amp you would have to install the nightly binary or build from master.</p>

username: 887574002
cooked: <p>Hi, thank you for your answer. The problem was solved when I set<br>
<code>torch.cuda.set_device(device)</code><br>
before constructing optimizer and model.<br>
That’s the point that should be taken into account.</p>
Validating C++ model number of elements in forward

username: FDS
cooked: <p>Hi,</p>
<p>If someone gives me (my C++ code) a model to run, I’d like to validate the input size it requires, and the number of elements in its output. I can get the number of output elements by loading the model and  <code>unsigned n_outputs = model.forward({test_input}).toTensor().numel()</code>.</p>
<p>However how to size the <code>test_input</code> tensor? Currently I’m just making one with the dimensions suggested by a config, but would like to query the model directly. I’m assuming there is something to be done with <code>model.get_method("forward")</code> as suggested by <a href="https://discuss.pytorch.org/t/how-to-retrieve-number-of-outputs-of-a-model/51264" class="inline-onebox">How to retrieve number of outputs of a model?</a> but instead of “num_inputs()” I’d like “number of elements required in input tensor 0”.</p>
<p>Thanks,<br>
FDS</p>

username: copythatpasta
cooked: <p>The issue is that pytorch does not require a models input layer to specify a dimension or size, only amounts of in features and out features (and kernel size maybe). You always have the option of looping over the modules within the script module to see these parameters, you just won’t see dimensions unless they are put in through customs modules. Models will not store this information unless explicitly made to do so</p>

username: FDS
cooked: <p>Thanks for the answer Charles. What I suspected. This makes it more difficult for a C++ program to validate a given TorchScript model, like you could in TensorFlow. But the fake-forward pass is better than nothing, at least allows early detection of mismatched dimensions.</p>
Working with half model and half input

username: 887574002
cooked: <p>Hi, I am trying to use the model.half() to get rid of out of memory error. So, first I want to check it for a simple model and then try it for a complex model.</p>
<p>In the following example, I am using <code>model.half()</code>for a simple CNN and MNIST dataset. I think I did all thing correctly, but I got <code>Nan</code> for loss.<br>
I am wondering if you kindly let me know where I am doing a mistake.</p>
<pre><code class="lang-auto">mnist = torchvision.datasets.MNIST('./data',train= True,download=True,transform =transforms.ToTensor())
data_loader = DataLoader(mnist,batch_size=20,num_workers=2,shuffle=True)

class Model(nn.Module):
    # Our model

    def __init__(self):
        super(Model, self).__init__()
        
        self.fc1 = nn.Conv2d(1,10,3)
        self.bn1 = nn.BatchNorm2d(10)
        self.fc2= nn.Conv2d(10,20,3)
        self.bn2 = nn.BatchNorm2d(20)
        self.fc3= nn.Linear(11520,10)
        
    def forward(self,x):
        
        x = F.relu(self.fc1(x))
        
        x = x.type(torch.float32)   
        x = self.bn1(x)
                
        x = x.type(torch.float16)
        
        x = F.relu(self.fc2(x))
        
        x = x.type(torch.float32)
        
        x = self.bn2(x)
        
        
        x = x.type(torch.float16)
        
        x = x.view(x.size(0),-1)
                
        x = self.fc3(x)
        return(x)

device = torch.device('cuda:6' if torch.cuda.is_available() else 'cpu')

model = Model().half().to(device)

for layer in model.modules():
    
    if isinstance(layer,nn.BatchNorm2d):
        
        layer.float()

optimizer = optim.Adam(model.parameters(),lr=0.1)

lr_sch = lr_scheduler.StepLR(optimizer,step_size=2,gamma=0.1)

criterion = nn.CrossEntropyLoss()

def train(epoch):
    model.train()
    t_loss = 0
    for X,y in data_loader:   
        X= X.half().to(device)
        y = y.long().to(device)
        pred = model(X)
        loss = criterion(pred,y)
        t_loss+= loss.item()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        return(t_loss/len(data_loader.dataset))  
num_epochs = 20
train_loss = []
for epoch in range(num_epochs):
    t_loss = train(epoch)
    train_loss.append(t_loss)
    print(f'Epoch: {epoch}: Loss: {t_loss:0.4}')

Epoch: 0: Loss: 4.346e-05
Epoch: 1: Loss: nan
Epoch: 2: Loss: nan
Epoch: 3: Loss: nan
Epoch: 4: Loss: nan
Epoch: 5: Loss: nan
Epoch: 6: Loss: nan
Epoch: 7: Loss: nan
Epoch: 8: Loss: nan
Epoch: 9: Loss: nan
Epoch: 10: Loss: nan
Epoch: 11: Loss: nan
Epoch: 12: Loss: nan
Epoch: 13: Loss: nan
Epoch: 14: Loss: nan
Epoch: 15: Loss: nan
Epoch: 16: Loss: nan
Epoch: 17: Loss: nan
Epoch: 18: Loss: nan
Epoch: 19: Loss: nan
</code></pre>

username: ptrblck
cooked: <p>Calling <code>model.half()</code> manually can easily yield NaN and Inf outputs, as some internal values can overflow.<br>
We recommend to use automatic mixed precision training as described <a href="https://pytorch.org/docs/master/amp.html">here</a>, which takes care of these issues for you.<br>
To use amp you would have to install the nightly binary or build from master.</p>
Help about VideoClips

username: sj_xu
cooked: <p>Hi,</p>
<p>I have a questions about how to use VideoClips. Torchvision uses VideoClips class to implement Kinetics dataset. However, it seems that this implementation can only support sampling 1 clip from 1 video at each time. My problem is, I want to sample 2 different clips from the same video each time. I have no idea how to use VideoClips class to achieve this goal.</p>
<p>Can anyone help me?</p>
cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

username: Bahar_Nikpour
cooked: <p>Hello everyone!<br>
I’m training a policy gradient algorithm with 3 BLSTM networks and using leave one out to test it on 60 data samples, but I get error "cuDNN error: CUDNN_STATUS_INTERNAL_ERROR"after passing 10 loops of for loop for leave one out. I neither can run it on google colab because of an unknown error (it crashes after hours). I don’t know whether its memory error or the problem with my code. It takes a lot of time on CPU so I cannot test it to see that my code is working properly but I have checked it several times and seems correct. Would you please give me an idea as I got stuck in my research now. I appreciate your help.</p>

username: ptrblck
cooked: <p>I would recommend to update to the latest PyTorch version, if you are using an older one.<br>
If you are already using the latest release, could you post an executable code snippet, which would recreate the issue and post your setup (GPU, CUDA version, cudnn version, PyTorch version)?</p>
<p>Also, as a workaround you could try to use <code>torch.backends.cudnn.benchmark=True</code> or disable cudnn via <code>torch.backends.cudnn.enabled=False</code>.</p>
How to properly save/load a mixed-precision trained model for CPU-based inference?

username: vgoklani
cooked: <p>Hey there,</p>
<p>I would like take advantage of mixed-precision to efficiently train a model, and then use my CPU for inference.</p>
<p>I’ve trained a model using apex (O2), and followed the instructions to save the checkpoint:</p>
<pre><code>checkpoint = {
	'model': model.state_dict(),
	'optimizer': optimizer.state_dict(),
	'amp': amp.state_dict()
}

torch.save(checkpoint, 'checkpoint.pt')
</code></pre>
<hr>
<p>Now I would like to load the trained model for CPU based inference…</p>
<p>I’ve defined a function that sums all the tensors from the state_dict, presumably this sum should be the same for both the saved and loaded model:</p>
<pre><code>def calculate_checksum(model):
	return sum(model.state_dict()[key].sum().item() for key in model.state_dict().keys())

device = torch.device('cpu')
checkpoint = torch.load("checkpoint.pt", map_location=device)
</code></pre>
<p>Now, it turns out that the checksum values don’t match unless I initialize amp, which <em>requires</em> both the optimizer to be loaded and the model to be <em>moved</em> to the gpu.</p>
<p>model, optimizer = apex.amp.initialize(model, optimizer, opt_level=“O2”)<br>
apex.amp.load_state_dict(best_checkpoint[‘amp’])</p>
<p>How do i save the trained model so that I can properly load it onto the CPU for inference? I shouldn’t need to initialize amp or load the optimizer. Thanks!</p>

username: ptrblck
cooked: <p>Summing the model parameters and the parameters stored in the <code>state_dict</code> might yield a different result, since <code>opt_level='O2'</code> uses FP16 parameters inside the model, while the <code>state_dict</code> will contain upcasted FP32 parameters.</p>
<p>That being said, we recommend to use the native mixed precision implementation by installing the nightly binaries or build from master. The documentation can be found <a href="https://pytorch.org/docs/master/amp.html">here</a>.</p>
Is it possible to create linear neural layers using only toepltiz matrices?

username: PhysicsHipster
cooked: <p>I’m trying to create a simple 10-layer neural network to classify MNIST. I am constrained, however, because I need the weight matrices to be toeplitz matrices (each diagonal in the matrix must be constant). Is there a way to enforce this condition using the Linear class? If not, how might I go about making a new subclass of nn.module that would allow me to model this?</p>
<p>I’m new to Pytorch, so apologizes if this is a silly question!</p>

username: ptrblck
cooked: <p>If I understand your use case correctly, you would like to use a Toeplitz matrix as the <code>weight</code> parameter in all linear layers and would like to keep the constraint during the training?<br>
Would each layer use a square matrix via <code>in_features=out_features</code> so that the trainable parameters would be <code>n+1</code> or any shape?</p>
<p>I think one valid approach could be to initialize the weight as an <code>nn.Parameter</code> in your custom linear layer implementation, create the Toeplitz matrix during the <code>forward</code> pass, and apply the operation using the functional API via <code>F.linear(input, weight)</code>.</p>
Implement Convpoint with dataset

username: shangguan91
cooked: <p>from torch_geometric.datasets import ShapeNet</p>
<p>dataset = ShapeNet(root=‘/tmp/ShapeNet’, categories=[‘Airplane’])</p>
<p>dataset[0]</p>
<blockquote>
<blockquote>
<blockquote>
<p>Data(pos=[2518, 3], y=[2518])</p>
</blockquote>
</blockquote>
</blockquote>
<p>Dear all, i am handling data of point cloud,  i want to implement ConvPoint <a>:https://github.com/aboulch/ConvPoint/blob/master/networks/network_classif.py</a></p>
<p>However, there is only benchmark pretrained model. what is the best way to modify those model and implement network on your owndataset</p>

username: ptrblck
cooked: <p>You could copy-paste the linked model and modify it as you wish.<br>
Alternatively you could also use this model as the base class and derive your custom model from it, but it depends on your coding preference I guess.</p>
How to use torch.jit for graph trace?

username: SerB
cooked: <p>Hi all,</p>
<p>I try to trace myself graph using torch.jit but i have next problem:</p>
<ol>
<li>Create simple model</li>
</ol>
<blockquote>
<pre><code>class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=1)
    def forward(self, x):
        x = self.conv1(x)
        return x
</code></pre>
</blockquote>
<ol start="2">
<li>Create variable and trace model:</li>
</ol>
<blockquote>
<pre><code>net = SimpleNet()
var = Variable(torch.randn(1, 3, 224, 224))
trace, out = jit.trace(net, var)
trace.export(list(net.state_dict().values()))
</code></pre>
</blockquote>
<p>And I get next output:</p>
<blockquote>
<pre><code>Traceback (most recent call last):
  File "run.py", line 17, in &lt;module&gt;
    trace.export(list(net.state_dict().values()))
RuntimeError: ONNX export failed: Couldn't export C++ operator ConvForward

Graph we tried to export:
graph(%1 : Float(1, 3, 224, 224)
      %2 : Float(10, 3, 1, 1)
      %3 : Float(10)) {
  %5 : Float(1, 10, 224, 224), %6 : Handle = CppOp[ConvForward](%1, %2, %3), uses = [[%0.i0], []];
  return (%5);
}
</code></pre>
</blockquote>
<p>And it’s behavior has reproduced for any networks.<br>
How can I fix it? Thanks in advance!</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>In PyTorch v1.5.1, you can use <code>trace.code</code>. Here is an example:</p>
<aside class="quote quote-modified" data-post="2" data-topic="88453">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-i-know-what-funcitonal-layers-e-g-f-relu-are-used/88453/2">How can I know what funcitonal layers (e.g. F.relu) are used?</a> 
  </div>
  <blockquote>
    Hi, 
As you might know that PyTorch creates graphs dynamically when you provide data to network, you can get it’s behavior by applying an input and tracing graph construction. Here is the code: 
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28*28, 300)
        self.fc2 = nn.Linear(300, 100)
        self.fc3 = nn.Linear(100, 10)

    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.re…
  </blockquote>
</aside>
How can I know what funcitonal layers (e.g. F.relu) are used?

username: ShengweiAn
cooked: <p>Take <code>nn.functional.relu</code> as an example.</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(28*28, 300)
        self.fc2 = nn.Linear(300, 100)
        self.fc3 = nn.Linear(100, 10)

    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.log_softmax(x, dim=1)
</code></pre>
<p>If I use <code>nn.ReLU</code>, then I can get a ReLU layer from the <code>named_modules()</code>. But if I use <code>nn.functional.relu</code>, how can I know it’s used. Similarly, for other functional layers (dropout/sigmoid/…)</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>As you might know that PyTorch creates graphs dynamically when you provide data to network, you can get it’s behavior by applying an input and tracing graph construction. Here is the code:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28*28, 300)
        self.fc2 = nn.Linear(300, 100)
        self.fc3 = nn.Linear(100, 10)

    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.log_softmax(x, dim=1)

net = Net()
x = torch.randn(1, 28*28)
traced_net = torch.jit.trace(net, x)
print(traced_net)
print(traced_net.code)
### prints

#Net(
#  original_name=Net
#  (fc1): Linear(original_name=Linear)
#  (fc2): Linear(original_name=Linear)
#  (fc3): Linear(original_name=Linear)
#)
#def forward(self,
#    x: Tensor) -&gt; Tensor:
#  _0 = self.fc3
#  _1 = self.fc2
#  _2 = self.fc1
#  input = torch.view(x, [-1, 784])
#  input0 = torch.relu((_2).forward(input, ))
#  input1 = torch.relu((_1).forward(input0, ))
#  _3 = torch.log_softmax((_0).forward(input1, ), 1, None)
#  return _3
</code></pre>
<p>Please read this tutorial about <a href="https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html" rel="nofollow noopener">TorchScript</a>.</p>
<p>Bests</p>
How to get GPU memory usage in pytorch code?

username: Naruto-Sasuke
cooked: <p>Is there any way to see the gpu memory usage in pytorch code?</p>

username: alwynmathew
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/alwynmathew/nvidia-smi-python" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/26373373?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/alwynmathew/nvidia-smi-python" target="_blank" rel="nofollow noopener">alwynmathew/nvidia-smi-python</a></h3>

<p>Pytorch code to get GPU stats. Contribute to alwynmathew/nvidia-smi-python development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: albanD
cooked: <p>You also have all the functions to get the memory allocated and the memory actually used by tensors. See the <a href="https://pytorch.org/docs/stable/cuda.html#memory-management">doc here</a> for more details.</p>
How can I include loss of derivatives?

username: yifanyang96
cooked: <p>Hi!</p>
<p>I’m working on a project on molecule energies. The loss function consists of two parts: the energy term and the force term and forces are derivatives wrt the input coordinates. This is part of my code:</p>
<pre><code class="lang-auto">def energies_forces_trainer(container, optimizer, loss_fn):
    def train_and_store_loss(engine, batch):
        model.train()
        optimizer.zero_grad()
        inputs, targets = batch
        final = {'energies': [], 'forces': []}
        for s, c in inputs:
            results = model((s, c)) # c stands for coordinates
            _, energies = energy_shifter(results) # Some transformation
            forces = -torch.autograd.grad(energies.sum(), c)[0] # I want to compute the result's derivatives wrt the input
            final['energies'].append(results[1])
            final['forces'].append(forces)
        final['energies'] = torch.cat(final['energies'])
        final['forces'] = torch.cat(final['forces'])
        loss = loss_fn('energies')(targets, final) + loss_fn('forces')(targets, final) # Energy loss and force loss
        loss.backward() # Error ocurrs
        optimizer.step()
        return loss.item()
    return ignite.engine.Engine(train_and_store_loss)
</code></pre>
<p>This code produces ‘Trying to backward through the graph a second time’ error. So does <code>torch.autograd.grad</code> accumulate grads? How can I include loss of derivatives?</p>
<p>Thank you!</p>

username: Leslie
cooked: <p>try:<br>
forces = -torch.autograd.grad(energies.sum(), c, retain_graph=True, create_graph=True)[0]</p>
How to use the AOT bundle in pytorch?

username: tazz
cooked: <p>Hi, I was trying to use the bundle that is generated using GLOW. In the instructions it mentions that I need to link the executable to my application. How can I link that executable to pytorch file. I wish to run some tests for pytorch execution vs glow AOT bundle execution. <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md" rel="nofollow noopener">https://github.com/pytorch/glow/blob/master/docs/AOT.md</a></p>

username: jfix
cooked: <p>If you want to run it driven via PyTorch, you’d need to still create a C++ wrapper (see <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md#how-to-use-the-bundle" rel="nofollow noopener">here</a> for how to use the bundle that’s generated) that PyTorch has a bridge to execute on.</p>
<p>But to take a step back, it might make sense to just run via <a href="https://github.com/pytorch/glow/blob/master/docs/pytorch.md" rel="nofollow noopener"><code>torch_glow</code></a> and without a bundle – not sure if there’s a specific reason you wanted to use a bundle here.</p>

username: tazz
cooked: <p><a class="mention" href="/u/jfix">@jfix</a>…the whole idea was to run inference in pytorch and compare it to the AOT bundle that is generated using glow. I assume that that pytorch has been focusing on inference on cpu and multi-core whereas  glow is focused more on hardware accelerators. I did have a question if I had to compare execution of both on cuda gpu what would be a good way to get some results?</p>

username: jfix
cooked: <aside class="quote no-group" data-username="tazz" data-post="3" data-topic="88399">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/8491ac/48.png" class="avatar"> tazz:</div>
<blockquote>
<p>the whole idea was to run inference in pytorch and compare it to the AOT bundle that is generated using glow.</p>
</blockquote>
</aside>
<p>Sure, but you don’t need to use an AOT bundle to do so. If you go through the torch_glow path, nothing is AOT, it’s done compiled just-in-time.</p>
<aside class="quote no-group" data-username="tazz" data-post="3" data-topic="88399">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/8491ac/48.png" class="avatar"> tazz:</div>
<blockquote>
<p>I did have a question if I had to compare execution of both on cuda gpu what would be a good way to get some results?</p>
</blockquote>
</aside>
<p>We don’t have support for CUDA right now directly. Theoretically you could try to target CUDA via our LLVM backend, but it’s been much more targeted toward CPUs, and I don’t know of anyone who has tried this. We also have an OpenCL backend, but it’s not a big focus of ours.</p>
<p>In open source we have focused more on inference accelerators backends like NNPI and Habana. There are also open-source users that contribute PRs usually targeted for LLVM-based backends, but those backends are all kept private by those contributors.</p>
Constrain sum of weights to maximum value during training

username: seand
cooked: <p>Hi all,</p>
<p>Hoping to get some help with a tricky optimization problem. I’m using PyTorch to minimize a function f subject to certain constraints. The constraints are that the coefficients of the function all must be &gt;= 0, and the sum of coefficients must be &lt;= some value W. With projected gradient descent, it’s easy enough to handle the first constraint, and indeed, it works as expected.</p>
<p>However, I’m having trouble coming up with a way to enforce the constraint that the sum of coefficients must be &lt;= W. One thought was to essentially apply a large penalty whenever W - sum(coefficients) &gt; 0. However, I’m not sure whether this is an “appropriate” way to solve this issue and what penalty function to apply. Ideally it’d be something that is 0 when x &lt;= 0 and some exponentially increasing value when x &gt; 0.</p>
<p>Without the 2nd constraint, my code is:</p>
<pre><code class="lang-auto">    opt = optim.SGD([x], lr=1)
    for e in range(max_iter):
        loss = f(x, *args)
        opt.zero_grad()
        loss.backward()
        opt.step()
        with torch.no_grad():
            x.clamp_(0, 1e10)
    return x
</code></pre>
<p>Any guidance or tips would be much appreciated!</p>
Starting with pytoch - check installation

username: rozo
cooked: <p>Hello,<br>
Sorry for the noby question. I installed torch 1.1 on my windows machine.<br>
I am not sure if it is ok since “torce.cuda.is_available()” return true but all examples I try seems to not run on the gpu.</p>
<p>Is there a simple code (A simple torch loop for example) I can run just to see that the gpu is loaded so I can verify that torch can really run on gpu?</p>
<p>Thanks in advanced.</p>

username: peterjc123
cooked: <p>torch.cuda.FloatTensor([1.])</p>

username: peterjc123
cooked: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/mnist/main.py</a></h4>
<pre><code class="lang-py">from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.optim.lr_scheduler import StepLR


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.dropout1 = nn.Dropout2d(0.25)
        self.dropout2 = nn.Dropout2d(0.5)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to use TorchScript with Dropout for training?

username: fermat97
cooked: <p>My model is:</p>
<pre><code class="lang-auto">import torch.nn as nn

class Net(nn.Module):
    def __init__(self, num_input, num_hidden, num_classes, dropout,
                 activation='tanh'):
        super(Net, self).__init__()
        self.dropout = nn.Dropout(dropout)
        self.fc1 = nn.Linear(num_input, num_hidden)
        self.fc2 = nn.Linear(num_hidden, num_classes)

        if activation == 'tanh':
            self.activation_f = torch.tanh
        elif activation == 'relu':
            self.activation_f = torch.relu

    def forward(self, x):
        x = self.activation_f(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x
</code></pre>
<p>I call my model for instance as:</p>
<pre><code class="lang-auto">model = Net(14,512,2,0.2).to(device)
</code></pre>
<p>However once I use <code>TorchScript</code> as:</p>
<pre><code class="lang-auto">traced_model = torch.jit.trace(model, torch.zeros([1, 14], dtype=torch.float))
</code></pre>
<p>I receive the following error:</p>
<pre><code class="lang-auto">IndexError: The shape of the mask [2] at index 0 does not match the shape of the indexed tensor [1, 2] at index 0
</code></pre>
<p>I know that if I use <code>model.eval()</code> I don’t receive any error BUT I want to use my model for training and not evaluation. Does anybody know any solution or workaround for such problem?</p>
<p>PS: I am using <code>PyTorch</code> version <code>1.4</code>.</p>

username: SplitInfinity
cooked: <p>I am unable to reproduce this error with <code>Python 3.8.3</code> and <code>pytorch-1.4.0</code> from <code>conda</code>.</p>
<p>Based on your error message, it seems like your input shape might be the problem. Try using an input with shape <code>[14]</code> as the input during tracing:</p>
<pre><code class="lang-auto">traced_model = torch.jit.trace(model, torch.zeros([14], dtype=torch.float))
</code></pre>
<p>Stepping back a bit, I would advise you to use scripting (<code>torch.jit.script</code>) rather than tracing for this use case. The reason is that control flow is not visible to tracing and from what I remember, the backward pass for dropout predicates whether or not the incoming gradient should be backpropagated based on whether the corresponding input was passed through during the forward pass.</p>

username: fermat97
cooked: <p>Hi, but it is not possible to take input of size <code>[14]</code>, where is the batch dimension (<code>[1,14]</code>)? I don’t think the dimension is the problem since once I fix the dropout rate to <code>0.0</code> then there is no error message and it works just fine.</p>

username: SplitInfinity
cooked: <p>I still cannot reproduce your error. Can you provide more details on your setup? There is an environment details collection script in the PyTorch repository.</p>
Differentiability of embedding layer

username: tiwa
cooked: <p>I have a model that produces logits that I sample from. When I sample, I get a sequence of one-hots.</p>
<p>Part of my loss function involves passing this sequence into an RNN which will output a scalar. This RNN takes a batch of lists of indices, which get packed (they’re of variable length) and sent through the network, which has an embedding layer.</p>
<p>If the output of my RNN is part of my loss, does my batch of lists of indices (the input to the RNN) need to be differentiable? If so, I think it needs to be of type float to store gradients, but if I do this I get the error <code>RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)</code>.</p>
<p>Am I correct in thinking that the need to have integer embedding indices poses a differentiability problem?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes. The “indexing” operation is not differentiable wrt to the indices. Mainly because they are not contiguous. And if you added a “floor” op to make it contiguous, you would get a gradient of 0 almost everywhere.</p>

username: tiwa
cooked: <p>Does that mean that in general it’s impossible to modularize models that employ embeddings?</p>

username: albanD
cooked: <p>Not sure what you mean by “modularize” here.<br>
But the indices in the embedding cannot be differentiable no.</p>

username: tiwa
cooked: <p>Here’s what I mean: my Model is supposed to take any PyTorch model as an argument and optimize against its outputs. If the model passed in is (say) a simple feedforward neural network (e.g. MLP), then the Model’s own weights can change in response to loss incurred by the MLP’s outputs by backpropagating the error through the MLP’s own layers/weights (and then through the Model’s own weights).</p>
<p>In order for the Model to actually learn, the function/model passed in needs to be differentiable so the Model’s weights can receive the gradients. As long as this criterion is satisfied, it seems to me that a model is “modular” in the sense that it can just be passed in and everything will work fine.</p>
<p>But it also seems to me that if the passed-in model makes use of embedding layers, it will impossible to determine loss with respect to the input, which means the Model won’t be able to change its weights in a way that will make the passed-in model reduce its incurred loss. Does that make sense?</p>

username: albanD
cooked: <p>Yes that’s right.<br>
Basically, the Embedding layer cannot be plugged in in a middle of any system as it is not differentiable wrt to its inputs. It has to be the first layer.</p>
How to calculate size of receptive field?

username: Alex_I
cooked: <p>I’d like to calculate the size of the receptive field in various architectures (EfficientNet, ResNext, …) by looking at a PyTorch model and examining the layers.  Is there an example of code that already does this?  If not, how can this best be done?</p>
<p>(It is relatively easy to look at a simple sequential model.  I think basically going from the top down, any time there is a Conv2d, the field expands by kernel size-1, and any time there is a strided conv or pool operation the field is multiplied by the stride.  Tracing the size of the field becomes a bit more complicated with more complex connectivity though…)</p>
PyTorch hangs in thread after large torch.zeros call in main process

username: kevinyang
cooked: <p>Hi, first time posting, apologies if I made a mistake in the categorization or anything.</p>
<p>I am trying to have a generator load objects in the background, and I am encountering an extremely strange bug which I have distilled down to the following example. When I try to run the following code, it hangs when trying to call <code>torch.zeros</code> in <code>split_loader_creator</code>, but if I remove the seemingly irrelevant line <code>torch.zeros(152*4, 168*4).float()</code> near the end, it seemingly can make progress. It also seems fine if I change 152<em>4 and 168</em>4 to much smaller numbers. This is on PyTorch 1.5.1, and I do not encounter the issue on 1.4.0. Am I somehow doing this multiprocessing incorrectly? Would really appreciate any help, and please let me know if I can provide more information that might be helpful.</p>
<pre><code class="lang-auto">import torch
import multiprocessing
import atexit

def split_loader_creator():
    for i in range(20):
        yield torch.zeros(10, 170, 70)

def background_generator_helper(gen_creator):
    def _bg_gen(gen_creator, conn):
        gen = gen_creator()
        while conn.recv():
            try:
                conn.send(next(gen))
            except StopIteration:
                conn.send(StopIteration)
                return
            except Exception:
                import traceback
                traceback.print_exc()

    parent_conn, child_conn = multiprocessing.Pipe()
    p = multiprocessing.Process(target=_bg_gen, args=(gen_creator, child_conn))
    p.start()
    atexit.register(p.terminate)

    parent_conn.send(True)
    while True:
        parent_conn.send(True)
        x = parent_conn.recv()
        if x is StopIteration:
            return
        else:
            yield x

def background_generator(gen_creator): # get several processes in the background fetching batches in parallel to keep up with gpu
    generator = background_generator_helper(gen_creator)
    while True:
        batch = next(generator)
        if batch is StopIteration:
            return
        yield batch

torch.zeros(152*4, 168*4).float()

data_loader = background_generator(split_loader_creator)
for i, batch in enumerate(data_loader):
    print(i)
</code></pre>

username: mrshenli
cooked: <aside class="quote no-group" data-username="kevinyang" data-post="1" data-topic="88484">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kevinyang/48/26473_2.png" class="avatar"> kevinyang:</div>
<blockquote>
<p>This is on PyTorch 1.5.1, and I do not encounter the issue on 1.4.0.</p>
</blockquote>
</aside>
<p>This sounds like a regression. And I confirm I can reproduce the reported behavior.</p>
<p>Could you please submit an issue to <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> to report this bug?</p>

username: kevinyang
cooked: <p>Thanks, I’ll give that a try.</p>
Tensorflow loss function

username: Mohamed_Nabih
cooked: <p>I have a dumpy question I have a loss function that depends on Tensorflow and Keras backend, the question is can I use this loss function to train my PyTorch network.</p>
<p>Another question is there is any PyTorch function that equivalent to this TensorFlow command.<br>
stft_true = tf.contrib.signal.stft(y_true,256,128,512,window_fn,pad_end=False)</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hey,</p>
<p>Have you tried <a href="https://pytorch.org/docs/master/generated/torch.stft.html#torch.stft"><code>torch.stft</code></a> that looks like the same thing.</p>

username: Mohamed_Nabih
cooked: <p>Yes,  <a class="mention" href="/u/alband">@albanD</a><br>
It seems the same function.</p>
How to improve inference time of pruned model using torch.nn.utils.prune?

username: DaVenBi
cooked: <p>Hi ,</p>
<p>I have used torch.nn.utils.prune to prune a pretrained model using below code.</p>
<pre><code> for name, module in model.named_modules():
     # prune 20% of connections in all 2D-conv layers
     if isinstance(module, torch.nn.Conv2d):
         prune.l1_unstructured(module, name='weight', amount=0.3)
         prune.remove(module, 'weight')
     # prune 40% of connections in all linear layers
     elif isinstance(module, torch.nn.Linear):
         prune.l1_unstructured(module, name='weight', amount=0.4)
         prune.remove(module, 'weight')
</code></pre>
<p>I have saved the pruned model. This pruned model’s size is halved and accuracy took a hit but did not improve its inference time. Please let me know how to improve the inference time of pruned model</p>
<p>Thanks in advance.</p>

username: Raghav_Gurbaxani
cooked: <p><a class="mention" href="/u/davenbi">@DaVenBi</a> can you tell me how you saved the model ?</p>

username: ypsoh
cooked: <p>Based on sparsity being applied in an <code>unstructured</code> manner, I don’t think there is an out-of-the-box solution that speeds up inference based on the fact that the matrices are more sparse. I suggest that you try pruning in a structured way and see how that changes inference speed.</p>

username: ani0075
cooked: <p>As <a class="mention" href="/u/ypsoh">@ypsoh</a> mentioned, the torch.sparse module is still in beta. So you won’t get any speed up for a model pruned in an unstructured way unless you make use of sparse matrix operations. Hopefully torch.sparse will be part of a stable release soon.</p>
How to get graph IR from jit::script::module

username: fiaifi
cooked: <p>I want to implement a c++ frontend which convert pytorch IRs to IRs of myself. In torch/csrc/jit/ir.h, I find the class torch::jit::Graph. Now I can load script::module from model file.<br>
Is there any function to translate jit::scirpt::module to jit::script::module.</p>

username: fiaifi
cooked: <p>I read the file jit/script/module.h, maybe I can use module.get_method(“forward”).graph().<br>
Anybody know about how?</p>

username: whitelok
cooked: <p>you can do something like this:</p>
<pre><code class="lang-auto">import torch

def foo(x, y):
    return 2 * x + y
traced_foo = torch.jit.trace(foo, (torch.rand(3), torch.rand(3)))

@torch.jit.script
def bar(x):
    return traced_foo(x, x)

print(traced_foo.graph)
</code></pre>
<p>and I got the graph like this</p>
<pre><code class="lang-auto">graph(%x : Float(3)
      %y : Float(3)) {
  %2 : Long() = prim::Constant[value={2}]()
  %3 : Float(3) = aten::mul(%x, %2)
  %4 : int = prim::Constant[value=1]()
  %5 : Float(3) = aten::add(%3, %y, %4)
  return (%5);
}
</code></pre>

username: sharkdeng
cooked: <p>Just wonder what is the usage of IR graph? Thanks for clarification.</p>
How does gradients gather on the main GPU when using nn.Dataparallel?

username: feiyuhuahuo
cooked: <p>I have a  set of pictures for the working principle of nn.Dataparallel:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/214f46df2c84ee96026db342fb825322a7f78940.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/214f46df2c84ee96026db342fb825322a7f78940" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/214f46df2c84ee96026db342fb825322a7f78940_2_690x215.jpeg" alt="image" data-base62-sha1="4KFBAUGWAyIHRc8g5MBAwX98vfO" width="690" height="215" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/214f46df2c84ee96026db342fb825322a7f78940_2_690x215.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/214f46df2c84ee96026db342fb825322a7f78940_2_1035x322.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/1/214f46df2c84ee96026db342fb825322a7f78940.jpeg 2x" data-dominant-color="D9E2E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1306×408 213 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have a question on the last picture. How does gradients actually gather on the main GPU?  If there’s a Conv2d, its related gradient on GPU:0 is w0 (with shape (256, 128, 3, 3)) and its related gradient on GPU:1 is w1 (with shape (256, 128, 3, 3)). So the  final gradient for this Conv2d on GPU:0 is w0 + w1 ? And the ‘+’ here means element-wise add?</p>

username: albanD
cooked: <p>Hi,</p>
<p>We use the autograd to automatically accumulate the gradients inside the original Parameter whichi is on the main GPU.<br>
Yes the gradients are added element-wise. The same as if you were re-using a given Tensor in multiple expressions when computing your loss.</p>
Can I comobine two loss function into one. One being pytorch tensor other being numpy

username: Hassan_Mahmood
cooked: <p>L=f1+f2<br>
where f1 is working with pytorch tensors, and output is a tensor.<br>
and f2 is using some numpy function.</p>
<p>and I am calling L.backward.<br>
I am not getting any error. Also Network is getting the effect of introducing f2 in loss during training.</p>
<p>Is everything ok here…Or some thing fishy.<br>
Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Since f2 is not computed in a differentiable manner, from the autograd point of view, it is the same as if you were adding a constant number to your loss.<br>
And when taking derivatives, a constant number does not change anything, so you can actually remove <code>f2</code> from this line and you will see the exact same behavior wrt to the gradients that are computed.</p>

username: Hassan_Mahmood
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Thanks a lot for your insight.<br>
1- What will happen to the following function ,</p>
<p>def f1(tensor1,tensor2):<br>
H=tensor1-tensor2<br>
NUM=((tensor1).data.cpu().numpy()-(tensor2).data.cpu().numpy())**2 # some operation in numpy   domain</p>
<p>return H-NUM</p>
<p>L=f1<br>
L.backward()</p>
<p>here Loss is affected by “num” everytime differently depending on tensor1 and tensor2.</p>
<p>2- if f2=NUM , f1=H<br>
and L=f1+f2<br>
is L in step 2 and L in step 1 same in term of training</p>
<p>3- My training is getting positively affected by my original posted question. What can be the reason for that.<br>
Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<ol>
<li>
<p>When you use <code>.data</code> (which you should never use in general <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ) you break the “link” with tensor1 and tensor2 from the point of view of the autograd. And so what you do there is ignore for gradient computations.<br>
Here you could replace <code>NUM = 42</code> and you will get the same result.</p>
</li>
<li>
<p>no because in 1, you do H - NUM but here you do H + NUM. But otherwise yes they will be the same.</p>
</li>
<li>
<p>There are gradients flowing to <code>H</code> (because you compute it in a differentiable manner) and so this will influence your training.</p>
</li>
</ol>