akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Adding nn.Narrow with pytorch

username: meeraone
cooked: <p>How can I do the same thing with pytorch?</p>
<pre><code class="lang-auto">local right = nn.Sequential()
  right:add(nn.Narrow(2, 1, output_size))
</code></pre>

username: ptrblck
cooked: <p>You could write a custom module and pass add it to <code>nn.Sequential</code>.<br>
Something like this might work:</p>
<pre><code class="lang-python">class Narrow(nn.Module):
    def __init__(self, dim, start, length):
        super(Narrow, self).__init__()
        self.dim = dim
        self.start = start
        self.length = length
    
    def forward(self, x):
        x = torch.narrow(x, self.dim, self.start, self.length)
        return x
</code></pre>

username: meeraone
cooked: <p>Are there any ways to hold this tensor input empty?</p>

username: ptrblck
cooked: <p>I don’t quite understand the use case. Could you explain it a bit more?</p>

username: meeraone
cooked: <p>So I could later create concattable and assign same tensor input to this module and others.</p>

username: ptrblck
cooked: <p>I still don’t understand the use case. You are not assigning a tensor input to the module, but create the module and pass the input during the forward pass to it.</p>
Best practice or suggestion for QAT?

username: eleflea
cooked: <p>I am curious about <code>disable_observer</code> and <code>freeze_bn_stats</code> in quantization aware training. I don’t know when should I apply them. I have tried different combinations of two parameters. It seems that has a big impact on accuracy. Is there any best practice for quantization aware training? Like should I disable observer first and when should I disable it, train from scratch or fine-tune a trained model?</p>

username: Vasiliy_Kuznetsov
cooked: <p>hi <a class="mention" href="/u/eleflea">@eleflea</a>, check out <a href="https://github.com/pytorch/vision/blob/master/references/classification/train_quantization.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/references/classification/train_quantization.py</a> for one example.  One approach which has proven to work well is:</p>
<ul>
<li>start QAT training from a floating point pre-trained model and with observers and fake_quant enabled</li>
<li>after a couple of epochs, freeze the BN stats if your network has any BNs (epoch == 3 in the example)</li>
<li>after a couple of epochs, disable observers (epoch == 4 in the example)</li>
</ul>

username: eleflea
cooked: <p>Thanks, I’ll try it.</p>
Switch from LBFGS to ADAM optimizer in the middle of training

username: peymanp
cooked: <pre><code class="lang-auto">#I want to switch the optimizer in the middle of the training 
#from LBFGS to Adam.
#Below is the code that I wrote. 

optimizer = optim.LBFGS(model.parameters(), lr=0.003)
Use_Adam_optim_FirstTime=True
Use_LBFGS_optim=True

for epoch in range(30000):
    loss_SUM = 0
    for i, (x, t) in enumerate(GridLoader):
        x = x.to(device)
        t = t.to(device)

        if Use_LBFGS_optim:
          def closure():
            optimizer.zero_grad()
            lg, lb, li = problem_formulation(x, t, x_Array,t_Array,bndry,pi)
            loss_total=lg+ lb+ li
            loss_total.backward(retain_graph=True)
            return loss_total
          loss_out=optimizer.step(closure)
          loss_SUM+=loss_out.item()
        
        elif Use_Adam_optim_FirstTime:
          Use_Adam_optim_FirstTime=False
          optimizerAdam = optim.Adam(model.parameters(), lr=0.0003)
          model.load_state_dict(checkpoint['model'])
          optimizerAdam.zero_grad()
          lg, lb, li = problem_formulation(x, t, x_Array,t_Array,bndry,pi)
          lg.backward()
          lb.backward()
          li.backward()
          optimizerAdam.step()
          loss_SUM += lg.item()+lb.item()+li.item()
              
        else:
          optimizerAdam.zero_grad()
          lg, lb, li = problem_formulation(x, t, x_Array,t_Array,bndry,pi)
          lg.backward()
          lb.backward()
          li.backward()
          optimizerAdam.step()
          loss_SUM += lg.item()+lb.item()+li.item()
            
    if loss_SUM&lt;.3 and use_LBFGS_optim == True:
      Use_LBFGS_optim=False
      checkpoint = {'model': model.state_dict(),
                    'optimizer': optimizer.state_dict()}

#My questions are:
#Regading the closure function: 
#(1) Is there a way that I can return more than one variable from the closure function?
#(2) Is there a way that I can make three backwards in the closure function instead of only one?
#(3) Why do I need to set retain_graph to True in loss_total.backward(retain_graph=True) of the closure function
#(4) Sometimes, the loss_SUM for the closure function approcheas to 1e+29. Is there a way to avoid this problem?
#-----------------------------
#Regading the switch from LBFGS to Adam:
#(5) When loss_SUM&lt;0.3 and we go to the "elif" part, the loss reduces however after one epoch,
#loss_SUM dramatically increases (e.g., 20 times)
#What is the correct way of switching the optimizer from LBFGS to Adam
#In general, what are the problems of the above code and how I can impprove it.
</code></pre>
How long would it takes for pytorch cuda version to support RTX30 series?

username: GregXu247
cooked: <p>How long would it takes for pytorch cuda version to support RTX30 series?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Do you see any error when you try to use them??</p>

username: GregXu247
cooked: <p>Its working, thanks, and I’m doing more test with yolov5</p>
Force a linear layer's weights to be a diagonal matrix

username: alihasn
cooked: <p>I have seen similar questions but without well defined solutions, but I want to force a layer’s weight matrix to be diagonal. Is there a way I can do this?</p>
<p>One workaround could be to simply set the off-diagonals to zero before each forward pass through the network. Would this work? I’m uncertain if it will cause issues with the backprop.</p>

username: alihasn
cooked: <p>I ended up setting up that linear layer to contain a vector of weights rather than a matrix. Then when it comes time to apply the vector I simply do torch.diag_embed(vector) (which is differentiable) to get the diagonal matrix. This way the weight matrix is only ever diagonal.</p>
So i can just use torch.jit.ScriptModule for every model and get speedup?

username: hadaev8
cooked: <p>Here i have simple test on noise.<br>
With jit 47 secs<br>
<a href="https://colab.research.google.com/drive/1UafCVRxnTNyWFr79f4HGqg-a2VwuLHAp?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1UafCVRxnTNyWFr79f4HGqg-a2VwuLHAp?usp=sharing</a><br>
Without 56 secs<br>
<a href="https://colab.research.google.com/drive/1dKLRgqsVAOrQSe8USSVfeg9a0xFYvCqw?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1dKLRgqsVAOrQSe8USSVfeg9a0xFYvCqw?usp=sharing</a></p>
<p>Does it work as it looks or i got something wrong?</p>
<p>Also, seems like jit version consumes more memory.</p>
Do Linear layers have neurons inside of their features like Conv2d layer channels?

username: ProGamerGov
cooked: <p>Extracting the center neuron of a channel from a Conv2d layer an is pretty straightforward and simple:</p>
<pre><code class="lang-auto"># Extract center neuron of Conv2d channel         
x = input.size(2) // 2
y = input.size(3) // 2
input = input[:, channel, y:y+1, x:x+1] #  (B, C, H, W)
</code></pre>
<p>But it appears like the same technique does not work for Linear layer channels/features as there is nothing after the channel to crop:</p>
<pre><code class="lang-auto">input = input[:, channel] # (B, C)
</code></pre>
<p>Is there a way to extract neurons from Linear layer channels/features?</p>

username: ptrblck
cooked: <p>I might misunderstand your code snippet, but it seems you are indexing the input, not any “neurons”.<br>
If you want to get the filters in the conv layer or weight connections in the linear layer, you can access them via <code>layer.weight</code>.</p>
Capturing Forward calls

username: SpaceCowboy850
cooked: <p>My other thread got no traction, so I’m looking at alternative ways of doing this.</p>
<p>For module derived layers, I can hook into register_forward_pre_hook and register_forward_hook and everything is great!</p>
<p>But I’m trying to capture the little non-module derived functions that frequently pepper these networks.  Stuff like slicing, *, /, +, -, and even stuff like torch.cat, etc.</p>
<p>The problem is, those functions have no hooks to hook into.</p>
<p>I tried Monkey Patching, but I am currently stumped with <strong>getitem</strong> (aka slicing) sometimes returning a different shape when patched vs not.</p>
<p>I’ve looked at Onnx, and it is all there, but way more detail than I want, and it makes it hard to actually see what the operations are.</p>
<p>Tensorboard.SummaryWriter.add_graph appears to just take the Onnx (which I think is torch.jit anyways) and makes a pretty picture (which is also not what I’m after)</p>
<p>I guess one thing could be - does every functional operation of a nn.module sibling?  If that’s the case I could just rewrite the networks to be only using nn.module derived classes.  I’m guessing this is not the case.</p>
<p>Looking for ideas.</p>
Do optimizers track tensors which are updated even after the forward pass through the network?

username: alihasn
cooked: <p>Say that we have a simple model:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__() 
        self.fc1 = nn.Linear(2,4) 
        self.fc2 = nn.Linear(4,4)
        self.fc3 = nn.Linear(4,2) 

    def forward(self, x):
        x = F.tanh(self.fc1(x))
        x = F.tanh(self.fc2(x))
        x = self.fc3(x)
        x = torch.sigmoid(x)
        return x

model = Net()
loss_func = nn.MSELoss()
optimizer = torch.optim.SGD(net.parameters(),lr=LEARNING_RATE,momentum=MOMENTUM,weight_decay=L2_REG)
</code></pre>
<p>and we do some operation (maybe a normalization of the weights of the last layer)</p>
<pre><code class="lang-auto">model.fc3.weight = nn.Parameter(model.fc3.weight/torch.max(model.fc3.weight))
</code></pre>
<p>Would this last operation be traced by the optimizer and when I call</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>will the appropriate gradients and optimization step be taken?</p>

username: ptrblck
cooked: <p>No as seen using your code snippet:</p>
<pre><code class="lang-python">model = Net()
loss_func = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),lr=1.)

model.fc3.weight = nn.Parameter(model.fc3.weight/torch.max(model.fc3.weight))

out = model(torch.randn(1, 2))
out.mean().backward()

print(model.fc3.weight)
&gt; Parameter containing:
tensor([[ 0.9226, -0.6592, -0.6971,  0.8911],
        [-0.8582,  0.9190,  0.2713,  1.0000]], requires_grad=True)

optimizer.step()
print(model.fc3.weight)
&gt; Parameter containing:
tensor([[ 0.9226, -0.6592, -0.6971,  0.8911],
        [-0.8582,  0.9190,  0.2713,  1.0000]], requires_grad=True)

print(model.fc3.weight.grad)
&gt; tensor([[ 0.0622, -0.0140, -0.0225, -0.0629],
        [ 0.0505, -0.0114, -0.0183, -0.0511]])

optimizer.zero_grad()
print(model.fc3.weight.grad)
&gt; tensor([[ 0.0622, -0.0140, -0.0225, -0.0629],
        [ 0.0505, -0.0114, -0.0183, -0.0511]])
</code></pre>
<p>You would have to add the new parameter to the optimizer again.</p>

username: alihasn
cooked: <p>I appreciate you taking the time to answer. Thanks!</p>
Is there a ready implementation of elastic distortion in PyTorch

username: crrrr30
cooked: <p>For elastic distortion applied to 3D images, the time spent on elastic deformation (which is typically implemented in NumPy, e.g. as in <a href="https://gist.github.com/nasimrahaman/8ed04be1088e228c21d51291f47dd1e6" rel="nofollow noopener">https://gist.github.com/nasimrahaman/8ed04be1088e228c21d51291f47dd1e6</a>) is almost 1min/image. I wonder if there is already an implementation in PyTorch using cuda that speeds things up.</p>

username: tom
cooked: <p>You can just write the Gaussians in terms of convolutions with a Gaussian kernel (i.e. <code>exp(-0.5*(x-center)**2/sigma**2))</code> times the normalization factor) cut of to some tasteful size (2 sigma or 3 sigma or what).<br>
I played with it some a while ago in 3d. The thing that screws you up is that with larger sigmas (=filter width) in the Gaussian you’ll have rather large convolution kernels and then the usual cuda kernels not terribly efficient. This could be less pronounced in 2d, but for 3d it was kind of slow. There are some easy and some more possibly more elaborate (but possibly also too compute intensive) ways to mitigate that and I wanted to write a blog post on them but I never got around to it.<br>
The other function you need is <code>torch.nn.functional.grid_sample</code>.</p>
<p>Best regards</p>
<p>Thomas</p>
How to perform backpropagation through time?

username: kchu
cooked: <p>I am using an LSTM for a sequential classification task. I am attempting to backpropagate through time (non-truncated) and update the parameters for each file in my training set. However, I am a bit confused on how to implement BPTT.</p>
<p>Here is a snippet of my code.</p>
<pre><code class="lang-auto">    for file in file_list:
        # Clear existing gradients
        optimizer.zero_grad()

        # Load features and labels for current file
        x_batch, y_batch = read_feat_file(file, conf_dict)

        # Normalize features
        x_batch = scaler.transform(x_batch)

        # Encode labels as integers
        y_batch = le.transform(y_batch).astype('long')

        # Move to GPU
        x_batch = (torch.from_numpy(x_batch)).to(device)
        y_batch = (torch.from_numpy(y_batch)).to(device)

        # Get outputs
        train_outputs = model(x_batch)

        # Calculate loss
        loss = F.nll_loss(train_outputs, y_batch, reduction='sum')
        losses.append(loss.to('cpu').detach().numpy())

        # Backpropagate and update weights
        loss.backward()
        optimizer.step()
</code></pre>
<p>However, based on others posts such as <a href="https://discuss.pytorch.org/t/correct-way-to-do-backpropagation-through-time/11701">this one</a>, it looks like the correct way to perform non-truncated BPTT is to pass the output at the t-1 time step as the input to the model at time t and then call .backward() at the end of the sequence.</p>
<p><strong>My question</strong>: As written, is my code actually performing backpropagation through time?</p>
What happens to memory when moving tensor to GPU?

username: lkc
cooked: <p>I’m trying to understand what happens to the both RAM and GPU memory when a tensor is sent to the GPU.</p>
<p>In the following code sample, I create two tensors - large tensor <code>arr = torch.Tensor.ones((10000, 10000))</code> and small tensor <code>c = torch.Tensor.ones(1)</code>. Tensor <code>c</code> is sent to GPU inside the target function <code>step</code>  which is called by <code>multiprocessing.Pool</code>. In doing so, each child process uses 487 MB on the GPU and RAM usage goes to <strong>5 GB</strong>. Note that the large tensor <code>arr</code> is just created once before calling <code>Pool</code> and not passed as an argument to the target function. Ram usage does not explode when everything is on the CPU.</p>
<p>I have the following questions on this example:</p>
<ol>
<li>
<p>I’m sending <code>torch.Tensor.ones(1)</code> to GPU and yet it consumes 487 MB of GPU memory. Does CUDA allocate minimum amount of memory on the GPU even if the underlying tensor is very small? GPU memory is not a problem for me, and this is just for me to understand how the allocation is done.</p>
</li>
<li>
<p>The problem lies in the RAM usage. Even though I am sending a small tensor to the GPU, it appears as if everything in memory (large tensor <code>arr</code>) is copied for every child process (possibly to pinned memory). So when a tensor is sent to the GPU, what objects are copied to pinned memory? I’m missing something here as it does not make sense to prepare everything to be sent to GPU when I’m only sending a particular object.</p>
</li>
</ol>
<p>Thanks!</p>
<pre><code class="lang-auto">from multiprocessing import get_context
import time
import torch

dim = 10000
sleep_time = 2
npe = 4  # number of parallel executions

# cuda
if torch.cuda.is_available():
    dev = 'cuda:0'
else:
    dev = "cpu"
device = torch.device(dev)


def step(i):
    c = torch.ones(1)
    # comment the line below to see no memory increase
    c = c.to(device)
    time.sleep(sleep_time)


if __name__ == '__main__':
    arr = torch.ones((dim, dim))

    # create list of inputs to be executed in parallel
    inp = list(range(npe))

    # sleep added before and after launching multiprocessing to monitor the memory consumption
    print('before pool')  # to check memory with top or htop
    time.sleep(sleep_time)

    context = get_context('spawn')
    with context.Pool(npe) as pool:
        print('after pool')  # to check memory with top or htop
        time.sleep(sleep_time)

        pool.map(step, inp)

    time.sleep(sleep_time)

</code></pre>
A question about the delay of DataLoader when re-iterate a dataset during training

username: alan_ayu
cooked: <p>Hello everyone,</p>
<p>I found a phenomenon when training a network model. When the iteration of dataset is finished in a epoch, the DataLoader spends about 10s before re-iterating the dataset for the next epoch. This may mean a lot for some models trained on small dataset.<br>
I defined a lot of data augmentation method in the initialization of Dateset, and I set pin_memory to be True, using 8 num_workers.<br>
I did a simple experiment to measure the time needed for initializing Dataset and DataLoader, but the mean time is just about 0.1~0.15s.<br>
Do you have any idea about this question? Thank you.</p>

username: ptrblck
cooked: <p>In each epoch the workers will recreate a batch, so the first iteration might be slower.<br>
Especially if you are using a lot of workers with a large batch size.<br>
IIRC there is an open PR to use multiple workers to create a single batch, which would speed up the first iteration.</p>

username: alan_ayu
cooked: <p>Thanks for your reply, but what is IIRC and the open PR, I have no idea about that:joy:</p>

username: ptrblck
cooked: <p>Oh sorry for the abbreviations. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
IIRC - If I recall/remember correctly<br>
PR - Pull request</p>
<p>In fact, it’s still in the discussion stage, e.g. <a href="https://github.com/pytorch/pytorch/issues/24985">this issue</a>.</p>

username: alan_ayu
cooked: <p>OK, I will try it, Thank you:grinning:</p>

username: ruppesh
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> why does the loading time for first batch increase on increasing the number of worker threads? I am assuming workers run in parallel so ideally the loading time for first batch should not increase significantly with more workers (some increase might be due to overhead of multi-threading). But I see that the time to load first batch increases almost linearly with numbers of workers</p>

username: ptrblck
cooked: <p>Multiple workers will all load a batch at the same time so you might run into a bottleneck from reading from your SSD or your CPU might create the bottleneck.</p>
<p>NIT: note that multiprocessing is used not multi-threading.</p>

username: ruppesh
cooked: <p>Why does the bottleneck not affect the later batches? I am sorry this still is not clear to me.</p>

username: ptrblck
cooked: <p>After the first iteration the workers should have added the loaded batches to the queue, while all workers need to create a new batch in the very first iteration.<br>
As explained before, the more workers you are using the more pressure you are putting on your system, which won’t scale after a system-specific number.</p>
Deep Learning with PyTorch: A 60 Minute Blitz in_features?

username: snowe
cooked: <p>Hi everyone,</p>
<p>I just went through the 60min Blitz tutorial and I have a question. How do they reach a value of 16 * 5 * 5 for the first fully connected layer in their network (in the last part of the tutorial)? 16 is the number of out_channels from the conv2 layer but I can’t figure out the 5 * 5. When I use the formula to calculate it starting with a value of 32 * 32 I reach a value of 10 * 10 and not 5 * 5.</p>
<p>Can anyone help me? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>My calculations:</p>
<ol>
<li>input size (32) - kernel size (5) + 1 = 28</li>
<li>((28 - kernel size (2)) / 2) + 1 = 14</li>
<li>14 - kernel size (5) + 1 = 10<br>
no more layers before the fully connected one so my guess would be 10…</li>
</ol>

username: ptrblck
cooked: <p>In step 3 you are missing the <code>self.pool</code> operation, which would reduce the spatial size by a factor of <code>2</code> and would thus result in an activation of <code>[batch_size, 16, 5, 5]</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: snowe
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
GradCAM Load_learner

username: alejo30
cooked: <p>Hi all</p>
<p>Here’s my situation:</p>
<ol>
<li>Test image (no label) that I want to classify based on a fine-tuned model trained on a similar dataset as the test image.<br>
<code>test_img = test_img.resize(torch.Size([test_img.shape[0],224, 224]))</code><br>
<code>test_img.data.shape</code><br>
<code>torch.Size([3, 224, 224])</code>
</li>
<li>Load an inference learner<br>
learned_model = load_learner(path, ‘export_models/new_model_weights.pkl’)</li>
<li>Predict classification (CORRECT!)<br>
learned_model.predict(test_img)<br>
(Category class 3,<br>
tensor(3),<br>
tensor([0.0157, 0.0804, 0.0055, 0.8985]))</li>
<li>To see the corresponding heat map:<br>
<code>interp = ClassificationInterpretation.GradCAM(learned_model, test_img)</code>
</li>
</ol>
<p>`AttributeError Traceback (most recent call last)<br>
in<br>
----&gt; 1 interp = ClassificationInterpretation.GradCAM(learn_se_rn50, test[0])</p>
<p>/exp/home/rdass/Research/tensorflowEnv/lib64/python3.6/site-packages/fastai/vision/learner.py in _cl_int_gradcam(self, idx, ds_type, heatmap_thresh, image)<br>
137<br>
138 def _cl_int_gradcam(self, idx, ds_type:DatasetType=DatasetType.Valid, heatmap_thresh:int=16, image:bool=True):<br>
–&gt; 139 m = self.learn.model.eval()<br>
140 im,cl = self.learn.data.dl(ds_type).dataset[idx]<br>
141 cl = int(cl)</p>
<p>AttributeError: ‘Learner’ object has no attribute ‘learn’`</p>

username: ptrblck
cooked: <p>This error seems to be raised by a FasiAI wrapper, so maybe <a class="mention" href="/u/jphoward">@jphoward</a> knows what might be wrong.</p>

username: jphoward
cooked: <p><a class="mention" href="/u/alejo30">@alejo30</a> could you please create a github issue (at <a href="https://github.com/fastai/fastai" rel="nofollow noopener">https://github.com/fastai/fastai</a>) with this information?</p>
Is there a way that feeding a input to layers parallelly in PyTorch?

username: I_H_Yoo
cooked: <p>I’m implementing ELMo with PyTorch. I want to feed ELMo’s CNNs, which have different filter map sizes, on word matrices but I’m considering an efficent way. Here’s my code:</p>
<pre><code class="lang-python"># fill the empty tensor iteratively
batch_size = word.size(0)
y = torch.zeros(batch_size, self.kernel_dim)
 
cnt = 0

for kernel in self.kernels:
    temp = kernel(word)
    pooled = torch.max(temp, dim=2)[0]
    y[:, cnt:cnt+pooled.size(1)] = pooled
    cnt += pooled.size(1)

# Using torch.cat
y = []
for kernel in kernels:
    temp = kernel(a)
    y.append(torch.max(temp, dim=2)[0]) # max pooling
    y = torch.cat(y, dim=1)
</code></pre>
<p>I have two questions. The first one is “Is there a way that feeding input to layers in parallel?”. It makes me avoiding for loop iteration so that make my code more efficient. The second one is "Which one is faster between using <code>torch.cat</code> and just filling an empty tensor.</p>
Unable to convert Model with Onnx

username: erict
cooked: <p>Hi i’m trying to convert this model to onnx.</p>
<pre><code class="lang-auto">class onnx_Autoencoder(nn.Module):
 def __init__(self):
        super(onnx_Autoencoder, self).__init__()

        self.Cov1=nn.Conv2d(1,32,3)
        self.batchNorm1=nn.BatchNorm2d(32)
        self.relu1 =nn.ReLU()

        self.Cov2 = nn.Conv2d(32, 64, 3)
        self.batchNorm2 = nn.BatchNorm2d(64)
        self.relu2 = nn.ReLU()



        #
        self.Transpose1= nn.ConvTranspose2d(64, 32, 3)
        self.batchNorm1Tra=nn.BatchNorm2d(32)
        self.relu1Tra=nn.ReLU(inplace=True)

        self.Transpose2 = nn.ConvTranspose2d(32, 1, 3)
</code></pre>
<p>Everything is working fine except when i add track_running_stats=False in batch norm layer</p>
<pre><code class="lang-auto">
self.batchNorm1=nn.BatchNorm2d(32,track_running_stats=False)
</code></pre>
<p>When i check the exported onnx model i get the following errror</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "onnx_check_model.py", line 5, in &lt;module&gt;
    onnx.checker.check_model(onnx_model)
  File "/home/velab/anaconda3/lib/python3.7/site-packages/onnx/checker.py", line 86, in check_model
    C.check_model(model.SerializeToString())
onnx.onnx_cpp2py_export.checker.ValidationError: Node ()'s input 3 is marked single but has an empty string in the graph

==&gt; Context: Bad node spec: input: "27" input: "batchNorm1Tra.weight" input: "batchNorm1Tra.bias" input: "" input: "" output: "30" output: "31" output: "32" output: "batch_norm_dead_output-72" output: "batch_norm_dead_output-73" op_type: "BatchNormalization" attribute { name: "epsilon" f: 1e-05 type: FLOAT } attribute { name: "momentum" f: 0.9 type: FLOAT }
</code></pre>
<p>Could someone provide me an idea on how to solve this?<br>
Thanks</p>

username: Zeleni9
cooked: <p>I have the same issue. Any new findings to solve this problem?</p>

username: jacek9
cooked: <p>I’ve filed a bug report about this error: <a href="https://github.com/pytorch/pytorch/issues/45333" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/45333</a></p>
Could you explain the CosineEmbeddingLoss to me?

username: hollowcodes
cooked: <p>As I understand it, the CBL takes two matrices and calculates the angle between them. But Im not sure what the label y = 1 ; -1 does. Id say that when the label is 1, the aim is that these two matrices should be close to each other. But when its -1, they shouldnt be close to each other. But I guess thats not right. So what does y do? There is nothing in the doc about it (at least Im not finding it). Thanks alot!</p>
Simple example on how BucketIterator works

username: dokato
cooked: <p>I have a numerical sequence data, not in csv format. I was wondering if someone could guide me how to make BucketIterator work as I tried many different versions and I’m hopeless now. The documentation is not to helpful on that <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<pre><code class="lang-python">import torchtext
from torchtext.data import Field
from torchtext.data import BucketIterator

SEQ = Field(sequential=True, use_vocab=False, pad_token=99)
LABEL = Field(sequential=False, use_vocab=False)

data = [dict(seq=[0,1,2,1,3,2], lab=(1,0,1)),
        dict(seq=[0,1,0],lab=(0,0,1)),
        dict(seq=[2], lab=(0,0,1)),
        dict(seq=[3], lab=(1,0,1))]

dt = torchtext.data.Dataset(data, fields = {'seq':SEQ, 'lab':LABEL})
bi = BucketIterator(dt, batch_size=2, sort_key=lambda x: len(x['seq']))
</code></pre>
<p>Expected behaviour: <code>bi</code> would contain batches of examples grouped by length of sequence and padded with 99 (if neccessary). There should be 2 batches of 2 elements each then.</p>
<p>Observed behaviour: bi returns 4 elements each being an example from my dataset, or errors out saying: <code>AttributeError: 'dict' object has no attribute 'seq'</code></p>
Mobile Net dimensions

username: picklerick
cooked: <p>Hello! The following is the architecture for “MobileNet”:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/483858e9c763b02a952f1e95f4810bdd1e8274bc.png" alt="mobile_net" data-base62-sha1="aiT6hWCilUGg5CXiwDSetqWK8e8" width="428" height="489"></p>
<p>The input size DOES get halved when going from the row 4 <code>conv_dw / s2</code> to the row 5 <code>conv / s1</code> layer  (112 x 112 x 64) to (56 x 56 x 64).</p>
<p>I don’t understand why the input size for the last <code>conv / s1</code> layer is 7 x 7 x 1024 even though the layer before it (<code>conv_dw / s2</code> layer) has a stride of 2. Shouldn’t the input size for the last <code>conv / s1</code> layer be<br>
3 x 3 x 1024 instead?</p>

username: ptrblck
cooked: <p>Based on the stride of 2 I would assume the same. However, since an avg pooling layer with a kernel size of <code>7x7</code> is used afterwards, I guess this particular layer should use a stride of 1?</p>

username: picklerick
cooked: <p>Yeah, it works with a stride of 1. Probably a typo in the original MobileNet-v1 paper I suppose.</p>
Mean and std for transfer learning

username: arjun_pukale
cooked: <p>I want to use resnet pretrained on Imagenet and fine tune the model on my new dataset (consisting of cell images).<br>
So what must be the mean and std in <code>transforms.Normalize(mean, std, )</code>.<br>
Should it be that of the ImageNet dataset (coz we are using a pretrained model trained on ImageNet)<br>
or we should pass the mean and std of this new dataset?</p>
Indexing cuda tensor with cpu tensor

username: Jihoo_Kim
cooked: <p>What is the difference between using cuda tensor and cpu tensor when indexing cuda tensor?<br>
For example,</p>
<p>if use_cuda:<br>
index = torch.randperm(batch_size).cuda()<br>
else:<br>
index = torch.randperm(batch_size)</p>
<p>mixed_x = lam * x + (1 - lam) * x[index, :]<br>
y_a, y_b = y, y[index]</p>
<p>Above code is from <a href="https://github.com/facebookresearch/mixup-cifar10" rel="nofollow noopener">https://github.com/facebookresearch/mixup-cifar10</a><br>
Is it faster? (I see no difference between them when training)</p>

username: Henry_Chibueze
cooked: <p>Depending how much computation u r doing u might not see the difference but if u r performing heavy computations then u’ll see that it’s faster</p>
Multiple GPU error _ not on the same device

username: fsh
cooked: <p>Hi,</p>
<p>I am running the code below on multiple GPU mode. When I take the <code>x</code>,<code>y</code> variables either to cuda() or not, I can not get the whole network running on the same device. Depending on the device, I get this error.</p>
<pre><code class="lang-auto">RuntimeError: expected device cuda:1 but got device  cuda:0
RuntimeError: expected device cuda:0 but got device cpu
</code></pre>
<pre><code class="lang-auto"># x = torch.linspace(0, args.max-1, args.max).cuda()
# y = torch.linspace(0, args.max-1, 4*args.max).cuda()

x = torch.from_numpy(np.linspace(0, args.max-1, args.max))
y = torch.from_numpy(np.linspace(0, args.max-1, 4*args.max))

model = network(args.max, x, y)
model = nn.DataParallel(model)
model.cuda()
</code></pre>
<p>I have tried taking <code>x</code>, <code>y</code> into the network, both in cuda and cpu mode, but the same error occurs again.</p>
<p>I would appreciate your help.</p>

username: ptrblck
cooked: <p>Are you using any <code>cuda()</code> or <code>to()</code> calls inside your model (<code>__init__</code> or <code>forward</code>)?<br>
If so, could you remove them as they might create the device mismatch.<br>
<code>nn.DataParallel</code> will automatically create model replicas for you and you don’t need to push internal model parameters to a specific device manually (only the <code>nn.DataParallel</code> model to the default device).</p>

username: fsh
cooked: <p>Yes, I was using them in the <code>__init__</code>. Now, I tried taking them into the <code>forward</code> in the model, and the problem seems to be solved.</p>
<p>Thanks!</p>
Multiplying tensors along selected dimensions

username: Anurag_Pandey
cooked: <p>I want to scale the matrices by a group of scalar values, consider input to be input tensor of dim [a,b,c,d](a being batch size and b being number of matrices) and scaling factors of dim [e] (Indicating e scaling factors). Is there any elegant way to do this other than looping on each factor and concat the final output</p>
<pre><code class="lang-auto">input = torch.rand((1500, 4, 3, 3))
scalar = torch.rand((12))
out = comb * scalar[None, :, None, None]
</code></pre>

username: ptrblck
cooked: <p>What would be the desired ouput shape?<br>
If you want to create an output of <code>[1500, 12, 4, 3, 3]</code>, you could use this code:</p>
<pre><code class="lang-python">input = torch.rand((1500, 4, 3, 3))
scalar = torch.rand((12))
out = input.unsqueeze(1) * scalar[None, :, None, None, None]
print(out.shape)
&gt; torch.Size([1500, 12, 4, 3, 3])

for i in range(scalar.size(0)):
    print((out[:, i] == input * scalar[i]).all())
&gt; tensor(True)
tensor(True)
...
</code></pre>

username: Anurag_Pandey
cooked: <p>Thanks for your time <a class="mention" href="/u/ptrblck">@ptrblck</a>!!<br>
The desired shape would be [1500, 48, 3, 3] .</p>

username: ptrblck
cooked: <p>In that case you could flatten <code>dim1</code> and <code>dim2</code> to a single dimension via:</p>
<pre><code class="lang-python">out = out.view(out.size(0), -1, out.size(3), out.size(4))
</code></pre>

username: Anurag_Pandey
cooked: <p>Yes that’ll work perfectly!!<br>
Thank you so much!</p>
Can we **not** reduce GPU memory consumption during inference?

username: loser
cooked: <p>I find that the memory consumed when doing inference (seen by nvidia-smi) is usually smaller than training. This is expected, but can be annoying when we are sharing GPUs with others. The afterward training may have OOM risk if others’ processes start while our code is doing inference.</p>
<p>I have read that PyTorch has a cache policy on GPU memory management: unused gpu memory will not be released to the OS, so that further gpu variables can be faster allocated.</p>
<p>Thus my questions are:<br>
(1) Is Pytorch clearing GPU memory cache when switching from training to inference?<br>
(2) How can we keep the GPU memory usage of inference as large as that of training, i.e., forcing Pytorch not to reduce memory consumption?</p>
Torch.nn.functional.interpolate VS skimage.transform.resize

username: tianmaliuxing
cooked: <p>I found the result is different between torch.nn.functional.interpolate and skimage.transform.resize, although I have used the same order or mode.<br>
Does anybody know the other resize operation, I want it to be the same result as torch.nn.function.interpolate.</p>

username: ptrblck
cooked: <p>I don’t know how skimage implements the interpolation, but <a href="https://github.com/python-pillow/Pillow/issues/2718#issuecomment-333669547">this post</a> might be interesting in this context, as it explains the difference between OpenCV and PIL (which might be the same root cause for the difference you are seeing).</p>
How to do ONNX to TensorRT in INT8 mode?

username: GB_K
cooked: <p>Hello.</p>
<p>I am working with the subject, PyTorch to TensorRT.</p>
<p>With a tutorial, I could simply finish the process PyTorch to ONNX.<br>
And, I also completed ONNX to TensorRT in fp16 mode.</p>
<p>However, I couldn’t take a step for ONNX to TensorRT in <strong>int8 mode</strong>.<br>
Debugger always say that `You need to do calibration for int8*.*</p>
<p>Does anyone know how to do convert ONNX model to TensorRT int8 mode?</p>
<p>Thank you in adavance</p>

username: seungjun
cooked: <p>This thread says exporting quantized pytorch models to onnx is not yet supported.</p><aside class="quote" data-post="14" data-topic="76884">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/db5fbb/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/onnx-export-of-quantized-model/76884/14">ONNX export of quantized model</a> <a class="badge-category__wrapper " href="/c/quantization/17"><span data-category-id="17" data-drop-close="true" class="badge-category " title="This category is for questions, discussion and issues related to PyTorch’s quantization feature."><span class="badge-category__name">quantization</span></span></a>
  </div>
  <blockquote>
    General export of quantized models to ONNX isn’t currently supported. We only support conversion to ONNX for Caffe2 backend. This thread has additional context on what we currently support - <a href="https://discuss.pytorch.org/t/onnx-export-of-quantized-model/76884/8">ONNX export of quantized model</a>
  </blockquote>
</aside>

<p>You can try quantizing after you export pytorch model to onnx by using <a href="https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/README.md" rel="noopener nofollow ugc">onnxruntime</a>.</p>
<p>Otherwise, you may want to check out if direct export from pytorch to tensorrt supports quantized models.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/NVIDIA-AI-IOT/torch2trt">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/NVIDIA-AI-IOT/torch2trt" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/11f4a06b788f381fe69e8c5db096abc577f1204739ec1f3587d1e6d166967a84/NVIDIA-AI-IOT/torch2trt" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/NVIDIA-AI-IOT/torch2trt" target="_blank" rel="noopener nofollow ugc">GitHub - NVIDIA-AI-IOT/torch2trt: An easy to use PyTorch to TensorRT converter</a></h3>

  <p>An easy to use PyTorch to TensorRT converter. Contribute to NVIDIA-AI-IOT/torch2trt development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Avoid loading Torch library multiple times in CUDA

username: Kumar_Govindam
cooked: <p>In my program, I am using my own decoder that reads video data in CUDA memory.  The decoder reads the video frame like OpenCV. I am using converting the frame data to torch tensor and using in my model.<br>
When I am initializing the decoder normally, it takes 50MB CUDA memory. Initialising means it starts decoding the frames. But, when I am initializing the decoder with torch c++ lib, its taking 500MB, which is much higher. And if I am initializing 2 decoders, its creating 2x500MB = 1GB memory, which is unusual, as it should not load the torch lib every time.</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;
#define kCHANNELS 3

void deleter(void* arg) {};
struct Net{

    void    *pReader = NULL;
 
    Net (const char * video_url) {   
    void *pCudaContext      = nullptr;
    int iGPUDeviceNumber    = 0;

    ///////////// Here, I am initialing the decoder which has nothing to do with torch lib
    ///////////// When I am just initialing the decoder, its taking lot of CUDA memory, although it does not need torch lib to run.
    pReader = decoder::Init (  video_url,
                                        NULL,
                                        iGPUDeviceNumber,
                                        ....................
                                        ....................); };
   
   torch::Tensor getFrame(){
            //////////////////////// Here, I am reading the data frame
            pOutData        = 0;
            nResult = decoder::GetVideoFrame (pReader, &amp;pOutData);
            auto options = torch::TensorOptions().dtype(torch::kUInt8).device(torch::kCUDA, 0);  
            auto input_tensor = torch::from_blob(
                    pOutData, {1, 360, 640, kCHANNELS}, deleter, options);
           return input_tensor.clone()
        }};
  
</code></pre>
Torch installed by pip was compiled with mkl or not

username: ckmufeng
cooked: <p>after pip install torch，i use<br>
<code>print(*torch.__config__.show().split("\n"), sep="\n")</code> to check mkl,  return somthing like this<br>
“… BLAS=MKL…”<br>
but, when use ldd libtorch.so, no mkl library was linked.<br>
and i use <code>perf top</code> to view some pytorch scripte invoke, no mkl lib was invoked.<br>
that’s weired， so i want to know whether torch(pip install) has mkl or not . if it does, how mkl is compiled into libtorch .</p>
How to get pooler_output from fine tuned BERT?

username: sasha
cooked: <p>Hi,</p>
<p>I have fine-tuned BERT on my text for multiclass classification with 11 classes and saved the models for five epochs.<br>
I have done BERT tokenizer and encoding data, and used BERT pretrained model as following:</p>
<pre><code class="lang-auto">model = BertForSequenceClassification.from_pretrained("bert-base-uncased",
                                                      num_labels=11,
                                                      output_attentions=False,
                                                      output_hidden_states=False)```
## Then for loading my model I used:

model_ft = BertForSequenceClassification.from_pretrained("bert-base-uncased",
                                                      num_labels=11,
                                                      output_attentions=False,
                                                      output_hidden_states=False)
model_ft.load_state_dict(torch.load('finetuned_BERT_epoch_5.model', map_location=torch.device('cpu')))  ```

##And for the prediction I have done:


model_ft.eval()
with torch.no_grad():
    output = model_ft(input_ids_val,attention_masks_val)
   
</code></pre>
<p>However, my output is: output[0].size() = (my_validation size, 11) and out_put[1] is empty.<br>
I also tried output_hidden_states=True but still I am getting a tuple ((my_validation size, 11, empty), tuple((tensr), (tesnor)))</p>
<p>So I have two questions:</p>
<ol>
<li>
<p>I think my output is supposed to be a tuple (last_hidden_state, pooler_output). Given that, did I miss any step that I am not receiving the pooler_output?</p>
</li>
<li>
<p>How can I pull out the pooler layer to use its result of that as an entire sequence representation and use it for other tasks?</p>
</li>
</ol>
<p>I appreciate any help!!</p>
Is there a good way to clear the graph connected to a tensor after it has gone through half of the layers?

username: jwillette
cooked: <p>I have a situation where I want to sometimes have an input only pass gradients to the top layers, and sometimes to all of the layers. The way I am currently accomplishing this is by doing this…</p>
<pre><code class="lang-auto">def forward(self, x, top=True):
  x = self.first_layers(x)
  if top:
    x = x.detach()
    x.requires_grad_(True)

  return self.last_layers(x)
</code></pre>
<p>I am wondering if this is actually accomplishing what I intend or not and if there is a better way to do this? Thanks</p>

username: JuanFMontesinos
cooked: <p>In fact detach is just to do that. Breaking the graph.<br>
So it’s the proper way to do it</p>
[Performance] Map style vs iterable dataset in PT for Remote storage

username: rn91
cooked: <p>A map-style dataset in Pytorch has the  <code>__getitem__()</code>  and  <code>__len__()</code>  and iterable-style datasets has  <code>__iter__()</code>  method. From what I understand, map-style is used when you know dataset length and iterable is used when you do not.</p>
<p>I have a use case where I want to use data from remote storage device where I have different types of datasets… 1) bunch of Tar files, 2) bunch of zip files and also 3) bunch of small images.(100K) I am looking into writing custom dataset but getting confused about which one to use. Should I use map-style or iterable dataset. What is more popularly used?</p>
<p>I believe iterabledataset has cons that you cannot use distributedsampler with it but also has advantage that you can stream data directly without calling each object from list.</p>
<p>Does iterable dataset provide any more advantages than map-style in my specific case?<br>
<a href="https://github.com/pytorch/pytorch/issues/38419" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/38419</a> I also saw if I can use webdataset for my usecase… Wondering why they used iterabledataset instead of map-style. As iterable dataset does not work with distributedsampler</p>
<p>What will be better in terms of performance?</p>
<p>Can someone suggest? Thanks in advance!</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a></p>
Eliminate data loading pause between train and val loops

username: Connor_Anderson
cooked: <p>I’m wondering if anyone has tried to do this. When shifting from train to val or vice versa, there is a pause in training as the active DataLoader changes and begins filling up it’s queue of batches (I’m talking particularly about using the <code>torch.utils.data.DataLoader</code> with <code>num_workers &gt; 1</code>). When using large batches, this pause can be significant. Is there some way to have the next dataloader start queuing batches as soon as the previous one is finished buffering; even if the model hasn’t processed all those batches yet?</p>

username: tom
cooked: <p>So the dataloaders are initialized <a href="https://pytorch.org/docs/stable/data.html#multi-process-data-loading" rel="nofollow noopener">when the iterator is created</a>. You could look into creating the iterator right after loading your last batch of data with the other iterator.</p>
<p>The other question is whether it is worth the complexity you’d be introducing with it. As you note, this only affects the first batch of each iteration, so if you have a sizeable number of batches, it’ll contribute at most 1/number of batches to the total run time, and you get better leverage from considerably smaller optimizations that affect each batch (from where/when do I do augmentation or other processing to where/how do I store my data).<br>
I’m saying this because in my experience it is very easy to be mislead by intuition when looking at optimizing the overall runtime. This bit is very visible during training (the progress bars aren’t ticking and all) but what kind of speedup of the overall time can you get from eliminating it?</p>
<p>Best regards</p>
<p>Thomas</p>

username: Connor_Anderson
cooked: <p>Thanks Tom, for the suggestion and also for the good point about optimization.</p>