akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dataloader is slower if model is on gpu

username: dodo
cooked: <p>I run ffmpeg (to preprocess data) as subprocess from the workers (num_workers=6). there is nothing related to gpu in dataloading, just subprocess.run ffmpeg.</p>
<p>I realized that if I run dataloaders when my model is on gpu, the dataloading becomes 3 to 4 times slower. I see less ffmpeg proceses in htop and less utilization overall.</p>
<p>The interesting part is that I don’t even enter the training loop, I commented out everything but batch retrieval loop and still if the initial model is on gpu the dataloading is slower. I verified with nvidia-smi that no processing is done at gpu.</p>
<p>I guees it has something to do with internal locks due to cuda or process spawn/data sharing logic but at the first glance I couldn’t find anything.</p>
<p>Is it ok to create a subprocess in workers which are themselves subprocesses? What happens to dataloaders if cuda is involved?</p>
<p>Thanks.</p>
Meaning of each entry in torch.autograd.profiler.profile

username: DanielWang
cooked: <p>Hi, All</p>
<p>I just try using the <code>torch.autograd.profiler.profile</code><br>
There are several entries</p>
<pre><code class="lang-auto">Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  
</code></pre>
<p>where could I find the explanation for these? does the time calculated in CPU total also overlap with the time in CUDA total if an operator involves the GPU computation?</p>
<p>Thanks!</p>
nn.Linear projection shortcut Resnet

username: TheUnnamed22
cooked: <p>Hello!</p>
<p>I am trying to use resnet architecture for reinforcement learning. Now in the original paper they use a conv layer to scale down the dimension. I want to use a normal nn.linear layer for that but idk how to accomplish it 100%. How should be the weights and bias initialized? Also If someone knows a paper where resnet is used in reinforcement learning would be helpful. Thank you very much!</p>
Loading LSUN dataset

username: Arash
cooked: <p>I am trying to load the LSUN dataset:</p>
<p>train_set = dset.LSUN ( db_path = … , classes=[‘bedroom_train’, … ], transform = … )<br>
trainloader = DataLoader(train_set, batch_size = 64)<br>
for i, data in enumerate(trainloader):<br>
…</p>
<p>But I am getting this error:</p>
<p>for i, data in enumerate(trainloader):<br>
File “/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py”, line 190, in <strong>next</strong><br>
batch = self.collate_fn([self.dataset[i] for i in indices])<br>
File “/usr/local/lib/python2.7/dist-packages/torchvision/datasets/lsun.py”, line 125, in <strong>i, getitem</strong><br>
img, _ = db[index]<br>
File “/usr/local/lib/python2.7/dist-packages/torchvision/datasets/lsun.py”, line 37, in <strong>getitem</strong><br>
imgbuf = txn.get(self.keys[index])<br>
IndexError: list index out of range</p>
<p>Any ideas? Thanks.</p>

username: smth
cooked: <p>i’m not entirely sure, but maybe the cache is somehow screwed up. If there are any files generated in your current directory that look like index files for LSUN dataset, try to delete them.</p>

username: aknirala
cooked: <p>Awesome… it fixed the issue for me.</p>
How can I change the path to the repository folder?

username: Pro100rus32
cooked: <p><strong>How can I change the path to the repository folder?</strong></p>
<blockquote>
<p>Using cache found in C:\Users\Андрей/.cache\torch\hub\ultralytics_yolov5_master</p>
</blockquote>
<p><strong>I am using yolov5:</strong></p>
<blockquote>
<p>torch.hub.load(‘ultralytics/yolov5’, ‘custom’, path_or_model=‘last.pt’)</p>
</blockquote>

username: ptrblck
cooked: <p>If you want to change the <code>torch.hub</code> directory, you could use <code>torch.hub.set_dir(DIR)</code>.</p>
RunTime Error:Expected batch2_sizes[0] == bs && batch2_sizes[1] == contraction_size to be true, but got false

username: Rabiul_Hasan
cooked: <p>def forward(self, xf):</p>
<pre><code>    a, b = self.model(torch.ones((self.size, self.size,2)), torch.zeros((self.size, self.size,2)))
    c, d = self.model(torch.zeros((self.size, self.size,2)), torch.ones((self.size, self.size,2)))
    b_c=b @ c  ***Error is shown here***
    di_v=torch.div(b_c,d)
    
    fin=a-di_v
    
    yf=fin@xf

    yf_abs = torch.sqrt(yf[..., 0] * yf[..., 0] + yf[..., 1] * yf[..., 1])

    output = self.softmax(detector_region(yf_abs))

    return output
</code></pre>
<p>for this code, I am getting following error. Any suggestion will be highly appreciated…</p>
<p>RuntimeError: Expected batch2_sizes[0] == bs &amp;&amp; batch2_sizes[1] == contraction_size to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>

username: ptrblck
cooked: <p>Could you print the shapes of <code>b</code> and <code>c</code> and check, if they would be compatible for a matrix multiplication?<br>
Based on the error message it seems that one of these two tensors might have an unexpected shape.</p>

username: Rabiul_Hasan
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37ec218e7284ceba2d2f43445e0a01875c8d5a8a.jpeg" alt="Capture" data-base62-sha1="7YIav5ZJ4y839IMMiqCpSW5hnHY" width="353" height="72"></p>
<p>The size for both b and c are ([200, 200, 2]). I am not  sure how still there is a mismatch in the size.</p>

username: ptrblck
cooked: <p>A shape of <code>[200, 200, 2]</code> for both tensors is incompatible for a matrix multiplication.<br>
The <a href="https://pytorch.org/docs/stable/generated/torch.matmul.html#torch.matmul">docs</a> explain the shape requirements.</p>
<p>Assuming the first dimension is the batch size, you would have to make sure the shapes are defined as:</p>
<pre><code class="lang-python">C = A @ B
[k, m, p] = [k, m, n] @ [k, n, p]
</code></pre>
How to properly implement discriminative learning rate in PyTorch

username: Shrey_Joshi
cooked: <aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/66933742/how-to-properly-implement-discriminative-learning-rate-in-pytorch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/10643571/shrey-joshi" target="_blank" rel="noopener nofollow ugc">
    <img alt="Shrey Joshi" src="https://lh3.googleusercontent.com/-j2zXoaxHFhk/AAAAAAAAAAI/AAAAAAAAAk4/8fhrW1Jc-is/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/66933742/how-to-properly-implement-discriminative-learning-rate-in-pytorch" target="_blank" rel="noopener nofollow ugc">How to properly implement discriminative learning rate in PyTorch</a>
</h4>

<div class="tags">
  <strong>machine-learning, computer-vision, pytorch, transfer-learning</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/10643571/shrey-joshi" target="_blank" rel="noopener nofollow ugc">
    Shrey Joshi
  </a>
  on <a href="https://stackoverflow.com/questions/66933742/how-to-properly-implement-discriminative-learning-rate-in-pytorch" target="_blank" rel="noopener nofollow ugc">05:01PM - 03 Apr 21 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Why can't I overfit a single example in LSTM?

username: learning_experiments
cooked: <p>I’m experimenting with LSTMs and tried to overfit a single example. However, I’m unable to do that. This seems very surprising. Can someone tell me if I’m going wrong somewhere?</p>
<p>Here’s the code</p>
<pre><code>import torch
import torch.nn as nn

lstm = nn.LSTM(1, 512).cuda()
linear = nn.Linear(512,1).cuda()

in_tensor = torch.Tensor([[1,2,3]]).reshape(3,1,1).cuda()
output_tensor = torch.Tensor([[2,5,23]]).reshape(3,1,1).cuda()

optimizer = torch.optim.Adam(lstm.parameters(), lr=2e-3)
loss_fn = nn.MSELoss()

for i in range(100000):
    pred_out = linear(lstm(in_tensor)[0])
    optimizer.zero_grad()
    loss = loss_fn(pred_out, output_tensor)
    if i % 1000 == 0:   
        print(i, loss)
    loss.backward()
    optimizer.step()
</code></pre>
<p>The loss saturates at ~100 and model predictions won’t change. May be it got stuck at a local minima, but feels extremely surprising that this is happening.</p>

username: Dwight_Foster
cooked: <p>It is because you are only optimizing the lstm not the linear layer. Because of this the linear layer is not learning so your model cannot overfit. To optimize both you can do this</p>
<pre><code class="lang-auto">params = list(linear.parameters()) + list(lstm.parameters())
optimizer = torch.optim.Adam(params, lr=2e-3)
</code></pre>
<p>I tested that and my model overfit the data.</p>
Exogenous variable inclusion to dataloader

username: dervishson
cooked: <p>Hi<br>
I am new at PyTorch, and try to model time series problem with 1-D CNN. I have to provide 3 different inputs, which are load, temperature, day dummies, separately to the network. However, I did not find any clue to provide this type of setting to the data loader in pytorch. Which way I can try to provide this setting to data loader?</p>

username: crcrpar
cooked: <p>Hi,</p>
<p>If your dataset’s <code>__getitem__</code> method returna a tuple of three <code>torch.Tensor</code>s, <code>DataLoader</code> will generate a tuple of three minibatches of those tensors.</p>

username: dervishson
cooked: <p>Thank you. I will give a shot with your recommendation.</p>
GPU MaxPool2d too slow with stride=1

username: aktgpt
cooked: <p>I have a large sparse matrix <code>[1, 1600, 512, 512]</code> with values either <code>0</code> or <code>1</code> and I wanted to calculate the number of elements with value <code>1</code> along <code>dim=1</code> within a window, let’s say <code>[40, 40]</code>.</p>
<p>I solved it by passing the tensor with a <code>nn.MaxPool2d((40, 40),stride=1)</code> and summing along <code>dim=1</code> in the end. This turned out to be very slow and consuming too much GPU memory (out of memory error). So, I divided the image into chunks along <code>dim=1</code> using <code>torch.chunk</code>. It solved out of memory issues, but that also turned out to be slow as well.</p>
<p>I made a minimum running code to benchmark these approaches inspired by code from <a class="mention" href="/u/ptrblck">@ptrblck</a> (<a href="https://discuss.pytorch.org/t/one-channel-conv2d-on-gpu-takes-too-much-time/53411/4" class="inline-onebox">One channel conv2d on GPU takes too much time - #4 by ptrblck</a>) like this:</p>
<pre><code class="lang-auto">import torch
import time

torch.backends.cudnn.benchmark = True

nb_iter = 10
# dummy image
img = torch.randint(low=0, high=2, size=[1, 1600, 512, 512]).cuda().float()

kernel_shape = [40, 40]
max_pool2d = torch.nn.MaxPool2d((kernel_shape[0], kernel_shape[1]), stride=1)

## Max Pooling
# warmup
for _ in range(10):
    max_pool_op = max_pool2d(img)

# test
torch.cuda.synchronize()
tic = time.time()
for _ in range(nb_iter):
    max_pool_op = max_pool2d(img)
torch.cuda.synchronize()
toc = time.time() - tic
print("GPU MaxPool Time:", toc / nb_iter)

## Max Pooling Chunking
torch.cuda.empty_cache()
# warmup
for _ in range(10):
    max_pool_chunk_op = []
    x = torch.chunk(img, 16, dim=1)
    for chunk in x:
        # chunk = torch.tensor(chunk)
        op = max_pool2d(chunk)
        max_pool_chunk_op.append(op)
    max_pool_chunk_op = torch.cat(max_pool_chunk_op, dim=1)

# test
torch.cuda.synchronize()
tic = time.time()
for _ in range(nb_iter):
    max_pool_chunk_op = []
    x = torch.chunk(img, 16, dim=1)
    for chunk in x:
        # chunk = torch.tensor(chunk)
        op = max_pool2d(chunk)
        max_pool_chunk_op.append(op)
    max_pool_chunk_op = torch.cat(max_pool_chunk_op, dim=1)
torch.cuda.synchronize()
toc = time.time() - tic
print("GPU MaxPool Chunk Time:", toc / nb_iter)

print(
    torch.sum(torch.eq(max_pool_chunk_op.cpu(), max_pool_op.cpu()))
    / max_pool_op.cpu().nelement()
)
</code></pre>
<p>And it gave me these results:</p>
<pre><code class="lang-auto">GPU MaxPool Time: 5.212354826927185
GPU MaxPool Chunk Time: 5.250830364227295
tensor(1.)
</code></pre>
<p>I’m using  GeForce GTX TITAN X GPU and Torch version 1.8.1 with CUDA 10.2.  My GPU was always at 100% while running the code.</p>
<p>Is there any apparent reason that max-pooling is so slow(5 secs/iter)? In my application, the number of channels can be even greater, so it is currently a big bottleneck for me. Is there a way to make it faster (under a second)?</p>
How to free GPU memory Changing Architectures While Training

username: jnx
cooked: <p>Hello Guys,</p>
<p>I need your help.</p>
<p>I only have one GPU installed in my computer. I am currently working on architecture selection hence I test different architectures (e.g. 10 architectures) with one run.  I do it one by one.</p>
<pre><code class="lang-auto">for i in Architectures:
    model = network(architecture = i)
    ....

    for mini_epoch in Total_epochs:
        ##trains the architecture
        loss = model(input)
        loss.backward()
        optimizer.step()
</code></pre>
<p>My problem is that something is accumulating in the GPU memory that I cannot finish training all the architectures even though I treat them as a single model at a time. Testing the architectures one at a time works but it is too tedious. Is there a way to release GPU memory when a new architecture is passed as new model?</p>
<p>Thanks in advance!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Two things:</p>
<ul>
<li>We have a custom allocator, so even when the memory is released, you won’t see it available on <code>nvidia-smi</code> but you will be able to use it in pytorch.</li>
<li>The memory is realeased only when you don’t reference it anymore. You might want to wrap the content of your inner loop in a function so that all the intermediary results go out of scope (and are thus released) between loop iterations.</li>
</ul>

username: jnx
cooked: <p>The second bullet does the job for me. Thanks!</p>

username: hadaev8
cooked: <p>If i want to remove module of model, can i just do model.layer = None? Or del model.layer</p>

username: albanD
cooked: <p>Both will work.<br>
The only difference is that if you access it later, in one case your will get <code>None</code> and in the other case you will get an error saying that there is no attribute with this name.</p>

username: hadaev8
cooked: <p>But how to remove param from optimizer?</p>

username: albanD
cooked: <p>That would be trickier, I don’t think our optimizer API supports removing parameters.<br>
That being said, if the <code>.grad</code> field of that Tensor is <code>None</code>. Then the optimizer will just ignore it. And because it is not in the network anymore, it won’t be updated. So you can leave it in the optimizer.</p>

username: xin_Wang1
cooked: <p>I also have same problem in my code. Luckily, I found the method to solve it.<br>
you have to set torch.backends.cudnn.benchmark=False</p>
Torch:nn::module (with parameters) how to add bias

username: Kallinteris-Andreas
cooked: <pre><code class="lang-auto">struct Net : torch::nn::Module {
	Net(int32_t numIn, int32_t numOut, int32_t numHid) {
		weightsA = register_parameter("input", torch::rand({numIn, numHid}))*0.013;
		weightsB = register_parameter("output", torch::rand({numHid, numOut}))*0.013;
	}
	torch::Tensor forward(torch::Tensor input) {
		torch::Tensor hidden_layer, output_layer;
		hidden_layer = (torch::mm(input, weightsA));
		output_layer = (torch::mm(hidden_layer, weightsB));
		return output_layer;
	}
	torch::Tensor weightsA, weightsB;
};
</code></pre>
<p>I have a simple module with parameters, is there an easy way to add a bias node in the hidden layer (preferably without doing some computationally expensive operation like memcopy)</p>
<p>Note: you can provide python code if you are uncomfortable with c++ i should be able to translate it</p>

username: Matej_Kompanek
cooked: <p>Something like this ? Maybe you’re looking for <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html" rel="noopener nofollow ugc">Linear</a></p>
<pre><code class="lang-auto">struct Net : torch::nn::Module {
	Net(int32_t numIn, int32_t numOut, int32_t numHid) {
		weightsA = register_parameter("input", torch::rand({ numIn, numHid }))*0.013;
		weightsB = register_parameter("output", torch::rand({ numHid, numOut }))*0.013;
		bias = register_parameter("bias", torch::zeros({ numOut }));
	}
	torch::Tensor forward(torch::Tensor input) {
		torch::Tensor hidden_layer, output_layer;
		hidden_layer = (torch::mm(input, weightsA));
		output_layer = (torch::mm(hidden_layer, weightsB));
		output_layer += bias;
		return output_layer;
	}
	torch::Tensor weightsA, weightsB, bias;
};

</code></pre>

username: Kallinteris-Andreas
cooked: <p>No I want to add a bias node in the hidden layer (I updated the original post to clarify it)</p>
<p>Also, I am not looking for linear because I read somewhere in the docs that linear is not the right choice for learning (it recommended torch::Tensor parameters instead)</p>

username: Matej_Kompanek
cooked: <p>Well, I don’t really catch your drift. Maybe you could point me to the doc you’re referring to. Adding the bias to hidden layer would mean changing the bias definition to:</p>
<pre><code class="lang-auto">bias = register_parameter("bias", torch::zeros({ numHid }));
</code></pre>
<p>and then</p>
<pre><code class="lang-auto">hidden_layer += bias;
</code></pre>
<p>Linear does basically what you’re doing …</p>
<pre><code class="lang-auto">output = torch::mm(input, weight) + bias
</code></pre>
Efficient way to multiply n matrices to another n matrices

username: dangne
cooked: <p>I have two sets of matrices.<br>
Set A includes 10 matrices with the same size of [11x3].<br>
Set B includes 10 matrices with the same size of [3x100].<br>
I want to multiply the i-th matrix in set A with the i-th matrix in set B. What is the most efficient way to do this?</p>
<p>Thanks a lot.</p>

username: AlphaBetaGamma96
cooked: <p>So, you have Tensor A of shape [10, 11, 3] and Tensor B of shape [10, 3, 100]? <code>torch.matmul</code> will do the trick.</p>
<pre><code class="lang-auto">A = torch.randn(10,11,3)
B = torch.randn(10,3,100)
C = torch.matmul(A,B) #returns Tensor of shape [10,11,100]
</code></pre>
Mapping of PyTorch tensors in device memory

username: Ajinkya_Ambatwar
cooked: <p>I was wondering how are pytorch tensors mapped in memory.<br>
For eg.</p>
<pre><code class="lang-auto">b = torch.rand(10)
</code></pre>
<p>Now if I do</p>
<pre><code class="lang-auto">&gt;&gt;&gt; id(b[0])
140426187280456
&gt;&gt;&gt; id(b[1])
140426187280456
</code></pre>
<p>Both of them are exactly same memory addresses?<br>
I was hoping given that the values inside <code>b</code> are <code>float32</code>, the <code>id</code>'s would be separated by 4 bytes.<br>
Am I wrong somewhere?<br>
I wanted to know how is a torch tensor mapped inside the memory?</p>
<p>Thank you!</p>
Concat tensors row column

username: Wolfy10
cooked: <p>Hey,</p>
<p>I am wondering if there’s a more efficient method using pytorch to concat 2 tensors together of different dimensions.</p>
<p>Suppose</p>
<p>Tensor A shape: 1, 2<br>
Tensor B shape: 4, 1</p>
<p>I want to concat these together to make</p>
<p>Tensor A_B: 4, 3</p>
<p>Where Tensor A 2nd dimension row is copied 4 times, with Tensor B’s respective second dimension value at the end of each row.</p>
<p>In other words, I want to do this:<br>
A: [ 1, 2 ]</p>
<p>B: [ 1  ],<br>
[ 2 ],<br>
[ 3 ],<br>
[ 4 ]</p>
<p>A_B: [ 1, 2, 1 ],<br>
[ 1, 2, 2 ],<br>
[ 1, 2, 3 ],<br>
[ 1, 2, 4 ],</p>
<p>I know how to do this with loops, but it takes time with large datasets, I am wondering if it’s possible to do with a built in function.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">A = torch.tensor([1, 2])
B = torch.tensor([[1], [2], [3], [4]])
C = torch.cat((A.expand(B.size(0), -1), B), dim=1)
</code></pre>
Weight initialization with specific distribution

username: emiled16
cooked: <p>I want to initialize the weights of my neural network with parameters samples from a specific distribution, that is not already present in nn.init module. Let’s say for example a beta distribution.</p>
<p>I saw that you can use the method described there: <a href="https://stackoverflow.com/questions/51125856/custom-weight-initialization-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">python - Custom weight initialization in PyTorch - Stack Overflow</a></p>
<p>But there you need to use the nn.init. What is an elegant way to initialize weights from a distribution that I specify?</p>

username: sio277
cooked: <p>You can customize it by referring to the initialization source code here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/init.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/init.py at master · pytorch/pytorch · GitHub</a>.</p>
Print('loaded {}, epoch {}'.format(model_path, checkpoint['epoch'])) KeyError: 'epoch'

username: creatives07
cooked: <pre><code>Traceback (most recent call last):
  File "train.py", line 98, in &lt;module&gt;
    main(opt)
  File "train.py", line 65, in main
    model, opt.load_model, trainer.optimizer, opt.resume, opt.lr, opt.lr_step)
  File "/home/sonic/sw/FairMOT/src/lib/models/model.py", line 37, in load_model
    print('loaded {}, epoch {}'.format(model_path, checkpoint['epoch']))
KeyError: 'epoch'
</code></pre>
<p>load_model def in model.py is this</p>
<pre><code>def load_model(model, model_path, optimizer=None, resume=False, 
               lr=None, lr_step=None):
  start_epoch = 0
  checkpoint = torch.load(model_path, map_location=lambda storage, loc: storage)
  print('loaded {}, epoch {}'.format(model_path, checkpoint['epoch']))
  state_dict_ = checkpoint['state_dict']
  state_dict = {}

  # convert data_parallal to model
  for k in state_dict_:
    if k.startswith('module') and not k.startswith('module_list'):
      state_dict[k[7:]] = state_dict_[k]
    else:
      state_dict[k] = state_dict_[k]
  model_state_dict = model.state_dict()

  # check loaded parameters and created model parameters
  msg = 'If you see this, your model does not fully load the ' + \
        'pre-trained weight. Please make sure ' + \
        'you have correctly specified --arch xxx ' + \
        'or set the correct --num_classes for your own dataset.'
  for k in state_dict:
    if k in model_state_dict:
      if state_dict[k].shape != model_state_dict[k].shape:
        print('Skip loading parameter {}, required shape{}, '\
              'loaded shape{}. {}'.format(
          k, model_state_dict[k].shape, state_dict[k].shape, msg))
        state_dict[k] = model_state_dict[k]
    else:
      print('Drop parameter {}.'.format(k) + msg)
  for k in model_state_dict:
    if not (k in state_dict):
      print('No param {}.'.format(k) + msg)
      state_dict[k] = model_state_dict[k]
  model.load_state_dict(state_dict, strict=False)

  # resume optimizer parameters
  if optimizer is not None and resume:
    if 'optimizer' in checkpoint:
      optimizer.load_state_dict(checkpoint['optimizer'])
      start_epoch = checkpoint['epoch']
      start_lr = lr
      for step in lr_step:
        if start_epoch &gt;= step:
          start_lr *= 0.1
      for param_group in optimizer.param_groups:
        param_group['lr'] = start_lr
      print('Resumed optimizer with start lr', start_lr)
    else:
      print('No optimizer parameters in checkpoint.')
  if optimizer is not None:
    return model, optimizer, start_epoch
  else:
    return model
</code></pre>
<p>Thank you.</p>

username: sio277
cooked: <p>Your <code>checkpoint</code> should include the key <code>epoch</code>. Try to save your model with something like this:</p>
<pre><code class="lang-auto">torch.save({'epoch': epoch, 'model_state_dict': model.state_dict(), ...}, PATH)
</code></pre>
PyTorch data loader bottleneck

username: weiqqi1028
cooked: <p>My model training is bottlenecked by IO, and I stream data from S3 using AWS wrangler. I only use 1 GPU for my model training. My machine has 8 GPUs, and I found when I run multiple training jobs, e.g. 4 jobs on my machine, the total IO of the machine increased 4 times. This demonstrated that the S3 throughput and network do not bottleneck my IO.</p>
<p>It seems that a single data loader has a bottleneck, and I am curious about where the bottleneck comes from. Is there a way to run 4 data loaders to load different part of the dataset simultaneously in one training job and interleave them together, to increase the IO performance of the data loading?</p>

username: omarfoq
cooked: <p>Hello</p>
<p>Normally DataLoader has an attribute <code>num_workers</code>, you can modify in order to speed-up data reading (by default it is 0)</p>

username: weiqqi1028
cooked: <p>I did tune num_workers, and tune the prefetch factor.</p>

username: omarfoq
cooked: <p>Hello,</p>
<p>In that case the only solution for you is to optimize the part of the code responsible for reading data (<code>Dataset.__getitem__</code>). If it’s possible load all the data into RAM at once, instead of reading from drive at each batch. Also note that when number of workers is different from 1, you have multiple processes loading data at once, so this corresponds to your initial question I guess</p>

username: weiqqi1028
cooked: <p>Thanks for reply. My question is why running multiple data loaders instances can increase the IO throughput, when I have already used multiple workers in data loader?</p>

username: omarfoq
cooked: <p>Hello,</p>
<p>Because that’s related to how money processes are working on loading data. If you have more processes loading data, they will be fast. But now, I am not sure if I understand your question</p>

username: weiqqi1028
cooked: <p>This is my data loader setting:</p>
<pre><code class="lang-auto">train_data_loader = DataLoader(
        train_dataset,
        batch_size=batch_size
        num_workers=os.cpu_count()
        collate_fn=model.get_feature_to_input_fn(),
        prefetch_factor=args.prefetch_factor,
    )
</code></pre>
<p>Inside the train_dataset iter function, I also used multiple threads to stream data from s3. The observation is, with this setup my IO is k/min, however if I create 4 data loaders with this setup, my IO is 4k/min. No matter what parameters I tune for a single data loader, the single data loader performance can not reach 4k/min.</p>
How to compute the grad of input with respect to loss

username: 111494
cooked: <p>I have a puzzle about the torch autograd. When I build a model, and write code:</p>
<pre><code class="lang-auto">x.requires_grad = True
pred = model(x)
pred.sum().backward()
print(x.grad)
</code></pre>
<p>I can get the grad of x. I want to go forward from here. Is there a method that I can get  d(x.sum())/d(pred) using the following code?</p>
<pre><code class="lang-auto">x.requires_grad = True
pred = model(x)
pred.requires_grad = True
pred.sum().backward(retain_graph=True)
x.sum().backward()
print(pred.grad)
</code></pre>
<p>I have tried this, but the pred.grad is None. and x.grad.grad_fn is None. Thanks for your help <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: soulitzer
cooked: <p>If you replace <code>pred.requires_grad=True</code> with <code>pred.retain_grad()</code> it should work.</p>
<p>This might be helpful: <a href="https://stackoverflow.com/questions/45988168/why-does-autograd-not-produce-gradient-for-intermediate-variables" class="inline-onebox" rel="noopener nofollow ugc">pytorch - Why does autograd not produce gradient for intermediate variables? - Stack Overflow</a></p>
TorchScript JIT can't handle Tuple[Tensor, Tensor] + Tuple[Tensor] statement

username: CaoZhong
cooked: <p>How can I workaround the errors?</p>
<p>RuntimeError:<br>
Arguments for call are not valid.<br>
The following variants are available:</p>
<p>aten::add.Tensor(Tensor self, Tensor other, *, Scalar alpha=1) → (Tensor):<br>
Expected a value of type ‘Tensor’ for argument ‘self’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>aten::add.Scalar(Tensor self, Scalar other, Scalar alpha=1) → (Tensor):<br>
Expected a value of type ‘Tensor’ for argument ‘self’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>aten::add.out(Tensor self, Tensor other, *, Scalar alpha=1, Tensor(a!) out) → (Tensor(a!)):<br>
Expected a value of type ‘Tensor’ for argument ‘self’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>aten::add.t(t[] a, t[] b) → (t[]):<br>
Could not match type Optional[Tuple[]] to List[t] in argument ‘a’: Cannot match List[t] to Optional[Tuple[]].</p>
<p>aten::add.str(str a, str b) → (str):<br>
Expected a value of type ‘str’ for argument ‘a’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>aten::add.int(int a, int b) → (int):<br>
Expected a value of type ‘int’ for argument ‘a’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>aten::add.float(float a, float b) → (float):<br>
Expected a value of type ‘float’ for argument ‘a’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>aten::add.int_float(int a, float b) → (float):<br>
Expected a value of type ‘int’ for argument ‘a’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>aten::add.float_int(float a, int b) → (float):<br>
Expected a value of type ‘float’ for argument ‘a’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>aten::add(Scalar a, Scalar b) → (Scalar):<br>
Expected a value of type ‘number’ for argument ‘a’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>add(float a, Tensor b) → (Tensor):<br>
Expected a value of type ‘float’ for argument ‘a’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>add(int a, Tensor b) → (Tensor):<br>
Expected a value of type ‘int’ for argument ‘a’ but instead found type ‘Optional[Tuple[]]’.</p>
<p>The original call is:<br>
File “…/transformers/src/transformers/models/bert/modeling_bert.py”, line 546<br>
for i, layer_module in enumerate(self.layer):<br>
if output_hidden_states:<br>
all_hidden_states = all_hidden_states + (hidden_states,)<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;— HERE</p>
<pre><code>        layer_head_mask = head_mask[i] if head_mask is not None else None</code></pre>

username: SplitInfinity
cooked: <p>It seems that the <code>Tuple[Tensor, Tensor]</code> is actually <code>Optional[Tuple[Tensor, Tensor]]</code>; you need to refine it so that TorchScript can safely conclude it is not none:</p>
<pre><code class="lang-auto">if all_hidden_states is not None:
   ...
</code></pre>
<p>or</p>
<pre><code class="lang-auto">assert all_hidden_states is not None
</code></pre>
Cross Compile for Hexagon

username: Kittu28
cooked: <p>How can i cross compile for the Hexagon backend.? Can you please mention all the necessary steps.<br>
Thank you.</p>

username: jfix
cooked: <p>Hi, we do not have any Hexagon backend. Are you asking about how to develop a backend?</p>

username: Kittu28
cooked: <p>I see <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md#cross-compile-a-bundle-for-a-specific-architecture" rel="noopener nofollow ugc">here</a> about cross compilation . How can i make that work? Where should i integrate the Hexagon files for it?</p>

username: jfix
cooked: <p>You can target any LLVM architecture that is supported by the version of LLVM that you’re compiling with alongside Glow. You’d need to figure out the name of the <code>-target</code> used by LLVM, since we just forward that down to LLVM I believe. You should be able to do that with the CPU backend. If you want to add your own specialized kernels better for the architecture then you’d need to add your own backend similar to the CPU backend which is an LLVM based backend, and then add your own <code>libjit</code> kernels.</p>

username: Kittu28
cooked: <p>I used -target=hexagon with backend=CPU ,I got Aborted(Core dumped) error. What do i need to do for that to work…?</p>

username: jfix
cooked: <p>It’s hard to know what the issue is just from that. I’d suggest running in debug mode to see if you get a better error message or assertion failure. Could also include running in a debugger like gdb to understand where it’s aborting.</p>
Iterative Pruning: LeNet-300-100

username: grid_world
cooked: <p>I am trying to implement iterative pruning algorithm which is: train a model, prune p% of smallest weights per layer, re-train the pruned model and repeat. For experiment purposes, I am using LeNet-300-100 neural network on MNIST.</p>
<p>The code can be accessed <a href="https://github.com/arjun-majumdar/Lottery_Ticket_Hypothesis-TensorFlow_2/blob/master/LeNet_300_100-Iterative_Pruning.ipynb" rel="noopener nofollow ugc">here</a></p>
<p>Within the function “train_with_grad_freezing(model, epoch)”, I am using the following lines of code for freezing the pruned weights by making their computed gradients equal to 0:</p>
<pre><code class="lang-auto">for layer_name, param in model.named_parameters():
    if 'weight' in layer_name:
        tensor = param.data.cpu().numpy()
        grad_tensor = param.grad.data.cpu().numpy()
        grad_tensor = np.where(tensor == 0, 0, grad_tensor)
        param.grad.data = torch.from_numpy(grad_tensor).to(device)
</code></pre>
<p>The first time I train the model, the code works fine after which I prune the layers by using the code:</p>
<pre><code class="lang-auto"># Prune 15% of smallest magnitude weights in FC layers and 10% in output layer-
pruned_d = prune_lenet(model = best_model, pruning_params_fc = 15, pruning_params_op = 10)

# Initialize and load pruned Python3 dict into a new model-
pruned_model = LeNet300()
pruned_model.load_state_dict(pruned_d)
</code></pre>
<p>However, on re-training this pruned model, the training metric is stuck for these values:</p>
<blockquote>
<p>training loss = 0.0285, training accuracy = 99.04%, val_loss = 0.0910 &amp; val_accuracy = 97.68%</p>
</blockquote>
<p>What’s going wrong?</p>
Fetch model attribute by name in C++

username: Linux-cpp-lisp
cooked: <p>Hi all,</p>
<p>Suppose I have a module like:</p>
<pre><code class="lang-python">In [1]: import torch

In [2]: class M(torch.nn.Module):
   ...:     thing: str
   ...:     def __init__(self):
   ...:         super().__init__()
   ...:         self.thing = 'hi'
   ...:         self.other = lambda x: x* 2
   ...: 

In [3]: m = M()

In [4]: ms = torch.jit.script(m)

In [5]: ms.save("test.pth")
</code></pre>
<p>If I load <code>test.pth</code> in C++ using <code>libtorch</code>, once I have a <code>torch::jit::script::Module</code> (as in the <a href="https://pytorch.org/tutorials/advanced/cpp_export.html#step-3-loading-your-script-module-in-c" rel="noopener nofollow ugc">tutorial</a>), how can I access the <code>thing</code> string by name as some kind of typical C++ string object or pointer?</p>
<p>Would the process be the same for an attribute annotated <code>thing2: float</code> in Python?</p>
<p>Thanks!</p>
How to get Tensor size from torch::jit::module?

username: Stonepia
cooked: <p>Hi there,<br>
How could I go through the module and print its tensor size?</p>
<p>For example, I want to do the following thing:</p>
<pre><code class="lang-auto">void print_op_input(torch::jit::module &amp; mod){
  auto fwd_mtd = mod.get_method("forward");
  for (torch::jit::Node * nd : fwd_method.graph()-&gt;nodes()){
    if (nd-&gt;kind().is_aten()){
      auto inputs = nd-&gt;inputs();
    
      // Question : How to access the input?
      // The inputs here are type intArrayRef&lt;Value *&gt;
    }
  }
}

</code></pre>
<p>The question is , how could I access the input? It is <code>Value*</code>, not <code>IValue</code>. How could I access its true stored value?</p>

username: hidefromkgb
cooked: <p>I think what you need is <a href="https://github.com/pytorch/pytorch/blob/f9ca0d87a7cd2afb3ae5866acf7a77af54c5ab91/torch/csrc/jit/ir/constants.h#L49" rel="noopener nofollow ugc"><code>toIValue()</code></a>, or even <code>IValue(toIValue())</code> that removes <code>c10::optional&lt;&gt;</code> from <code>c10::optional&lt;IValue&gt;</code>.<br>
No idea why we even need this whole <code>c10::optional&lt;&gt;</code> construct when <code>IValue</code> can simply return <code>.isNone() == true</code> — which it actually does after the proposed conversion if the value was empty.</p>

username: SplitInfinity
cooked: <p>The solution proposed by <a class="mention" href="/u/hidefromkgb">@hidefromkgb</a> should work, but keep in mind that <code>toIValue(const Value* v)</code> will return an <code>IValue</code> only for <code>Value*</code>s that point to constants.</p>
Where can I find `torch.ops.quantized.add`'s c++ code?

username: Jonson
cooked: <p>I want to know what happened when I call <code>torch.ops.quantized.add</code> API, but I have no idea where the corresponding c++ code is?</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/jonson">@Jonson</a> , you can find the code for quantized add here: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qadd.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/qadd.cpp at master · pytorch/pytorch · GitHub</a></p>
Pretrained resnet to multilabeling classification

username: Tomash
cooked: <p>Hi,<br>
I have a question. I operate on the dateset where every image can be classify to more than one class. The input (label) for every element looks like a vector: [0,1,0,0,1] what means that some element belongs to two different classes. I would like to use pretrained resnet for this task but I am not sure how to implement it. Which loss function is the best for that kind of problem?</p>

username: aza
cooked: <p>Hello, crossentropyloss is a good place to start: <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.8.1 documentation</a></p>
<p>Depending on the level of noise in your dataset however you may need to utilize a more customized loss function</p>

username: Tomash
cooked: <p>Yep, but what about resnet? How make it to return 1x6 output?</p>
Draw colored mask on original image in Detectron2

username: demonvferrari
cooked: <p>Hi,</p>
<p>There is a function to draw binary mask, but there is no function to draw mask on the colored image. How to draw mask on the original input color image?</p>
<pre><code class="lang-auto">output = predictor(input_image)        
V = Visualizer(input_image[:, :, ::-1], 
MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale = 1.2)
mask=output["instances"].get("pred_masks")
mask=mask.to("cpu")
V.draw_instance_predictions(output["instances"].to("cpu"))
for m in mask:
    V.draw_mask(m)  # there is no function to draw colored mask

V = V.get_output()
cv2.imwrite("frame%d.jpg" % count, V.get_image()[:, :, ::-1]) 
</code></pre>
<p>Thank you</p>
A question about model training

username: flfl1111
cooked: <p>In fact, the model I built is a regression model rather than a classification model.<br>
The main issue is the accuracy of training set is far lower than  that of  testing set.<br>
As is shown in the figures below.<br>
the first figure is the result of training set(the red line  is  true data and the green is the predictive value)<br>
the second is the result of testing set(the blue line is true data and the green is the predictive value)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6db4647b056c33c8a2637e59705887a75617664c.png" alt="train" data-base62-sha1="fEux4vRxtxirN7jj5aNEIzSGCgA" width="406" height="500"><br>
the RMSE of traiing set:39.4701<br>
the RMSE of testing set:6.1752</p>

username: aza
cooked: <p>without more information it’s hard to decipher where the issue lies, a few thoughts of what may be causing this:</p>
<ol>
<li>training set may feature noisy samples that make convergence in training a tough task</li>
<li>test set may feature samples that are not representative of the noise in the training set</li>
</ol>
<p>Is the model showing progress in regards to the loss and metric you are using while training?</p>
One hot encoding for multi label classification using BCEWithLogitsLoss() loss

username: IrfanUmar
cooked: <p>I am using resnet18 with BCEWithLogitsLoss()<br>
and i am encoding my labels using<br>
<code>    y_onehot = nn.functional.one_hot(labels, num_classes=3)</code><br>
<code>     y_onehot = y_onehot.float()</code></p>
<p>Which is I think not true for multi label data.<br>
How should I encode my labels to get multi labels</p>

username: KFrank
cooked: <p>Hello Irfan!</p>
<aside class="quote no-group quote-modified" data-username="IrfanUmar" data-post="1" data-topic="79262" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/5e9695/48.png" class="avatar"> IrfanUmar:</div>
<blockquote>
<p><code>    y_onehot = nn.functional.one_hot(labels, num_classes=3)</code></p>
<p>How should I encode my labels to get multi labels</p>
</blockquote>
</aside>
<p>What is the <code>shape</code> of the output of your network (that you will<br>
pass as the <code>input</code> to your <code>BCEWithLogitsLoss</code> loss function)?</p>
<p>What is the <code>shape</code> of <code>labels</code>, and what are typical values of the<br>
<code>labels</code> tensor, and what do they mean conceptually?</p>
<p>Best.</p>
<p>K. Frank</p>

username: Kai_Middlebrook
cooked: <p>One workaround I use for multi-label classification is to sum the one-hot encoding along the row dimension.</p>
<p>For example, let’s assume there are 5 possible labels in a dataset and each item can have some subset of these labels (including all 5 labels). The code to one-hot encode an item’s <code>labels</code> would look like this:</p>
<pre><code class="lang-auto">labels = torch.LongTensor([1, 2, 4])
y_onehot = nn.functional.one_hot(labels, num_classes=5)
y_onehot = y_onehot.sum(dim=0).float()
</code></pre>
<p>Hope this helps!</p>

username: dazzle-me
cooked: <p>Thanks! If was easier than I thought it would be.</p>
Dynamically changing parameter sizes

username: Tijmen_Blankevoort
cooked: <p>Hey all!</p>
<p>I’m trying to prune neurons from a feed forward layer in Pytorch while training.<br>
Thus after an epoch, I remove row from the weight matrix of layer 2, and I remove a column and bias from the matrix of layer 1.</p>
<p>Something along the lines of:<br>
self.W = Parameter(self.W[inverse_mask, :].data)</p>
<p>This operation works, and calculation happens as planned.</p>
<p>However, now self.W does not get updated anymore? Does anyone have an idea how I can ‘reregister’ the parameter, or change the data of the parameter on runtime in such a way that it works?</p>
<ul>
<li>I tried to change the data itself, but that runs into gradient problems from which the sizes are stored.</li>
<li>I tried to register the parameter in <strong>init</strong> as self.register_parameter(‘W’, None), but that didn’t help.</li>
</ul>
<p>Cheers,<br>
Tijmen</p>

username: Tijmen_Blankevoort
cooked: <p>Small update. I figured out that the gradient problem is mainly because of Adam storing momentum parameters that should also decrease in size. However, hacking around with that seems suboptimal. Any suggestions?</p>

username: smth
cooked: <p>you will have to reconstruct the optimizer if you change the shapes of the parameters. there’s no real workaround.</p>
<p>Alternatively, you can simply slice self.W in the <code>forward</code> function and keep it’s original shape intact.</p>
<p>Like:</p>
<pre><code class="lang-auto">out = torch.mm(input, self.W[inverse_mask, :]) # just an example
</code></pre>
<p>Here you dont repackage or change self.W, you just enforce a mask at runtime.</p>

username: roee
cooked: <p>Is there any reason not to just reinitialize the optimizer?<br>
(except for the loss of momentum)</p>

username: dipfit
cooked: <p>Assume we use an optimizer like SGD, which has no shape-dependent parameters, then it should be no problem, right?</p>
convNd pick 2nd instead of 1st element for stride 2

username: sans_sense
cooked: <p>When we use stride, say stride 2, it will pick the 0th, 2nd and so on, is there a way to instead pick 1st, 3rd and so on.</p>
<p>Is there a parameter to conv1d or any other way that will help me get with stride 2 same results as</p>
<pre><code class="lang-auto">F.conv1d(t_input, t_weight, padding=5, stride=1)[0,0,1:10:2]
</code></pre>
<p>Actual Stride invocation that I am using</p>
<pre><code class="lang-auto">F.conv1d(t_input, t_weight, padding=5, stride=2)[0,0,0:5]
</code></pre>
<p>The default result that I get is same as</p>
<pre><code class="lang-auto">F.conv1d(t_input, t_weight, padding=5, stride=1)[0,0,0:10:2]
</code></pre>
<p>I am porting a TF model and somehow the conv there seems to be picking the 1::2 element and torch seems to be picking 0::2. Wanted both model implementations to give same result.</p>

username: ptrblck
cooked: <p>Would it work, if you slice the input starting at <code>index1</code> in PyTorch? This might not be the most convenient approach, but I don’t think the start index is easily changeable in the backend.</p>

username: sans_sense
cooked: <p>Thanks a lot for the reply, wanted to get an expert’s opinion. If there are no params, will try slicing and other options. Would be great if someone can confirm that the code at least from a intel cpu flow is<br>
F.conv1d → ATen/native/Convolution.cpp → (assuming mkl) -&gt;ATen/native/mkldnn/Conv.cpp → ideep::convolution_forward .</p>
Concatenation of diff tensors

username: Merwa_Mehmood
cooked: <p>how to concatenate weights of different layers pf different dimensions e.g<br>
torch.Size([64, 256, 4, 4])<br>
torch.Size([1, 256, 1, 1])<br>
torch.Size([256, 256, 3, 3])…</p>

username: nivesh_gadipudi
cooked: <p>You can use <code>torch.cat(tensors, dim)</code> to concat two or more tensors. However, torch.cat expects the tensors to be of same shape, except in cat dimensions.</p>
<p>one way you could concat the different dimensions is to pad the tensor:</p>
<pre><code class="lang-auto">a = torch.zeros(64, 256, 4, 4)
b = torch.zeros(1, 256, 1, 1)
b = torch.nn.functional.pad(b, (2, 1, 2, 1), 'constant', 0)
e = torch.cat((a, b), dim = 0)

Out: torch.Size([65, 256, 4, 4])
</code></pre>
<p>Other way would be:</p>
<pre><code class="lang-auto">a = torch.zeros(64, 256, 4, 4)
a = torch.reshape(a, (-1, 256, 1, 1))
b = torch.zeros(1, 256, 1, 1)

c = torch.cat((a, b), dim = 0)

Out: torch.Size([1025, 256, 1, 1])
</code></pre>
<p>It entirely depends on the application or the result you are expecting.</p>
<p>Hope this helps</p>

username: Merwa_Mehmood
cooked: <p>Thanks it worked <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>
How to do this operation fast?

username: KaiHoo
cooked: <p>I have some video clips with different length. The video clip length is stored in seq_len = [T_1, T_2, …, T_n]. The video clips are stored in a tensor of shape [\sum T_i, C, H, W].</p>
<p>Now I want to converted it into shape [B, C, \max T_i, H, W], i.e., pad the shorter video clips with zero and concatenate them into the shape that can be sent to nn.Conv3d.</p>
<p>Here is an implementation using loops:</p>
<pre><code class="lang-auto">def pack_x(x, seq_len):
    T = seq_len.max().item()
    _, C, H, W = x.shape
    x = x.transpose(0, 1)
    x = torch.split(x, seq_len.tolist(), dim=1)
    
    pack_x = []
    for clip in x:
        clip = F.pad(clip, [0,0,0,0,0,T-clip.size(1)])
        pack_x.append(clip.unsqueeze(0))
    x = torch.cat(pack_x)
    return x


seq_len = torch.randint(5, 10, size=[5])
inputs = torch.randn(seq_len.sum(), 512, 28, 28)
y = pack_x(inputs, seq_len)
</code></pre>

username: Naruto-Sasuke
cooked: <p>Override the function of <code>collate_fn</code> in DataLoader,</p>
<pre><code class="lang-python">torch.utils.data.DataLoader(dataset=train_dataset, batch_size=setting.batch_size, num_workers=opt.nThreads, collate_fn=pack_x_new)
</code></pre>
<p>You should do something to update <code>pack_x</code> to <code>pack_x_new</code> which consumes the data from <code>__get_item__()</code> function.</p>

username: KaiHoo
cooked: <p>Hi <a class="mention" href="/u/naruto-sasuke">@Naruto-Sasuke</a> I need to do this in the middle of model forward.</p>