akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Patch wise training with high resolution images

sebdei: <p>I got high resolution images about 5000x5000 of histology slides. I want to train a cycleGAN (or others) to generate images from one domain to another. Obviously, feeding the whole image into the network is impossible without downscaling to rougly 400x400 pixels. Therefore I want to train the network on cropped patches.</p>
<p>For now I want to use FiveCrop to get the crops</p>
<pre><code class="lang-auto">transform = [
    transforms.Resize(int(5000 * 1.12), Image.BICUBIC),
    transforms.FiveCrop(opt.img_size),
    transforms.Lambda(lambda crops: torch.stack(
        [transforms.ToTensor()(crop) for crop in crops])),
    transforms.Lambda(lambda tensors: torch.stack(
        [transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))(tensor) for tensor in tensors]))
]
</code></pre>
<p>Further, on the batch level I split the batches into patches and feed these into the network. However, after the second iteration of the inner loop I get CUDA mem allocation errors.</p>
<pre><code class="lang-auto">prev_time = time.time()
for epoch in range(opt.epoch, opt.n_epochs):
    for i, batch in enumerate(dataloader):
        assert batch['from'].size() == batch['to'].size()

        bs, ncrops, channels, height, width = batch['from'].size()

        loss_G = 0

        for patch_from, patch_to in zip(batch['from'].view(-1, bs, channels, height, width),
                                        batch['to'].view(-1, bs, channels, height, width)):
            real_from = Tensor(patch_from.cuda())
            real_to = Tensor(patch_to.cuda())

            # Adversarial ground truths
            valid = Tensor(
                np.ones((real_from.size(0), *D_A.output_shape)))
            fake = Tensor(
                np.zeros((real_from.size(0), *D_A.output_shape)))

            # ------------------
            #  Train Generators
            # ------------------

            G_AB.train()
            G_BA.train()

            optimizer_G.zero_grad()

            # Identity loss
            loss_id_A = criterion_identity(G_BA(real_from), real_from)
            loss_id_B = criterion_identity(G_AB(real_to), real_to)

            loss_identity = (loss_id_A + loss_id_B) / 2

            # GAN loss
            fake_to = G_AB(real_from)
            loss_GAN_AB = criterion_GAN(D_B(fake_to), valid)
            fake_from = G_BA(real_to)
            loss_GAN_BA = criterion_GAN(D_A(fake_from), valid)

            loss_GAN = (loss_GAN_AB + loss_GAN_BA) / 2

            # Cycle loss
            recov_A = G_BA(fake_to)
            loss_cycle_A = criterion_cycle(recov_A, real_from)
            recov_B = G_AB(fake_from)
            loss_cycle_B = criterion_cycle(recov_B, real_to)

            loss_cycle = (loss_cycle_A + loss_cycle_B) / 2

            # Total loss
            loss_G = loss_G + loss_GAN + opt.lambda_cyc * \
                loss_cycle + opt.lambda_id * loss_identity

        loss_G.backward()
        optimizer_G.step()
</code></pre>
<p>Can someone explain how I’m able to feed patches to the NN? Further any other recommendation, how to train GANs on high resolution images?</p>
<p>greets</p>
Pytorch nightly vs Stable

yoad.tewel: <p>Hey <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>So I saw this post: <a href="https://discuss.pytorch.org/t/rtx-3070-slower-with-mixed-precision-autocast-in-pytorch-1-7/105343" class="inline-onebox">Rtx 3070 slower with mixed precision autocast in pytorch 1.7</a>, where <a class="mention" href="/u/ptrblck">@ptrblck</a> said:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="105343">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/rtx-3070-slower-with-mixed-precision-autocast-in-pytorch-1-7/105343/2">Rtx 3070 slower with mixed precision autocast in pytorch 1.7</a></div>
<blockquote>
<p>cudnn8.0.5 ships with the updated heuristics for the 3090 and cudnn8.1.x will cover the complete 30xx series. You could try out the nightly PyTorch build, which already uses cudnn8.0.5 and check, if the performance is improved.</p>
</blockquote>
</aside>
<p>So I have some questions about nightly build:</p>
<ol>
<li>Which version of pytorch is the nightly? (If stable is 1.7.0, what is the version number of nightly)</li>
<li>How stable is nightly? I guess its less stable than the <strong>stable</strong> version, but should I expect problems working with it? (I work with CNNs and GANS, nothing speical)</li>
<li>When I install the nightly build with pip/conda, all the cuda and cudnn stuff are inside the binaries? And if so, the only thing I need is nvidia drivers?</li>
<li>Are there any specifications on the nightly build, like which cuda and cudnn it uses, when was the latest version released etc?</li>
</ol>
<p>Thank you!</p>

ptrblck: <ol>
<li>The nightly release is tagged with the date e.g. as: <code>1.8.0.dev20201109</code>
</li>
<li>You might see more issues, as the nightly will be rebuild every day from the current master branch. That being said, I don’t have the feeling that the nightly is terribly unstable.</li>
<li>Yes</li>
<li>You can use the install command from the <a href="https://pytorch.org/get-started/locally/">website</a> (same as for the stable release) and specify which <code>cudatoolkit</code> you would like to use.</li>
</ol>

yoad.tewel: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks!</p>
<p>So another quesiton, if the current nightly is 1.8.0, are there binaries of earlier nightly versions which are maybe more stable, like 1.7.1?</p>

ptrblck: <p>The nightly is not <code>1.8.0</code>, which would be a future stable release, but <code>1.8.0.devDATE</code>.<br>
The stable releases are not created by tagging a specific nightly release as stable, but a versioned branch is created (e.g. at the moment for <code>1.7.1</code>), which gets specific commits/fixes etc., while the master branch and the nightlies keep moving.</p>
BatchNorm and zero/nan input: Different behaviour of eval() and train() modes even after model is broken

egundogdu: <p>Most people struggling nans at the output  know that when you feed nan or zero input to the model which has layers such as batchnorm, then the model may become broken. During the training, if you do not do backward pass, the model does not break down. However, since you already did a forward pass with these undesired inputs once, the eval mode outputs nan while the train mode does not. Any explanation on this?</p>

ptrblck: <p>The running stats of batchnorm layers (<code>running_mean</code> and <code>running_var</code>) will be poisoned by the invalid values and will thus yield invalid outputs during <code>model.eval()</code> forward passes.</p>

egundogdu: <p>Thanks for the reply. This was what I guessed. Apart from not feeding zero/nan inputs, are there any guard rails to use such as being able to automatically ignore such samples? Or a mechanism to avoiding updating the <code>running_mean</code> or <code>running_var</code>?</p>

ptrblck: <p>I’m not aware of any other approaches that to make sure these invalid values are filtered out (e.g. via <code>torch.isfinite(input)</code>).</p>
There is no pytorch_android_fbjni-release.aar but logs show " build is successful"

ZackPashkin: <p>Could you please help resolve following issue. I’m trying to make custom pytorch android build but in the end I have only two .aar files.  There is no pytorch_android_fbjni-release.aar . Can’t find any reason, in logs.</p>

hubbup: <p>It seems that the part building fbjni was removed.<br>
please refer to the PR:<br>
</p><aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/40058/" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Pull Request">
    <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/pull/40058" target="_blank" rel="noopener nofollow ugc">[android][fbjni] Test_app and Readme update with the recent fbjni dep state</a>
    </h4>

    <div class="branches">
      <code>pytorch:gh/IvanKobzarev/41/base</code> ← <code>pytorch:gh/IvanKobzarev/41/head</code>
    </div>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-06-15" data-time="22:16:13" data-timezone="UTC">10:16PM - 15 Jun 20 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/IvanKobzarev" target="_blank" rel="noopener nofollow ugc">
          <img alt="IvanKobzarev" src="https://avatars3.githubusercontent.com/u/6638825?v=4" class="onebox-avatar-inline" width="20" height="20">
          IvanKobzarev
        </a>
      </div>

      <div class="lines" title="1 commits changed 3 files with 22 additions and 24 deletions">
        <a href="https://github.com/pytorch/pytorch/pull/40058/files" target="_blank" rel="noopener nofollow ugc">
          <span class="added">+22</span>
          <span class="removed">-24</span>
        </a>
      </div>
    </div>

  </div>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Make type mistake

xiashu: <p>when i run my project,there show that “an interger is required(got type list)”,i know maybe i need to change my data type,but yesterday there is no wrong,I don’t know how to do that</p>
<p>the error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90c8190931d3d6f812955ea09496a3bc656733e8.png" data-download-href="https://discuss.pytorch.org/uploads/default/90c8190931d3d6f812955ea09496a3bc656733e8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90c8190931d3d6f812955ea09496a3bc656733e8.png" alt="image" data-base62-sha1="kENsIUy6PJ2SAMfTDcRhOxHJdLG" width="690" height="152" data-dominant-color="131313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">990×219 4.22 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code>def flatten_list_of_lists(list_of_lists):


list_of_lists = [[]] + list_of_lists                               
offsets = np.cumsum([len(x) for x in list_of_lists])[:-1]                
flatten = sum(list_of_lists[1:], [])                            

return flatten, offsets*emphasized text*
</code></pre>
<p>========================================================================</p>
<pre><code>token_ids = [l + m + r if len(l) + len(r) &gt; 0 else [self.prerank_model.word_voca.unk_id] for l, m, r in zip(lctx_ids, ment_ids, rctx_ids)]                            

entity_ids = [m['cands'] for m in content]                                                    

entity_ids = Variable(torch.LongTensor(entity_ids))

entity_mask = [m['mask'] for m in content]                                                            
entity_mask = Variable(torch.FloatTensor(entity_mask))
token_ids, token_offsets = utils.flatten_list_of_lists(token_ids)                                
token_offsets=token_offsets/1.0

token_offsets = Variable(torch.LongTensor(token_offsets))

token_ids = Variable(torch.LongTensor(token_ids))
print("token_ids",type(token_ids))
</code></pre>

Usama_Hasan: <p>Hy <a class="mention" href="/u/xiashu">@xiashu</a>, it would be really great if you the complete error and the code here.</p>

xiashu: <p>I have put the code and error on it,Can you help me to check it</p>
How to adjust the batch size for single vs. multiple GPU training?

snowe: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Let’s assume I train a model with a batch size of 64 on a single GPU. Now I want to train the model on multiple GPUs using <code>nn.DataParallel</code>. If I keep all my parameters the same, I expect the two experiments to yield the same results.</p>
<p>But how do I have to specifiy the batch size to get the same results? Do I pass 64 and then every GPU (let’s assume I have 4) gets a batch size of 16 (64 / 4) or do I set the batch size to 4 * 64 and then every GPU gets 64? The <a href="https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html?highlight=nn%20dataparallel#torch.nn.DataParallel" rel="noopener nofollow ugc">documentation</a> of <code>nn.DataParallel</code> says it splits along the batch dimension, so I would assume that I have to set the batch size to 4 * 64 to get the same results. This goes in hand with <a href="https://discuss.pytorch.org/t/batch-size-of-dataparallel/34117">this</a> but then I find <a href="https://discuss.pytorch.org/t/accuracy-difference-on-multi-gpu-with-nn-dataparallel/65481">this</a> which says I should keep the batch size at 64…</p>
<p>Can anyone clarify this topic? Any help is very much appreciated!</p>
<p>All the best<br>
snowe</p>
Cannot build iOS benchmark app

windmaple: <p>I’m closely following the instructions <a href="https://pytorch.org/tutorials/recipes/mobile_perf.html#ios-benchmarking-setup" rel="noopener nofollow ugc">here</a>, but hit this error in Xcode 12.2 when trying to run the app on iPhone XR (iOS 14.2):</p>
<blockquote>
<p>/Users/wind/Desktop/project-src/pytorch/build_ios/install/lib/libXNNPACK.a(1x8inc-minmax-aarch64-neonfp16arith-ld64.S.o), building for iOS, but linking in object file built for macOS, file ‘/Users/wind/Desktop/project-src/pytorch/build_ios/install/lib/libXNNPACK.a’ for architecture arm64</p>
</blockquote>
<p>Looks like wrong library is built. Anybody has any idea what needs to be changed?</p>
RuntimeError: The size of tensor a (132) must match the size of tensor b (64) at non-singleton dimension 2

Muhammad_Izaz: <p>RuntimeError: The size of tensor a (132) must match the size of tensor b (64) at non-singleton dimension 2<br>
my output size is 1,2,132 *132</p>

ptrblck: <p>This error is raised, if an operation cannot be applied to tensors having a different shape.<br>
In this case <code>dim2</code> seems to be different, so you would need to check the shape of both tensors and make sure they are in the expected shape (depends on the operation).<br>
Feel free to add more information to this thread in case you get stuck.</p>

Muhammad_Izaz: <p>ii am getting this error now please help.<br>
model  = UNet()<br>
criterion = torch.nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adam(model.parameters() , lr=0.0001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)<br>
num_epochs=7<br>
total_step = len(test_data)<br>
loss_list = []<br>
acc_list = []<br>
lbls=[]<br>
pred=[]<br>
for epoch in range(num_epochs):<br>
for i ,(images, labels) in enumerate(test_data):<br>
outputs = model(images)</p>
<pre><code>    #preds = torch.argmax(outputs,dim=1)
    preds = outputs.unsqueeze(dim=1)
    labels=labels.unsqueeze(dim=-1)
    #outputs= outputs.argmax(outputs,dim=1)###
    #loss = F.nll_loss(pred, labels)

    loss = criterion(preds.float() , labels)
    loss_list.append(loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    total = labels.size(0)
    _, predicted = torch.max(preds, 1)
    correct = (predicted == labels).sum().item()
    acc_list.append(correct / total)
    lbls+=labels
    pred+=predicted
    if (i + 1) % 1 == 0:
        print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
              .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                      (correct / total) * 100))
</code></pre>
<p>“”“ValueError: Expected target size (2, 2, 132, 132), got torch.Size([2, 1])”""</p>
Mixed Precision to modify Models for Mixed Precision Training

Stanley_C: <p>I’ve currently written a model that works on float32. However, I’d like to try mixed precision training due to uses not having enough memory. However, after following this page: <a href="https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html</a>, and adding the grad scaler and with autocast, I noticed that <strong>no</strong> memory was being saved, and when checking params within the autocast while section of my code, I noticed it was all fp32. Are there any other steps I need to take to modify the model code to work with mixed precision, and if necessary, could you link resources with steps on how to modify my model?</p>

ptrblck: <p>The parameters will still be stored on FP32 and the memory saving might come from the activations, which could be stored in FP16 (if the operation is save to be used in FP16).<br>
The <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">amp example</a> would be a good starter and shows how to use <code>autocast</code>.</p>

Stanley_C: <p>Thanks for your response! If I wrote a custom loss function that only takes fp.32, should I use the normal backwards and step instead of the scaling the loss?</p>

ptrblck: <p>No, the loss scaling is applied to avoid underflows in the gradient calculation, so you should still use it.</p>

Stanley_C: <p>Should I convert the loss function’s outputs (which are in fp32) back down to fp16, then apply scaling?</p>

ptrblck: <p>No, you can keep them in FP32.</p>
How to call a C++ file CUDA File in Pytorch and Pass Tensors as Argument and Return Tensors?

Stanley_C: <p>If I rewrite custom data preprocessing or loss CUDA scripts in a C++ file and compile it with NVCC, how can I pass tensors in, and how can I return arrays from the script as tensors? I was also wondering how the tensors were represented in Pytorch, and it is possible to represent them as standard array?</p>

InnovArul: <p>Perhaps you can look into these:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" target="_blank" rel="noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" target="_blank" rel="noopener">Custom C++ and CUDA Extensions — PyTorch Tutorials 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/extension-cpp" target="_blank" rel="noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/pytorch/extension-cpp" target="_blank" rel="noopener">pytorch/extension-cpp</a></h3>


  <p><span class="label1">C++ extensions in PyTorch. Contribute to pytorch/extension-cpp development by creating an account on GitHub.</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Model move to device - gpu

sujay: <p>In the documentation,</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" target="_blank" rel="noopener nofollow ugc">Optional: Data Parallelism — PyTorch Tutorials 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It is mentioned that the tensors must be assigned to a new variable when using to device.<br>
mytensor = my_tensor.to(device)</p>
<p>Should we not do the same for the model?</p>
<p>Like,<br>
model = model.to(device)</p>
<p><a class="mention" href="/u/sungkim11">@sungkim11</a> <a class="mention" href="/u/fritzo">@fritzo</a> <a class="mention" href="/u/neerajprad">@neerajprad</a> <a class="mention" href="/u/alicanb">@alicanb</a> <a class="mention" href="/u/vishwakftw">@vishwakftw</a></p>

InnovArul: <p><code>.to()</code> modifies the model in-place.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.to" target="_blank" rel="noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.to" target="_blank" rel="noopener">Module — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Multiple inputs in muti-task learning

wqsgsutd: <p>Hi, Everyone<br>
I hope can run a two inputs in multi-tasking learning<br>
Here is the code:</p>
<p>class MTLnet(nn.Module):</p>
<pre><code>def __init__(self):

    super(MTLnet, self).__init__()

    self.sharedlayer = nn.Sequential(

        nn.Linear(feature_size, shared_layer_size),

        nn.ReLU(),

        nn.Dropout()

    )

    self.tower1 = nn.Sequential(

        nn.Linear(shared_layer_size, tower_h1),

        nn.ReLU(),

        nn.Dropout(),

        nn.Linear(tower_h1, tower_h2),

        nn.ReLU(),

        nn.Dropout(),

        nn.Linear(tower_h2, output_size)

    )

    self.tower2 = nn.Sequential(

        nn.Linear(shared_layer_size, tower_h1),

        nn.ReLU(),

        nn.Dropout(),

        nn.Linear(tower_h1, tower_h2),

        nn.ReLU(),

        nn.Dropout(),

        nn.Linear(tower_h2, output_size)

    )    

def forward(self, x1,x2):

    h_shared = self.sharedlayer(x1)

    h_shared = self.sharedlayer(x2)

    out1 = self.tower1(h_shared)

    out2 = self.tower2(h_shared)

    return out1, out2
</code></pre>
<p>MTL = MTLnet()</p>
<p>print(MTL)</p>
<p>I am not sure in the forward () part is right.<br>
but the results looks bad.<br>
Does anyone of you know a smart way of solving this issue?</p>

Usama_Hasan: <p>Hy <a class="mention" href="/u/wqsgsutd">@wqsgsutd</a></p>
<aside class="quote no-group" data-username="wqsgsutd" data-post="1" data-topic="105503">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wqsgsutd/48/14503_2.png" class="avatar"> wqsgsutd:</div>
<blockquote>
<pre><code class="lang-auto">h_shared = self.sharedlayer(x1)
h_shared = self.sharedlayer(x2)
</code></pre>
</blockquote>
</aside>
<p>This is incorrect h_shared will get override in the second line which is causing problem</p>

wqsgsutd: <p>Hi, Usama,<br>
Thank you.<br>
is this right?<br>
def forward(self, x1,x2):</p>
<pre><code class="lang-auto">h_shared1 = self.sharedlayer(x1)

h_shared2 = self.sharedlayer(x2)

out1 = self.tower1(h_shared1)

out2 = self.tower2(h_shared2)

return out1, out2
</code></pre>

Usama_Hasan: <p>Yeah Seems fine for your use case.<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:"></p>

wqsgsutd: <p>Hi, Usama.</p>
<p>Thank you!</p>
Get rid of Batch Normalization for Batch Size 1

JamesDickens: <p>I am training a region proposal network as in Faster R-CNN with a Resnet 101 backbone. My Issue is that the batch size is 1, since for each image you consider 256 proposals, half positive and half negative with the ground truth bounding boxes. Girschick calls this the “image-centric” batch method, although he does use a batch size of 8 with 8 GPU’s for MS Coco 2014.</p>
<p>My question is should I then remove the batch normalization layer (or deactivate it as in this post <a href="https://discuss.pytorch.org/t/how-to-close-batchnorm-when-using-torchvision-models/21812/2">here</a>).</p>
<p>What is the effect on a convolved image of batch normalization if the batch has size 1 ?</p>
<p>The other option is to modify the learning rate and increase the batch size, but I’d like to have flexibility to have whatever batch size I want.</p>
Diffrences in tensor indexing?

drydenwiebe: <p>Hello,</p>
<p>I have a quick question about how tensor elements are accessed.</p>
<p>What would be the difference between:</p>
<pre><code class="lang-auto">c = x[a][b]
</code></pre>
<p>and:</p>
<pre><code class="lang-auto">c = x[a, b]
</code></pre>
<p>Does the first way create an intermediate tensor and then index that? Or are these functionally identical?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>Assuming both a and b are integers:<br>
Yes the first one uses the method to index a single dimension twice with a temporary Tensor in the middle.<br>
The second one calls the general indexing method that will handle this call in one go.</p>
<p>If they are Tensors or iterables, the story is a bit different as each indexing is advanced indexing but it will create a temporary Tensor either way.</p>

drydenwiebe: <p>Right, makes sense. Thank you!</p>
Weighted Multi-class loss

ADONAI_TZEVAOT: <p>Please may I know how to code weighted multiclass loss ?</p>

ptrblck: <p>Assuming you are using <code>nn.CrossEntropyLoss</code> you could pass the <code>weight</code> argument as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss">docs</a>:</p>
<blockquote>
<p><strong>weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor"><em>Tensor</em></a><em>,</em> <em>optional</em> ) – a manual rescaling weight given to each class. If given, has to be a Tensor of size C</p>
</blockquote>
Multilabel classification model

111435: <p>Hi,<br>
I want to train a classification model with 16 classes. Each sample can belong to more than one class.<br>
How can I implement it?<br>
In addition, is it possible to define a score (or weight) for each class and for each sample?<br>
For example:<br>
Assume that we deal with three different classes.<br>
If the relation order between some sample to the classes is:<br>
First place = class 2<br>
Second place = class 3<br>
Third place = class 1<br>
Is it possible to express this order by a vector with the following weights? [ 1/3,0.5,1/6 ]<br>
Is it possible to associate different weights for each sample?<br>
I have an implementation for multiclass classification:<br>
model = Classifier()<br>
loss_func = nn.CrossEntropyLoss()<br>
…<br>
loss = loss_func(prediction, label)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>

ptrblck: <aside class="quote no-group" data-username="111435" data-post="1" data-topic="105518">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111435/48/32101_2.png" class="avatar"> 111435:</div>
<blockquote>
<p>How can I implement it?</p>
</blockquote>
</aside>
<p>You can use <code>nn.BCEWithLogitsLoss</code> as the criterion for a multi-label classification.</p>
<aside class="quote no-group" data-username="111435" data-post="1" data-topic="105518">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111435/48/32101_2.png" class="avatar"> 111435:</div>
<blockquote>
<p>Is it possible to express this order by a vector with the following weights? [ 1/3,0.5,1/6 ]<br>
Is it possible to associate different weights for each sample?</p>
</blockquote>
</aside>
<p>Yes, you can define the class weight tensor and multiply it with the corresponding sample in the unreduced loss (use `reduction=‘none’ while creating the criterion).</p>
Backward Function Produces an Error After zero_infinity=True

paarandika: <p>Hi,<br>
I have implemented a CRNN model based on the <a href="https://arxiv.org/pdf/1507.05717.pdf" rel="nofollow noopener">paper</a>. I have used CTC loss implemented in Pytorch and the model was working fine with one dataset. However, when I changed the dataset it produced <code>nan</code> loss. To avoid that I tried setting <code>zero_infinity=True</code>. Then the backward function produces the following error.<br>
<code>RuntimeError: The size of tensor a (0) must match the size of tensor b (93) at non-singleton dimension 2</code><br>
This happens with both datasets. Does <code>zero_infinity</code> change the shape of tensors? ('m doing some tensor transformations within the model to map CNN output to LSTM input) Can someone point to me what I’m doing incorrectly here?<br>
Thanks.</p>

albanD: <p>Hi,</p>
<p>Where do you set <code>zero_infinity=True</code> ?<br>
Also have you tried reducing the learning rate to prevent your loss from diverging?</p>

paarandika: <p>Hi,<br>
At the CTC loss initiation.</p>
<pre><code class="lang-auto">crit = CTCLoss(zero_infinity = True).to(device)
</code></pre>
<p>I haven’t tried reducing the learning rate. I will try that.<br>
Thanks.</p>

albanD: <p>Interesting. Can you make a small code repro (30 lines) that reproduces this issue please?</p>

paarandika: <p>Hi,<br>
I’m using the model from this repo.<br>
<a href="https://github.com/Holmeyoung/crnn-pytorch" rel="nofollow noopener">https://github.com/Holmeyoung/crnn-pytorch</a></p>

AlexanderKazakov: <p>I have also faced that error. I’ve created a bug report here <a href="https://github.com/pytorch/pytorch/issues/49046" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/49046</a></p>
Does Conv2d filter need bias per weights?

Yupjun: <p>Let’s say I have an image with (3,32,32) size and I want to apply (32,4,4) filter.<br>
If I use “torch.nn.Conv2d” then<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e0db55f158554e2d5e83665d7772dda85dccd5c.png" alt="image" data-base62-sha1="6zpkHCqnW9snNTD3ZxXjIdrJhow" width="574" height="76"><br>
by this formula, I have 32 bias values.<br>
If let’s say adding bias at each weight, then I’ll have 32x(4x4)= 512 bias values.<br>
Is there any reason not applying bias terms at each weight in the Conv2d filter?</p>

ptrblck: <p>The filter will be applied as an elementwise multiplication to the current input window and summed afterwards to create a scalar output value. The bias will then be added to the scalar value, which explains the shape.</p>
Trained model does not work during preictions

Michael_Lempart: <p>Hi all,</p>
<p>I trained a 3D Unet for a segmentation task and training works great.<br>
When I now load my model and set the state to model.eval() nothing is working.</p>
<p>The predictions are totally empty.</p>
<p>When I do the same, But with model.train() everything works like a charm.</p>
<p>Has nyone experienced that kind of behaviour?</p>
<p>Thanks in advance,</p>
<p>Cheers,</p>
<p>Michael</p>
Using LSTMCell for sequence to sequence encoder-decoder

Wesley_Neill: <p>Here is a summary of my attempt at a sequence-to-sequence autoencoder. This image was taken from this paper: <a href="https://arxiv.org/pdf/1607.00148.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1607.00148.pdf</a><br>
<a href="https://i.stack.imgur.com/RbmCF.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/RbmCF.png" alt="enter image description here" width="690" height="342"></a></p>
<p>What I have found is that I can implement this if my decoder only has a single layer. This is because the final hidden state of the encoder half is shaped like <code>(num_layers, batch_size, emb_size</code>), where <code>emb_size == hidden_size</code>  of <code>nn.LSTM</code> layer.</p>
<p>Here is a functioning code snippet:</p>
<pre><code class="lang-auto"># Standalone trial of encode-decode process:
batch_size = 1
seq_len = 3
n_features = 20
emb_size = 10
n_layers = 1

# generate random sequence of numbers
seq = np.random.randn(batch_size, seq_len, n_features)
seq = torch.tensor(seq).float()

# pass sequence through LSTM and get final hidden state
enc_lstm = nn.LSTM(n_features, emb_size, n_layers, batch_first=True)
out, hs_enc = enc_lstm(seq)

# unsqueeze first dimension of hidden state:
hs_enc = tuple([h.squeeze(0) for h in hs_enc])

# initialize LSTM Cell with encoder hidden state:
dec_cell = nn.LSTMCell(emb_size, emb_size)
hs_3 = dec_cell(hs_enc[0], hs_enc)

# reconstruct element x_i of original sequence:
dense = nn.Linear(emb_size, n_features)
x_3 = dense(hs_3[0])
x_3.shape # 
</code></pre>
<p>But here I run into a problem. According to the diagram, <code>x_3</code> and the new hidden state <code>hs_3</code> should be passed to the LSTM cell to get <code>hs_2</code>… but <code>x_3</code> has shape <code>(batch_size, n_features)</code>, which can’t really be resized to fit <code>(batch_size, input_size)</code>.</p>
<pre><code class="lang-auto">hs_2 = dec_cell(x_3, hs_3)
x_2 = dense(hs_2[0])

RuntimeError: input has inconsistent input_size: got 20, expected 10
</code></pre>
<p>Well, that’s because <code>x_3</code>  is an element of the original time series and contains 20 features… So what do I do?</p>
Dense Network returns all the same output before training

contagon: <p>I’m trying to train a simple dense network that takes in 6 inputs, and outputs 3. For some reason, regardless of what I put it, it always outputs the same thing right away. Training doesn’t seem to solve it.</p>
<p>Is there something I’m doing wrong here? I’m sure it’s something simple, but it’s driving me a bit nuts.</p>
<pre><code class="lang-python">import torch.nn as nn
import torch

input_size = 6
hidden_size = 64
n_layers = 24
output_size = 3

layers = [nn.Linear(input_size, hidden_size), nn.ReLU()]
for i in range(n_layers):
    layers.append( nn.Linear(hidden_size, hidden_size) )
    layers.append( nn.ReLU() )
layers.append( nn.Linear(hidden_size, output_size) )

net = nn.Sequential(*layers)

x = torch.normal(0,1,(5,6))
print(net(x))
</code></pre>
<p>This outputs something like</p>
<pre><code class="lang-python">tensor([[ 0.0090, -0.0353, -0.0797],
        [ 0.0090, -0.0353, -0.0797],
        [ 0.0090, -0.0353, -0.0797],
        [ 0.0090, -0.0353, -0.0797],
        [ 0.0090, -0.0353, -0.0797]], grad_fn=&lt;AddmmBackward&gt;)
</code></pre>

Ouasfi: <p>Hi, Are setting a seed somewhere in your code? I’m able to reproduce your issue by adding <code>torch.random.manual_seed(1)</code> at the top of your code and using a learning rate that is too high. The problem during training maybe due to vanishing gradients or an optimizer that is not well initialized.</p>

contagon: <p>I’m not setting any seeds in my code. The outputs are different each time I run it, but the strange part is that all 5 of them are the same.</p>
<p>I’ll try lowering my learning rate, that’s a great suggestion. I’m still weirded out that all those are the same BEFORE training. Is that normal behavior? I’ve just never heard of anything like it.</p>

pchandrasekaran: <p>I assumed the problem was a classification one and obtained a probability of approx 1/3 for each class with the same values for each sample. That is to be expected (albeit exactly the same maybe down to the no. of significant digits after decimal) at the start, since the network hasn’t learned anything yet and by pure chance, you get 1/3. If you continue to get the same values a few epochs into training, it may be a problem.</p>
<p>Out of curiosity, could you try manually initializing the linear layer weights using a different distribution? I believe the default is uniform in <code>[-1/sqrt(1/(no.of columns in weight matrix),  1/sqrt(1/(no.of columns in weight matrix)]</code></p>
Extending parameter tensor dimension

stefano_d: <p>Hi,</p>
<p>I am trying to extend the size of a nn.parameter, but without creating a new one. Is it possible?</p>
<p>Here is a simplified code of what I want to do:</p>
<pre><code class="lang-auto">import torch

w = torch.nn.Parameter(torch.randn((2,5)),requires_grad=True)

l = torch.sum(w)
l.backward()

# if "del l" is added here then the code runs 

ext_w = torch.randn((1,5),requires_grad=True) # now the shape is (3,5)
w.data = torch.cat((w.data, ext_w), dim=0)
w.requires_grad = True
w.grad= None

l = torch.sum(w)
l.backward()
</code></pre>
<p>if I run this code I get this error:</p>
<p>RuntimeError: Function SumBackward0 returned an invalid gradient at index 0 - got [3, 5] but expected shape compatible with [2, 5]</p>
<p>my question is why is this happening? which object is expecting a shape [2,5]?</p>
<p>The weird thing is that, I tried deleting the l variable (del l) before extending the w parameter, and in that case it seems to work</p>
<p>thanks a lot in advance</p>

albanD: <p>Hi,</p>
<p>You should <em>never</em> use <code>.data</code> as it is unsafe and as you can see, leads to weird errors.</p>
<blockquote>
<p>but without creating a new one. Is it possible?</p>
</blockquote>
<p>What do you mean by this? You actually create a brand new Tensor there.</p>

stefano_d: <p>Hi,</p>
<p>thanks for the quick answer.</p>
<p>well, I do not create a new parameter object. The longer story is that the parameter is inside a model that is training, and an optimizer is used for training. I do not want to create a new parameter because then I would also need to re-create a new optimizer each time, whereas if I could only expand the inner data I could keep using the same optimizer.</p>
<p>(I adapted the optimizer code to deal with “expanding” gradients, needed for methods like sgd with momentum)</p>

albanD: <p>In that case you can use set:</p>
<pre><code class="lang-auto">import torch

w = torch.nn.Parameter(torch.randn((2,5)),requires_grad=True)

l = torch.sum(w)
l.backward()

# if "del l" is added here then the code runs 

ext_w = torch.randn((1,5),requires_grad=True) # now the shape is (3,5)
with torch.no_grad():
    w.set_(ext_w)

l = torch.sum(w)
l.backward()
</code></pre>
<p>And you will need to update the optimizer if it has state as well <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Will torch.exp cut off compute graph?

Tiffany_Zhao: <p>I want to implement this function: x / ( 1 - e^(-x )). However, after I’ve done that, all of my loss became nan. I don’t know what happen to me <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=9" title=":sob:" class="emoji" alt=":sob:"></p>
<p>and here is my code:</p>
<pre><code class="lang-auto">class MyAttn(nn.Module):
    def __init__(self, channel, reduction=16):
        super(MyAttn, self).__init__()
        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Sequential(
            spectral_norm(nn.Linear(channel, channel // reduction, bias=False)),
            nn.ReLU(inplace=True),
            spectral_norm(nn.Linear(channel // reduction, channel, bias=False)),
        )

    def forward(self,  x):
        b, c, _, _ = x.size()
        y = self.avg_pool(x).view(b, c)
        y = self.fc(y)
        
        energy = y / (1  - torch.exp(-y))
        energy = energy.view(b, c, 1, 1)
        return x * energy.expand_as(x)
</code></pre>

albanD: <p>Hi,</p>
<p>exp will no break the graph.<br>
But given the function, if <code>x</code> is 0 then you will end up dividing by <code>0</code> and will get bad numbers. Can you check that this is not the cause of your problem?</p>

Tiffany_Zhao: <p>thanks a lot!<br>
but what should I do if I just want to make dividing operation?</p>

albanD: <blockquote>
<p>but what should I do if I just want to make dividing operation?</p>
</blockquote>
<p>Not sure what you mean by that?<br>
In general, you need to make sure you don’t divide by 0 (by either clamping to eps or adding eps to before the division).</p>
Data Subsampling for Hyperparameter Optimisation

hH1sG0n3: <p>Fundamentally, under what circumstance is it reasonable to do HPO only on a subsample of the training set.</p>
<p>I am using Population Based Training with RayTune to optimise hparameters for a sequence model. My dataset consists of 20M sequences and was wondering if it would make sense to optimise over a subsample due to restricted time budget.</p>
Infer the size of the output of an intermediate layer considering the input size

trinity: <p>Hello, is there any efficient way to determine the size of the output of an intermediate layer considering the input size ? For an instance, in the following code snippet the output of conv_3 has the size of N x 256 x 8 x 8 for for an input of size N x 1 x 64 x 64 . So the  <code>in_features</code> of <code>fc1</code> will be 256 x 8 x 8 . Instead of manually calculating this value for any particular input dimension, is there any way to automate this? Thank you for any help.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 64, 3, padding=1)
        self.conv1_ = nn.Conv2d(64, 64, 3, padding=1)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(64, 128, 3, padding=1)
        self.conv2_ = nn.Conv2d(128, 128, 3, padding=1)
        self.conv3 = nn.Conv2d(128, 256, 3, padding=1)
        self.conv3_ = nn.Conv2d(256, 256, 3, padding=1)
        self.flat = nn.Flatten()
        self.fc1 = nn.Linear(256 * 8 * 8, 15)           

    def forward(self, x):         
        x = (F.leaky_relu(self.conv1(x)))          
        x = self.pool(F.leaky_relu(self.conv1_(x)))  
        x = (F.leaky_relu(self.conv2(x)))           
        x = self.pool(F.leaky_relu(self.conv2_(x)))           
        x = (F.leaky_relu(self.conv3(x)))           
        x = self.pool(F.leaky_relu(self.conv3_(x)))        
        x = self.flat(x)      
        x = (self.fc1(x))        
        return x
</code></pre>

superunification: <p>You can try sending a test tensor through the encoding conv / pooling layers before declaring your linear layer, and check its output shape, then use that shape for calculating the flat size of the linear layer.</p>
<p>For example, it’s done like that here: <a href="https://github.com/yjlolo/vae-audio/blob/3a43e3122da55daf89fffaf5bbf11fdb805a597f/model/model.py#L180" rel="noopener nofollow ugc">https://github.com/yjlolo/vae-audio/blob/3a43e3122da55daf89fffaf5bbf11fdb805a597f/model/model.py#L180</a></p>

trinity: <p>Hi, that sounds like a neat idea to handle this issue. Thank you for the suggestion.</p>
Progressively loading data in deep regression

xsquared: <p>Hello guys and ladies,</p>
<p>In Keras you can progressively load data in a deep regression problem using generators ie <code>generatorA.flow_from_dataframe(...)</code>.</p>
<p>How do you do this in Pytorch?</p>
<p>The reason I ask this is because it was not so obvious in Keras; I had to Google for a while. The generator introduced in the main Keras book assumed a classification problem. So, I had look for a generator specifically for deep regression.<br>
Thanks!</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/recipes/recipes/loading_data_recipe.html">data loading recipe</a> and the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">custom data loading tutorial</a> might be a good starter.</p>
<p>Let us know, if you need more information. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

xsquared: <p>thank you for your help!</p>
Dot product and matmul

Rohit_Modee: <p>I have tensor x shape = torch.Size([32, 69, 64]) and y = torch.Size([32, 69, 68, 64]). I want output to be torch.Size([32, 69, 68, 1]). matrix multiplication of [1,64] and [68,64] to [68,1].</p>

KFrank: <p>Hi Rohit!</p>
<aside class="quote no-group" data-username="Rohit_Modee" data-post="1" data-topic="105499" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rohit_modee/48/12231_2.png" class="avatar"> Rohit_Modee:</div>
<blockquote>
<p>I have tensor x shape = torch.Size([32, 69, 64]) and y = torch.Size([32, 69, 68, 64]). I want output to be torch.Size([32, 69, 68, 1]). matrix multiplication of [1,64] and [68,64] to [68,1].</p>
</blockquote>
</aside>
<p>Try:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.einsum ('ijl, ijkl -&gt; ijk', x, y).unsqueeze (-1)
</code></pre>
<p>The <code>einsum()</code> “contracts” over the “64” dimension, while the<br>
<code>.unsqueeze (-1)</code> adds the trailing singleton dimension.</p>
<p>Best.</p>
<p>K. Frank</p>
Global argmax as index in tensor

booQorm: <p>I’m new to Torch, and there’s one thing I find counterintuitive in <code>argmin</code> and <code>argmax</code> operations,<br>
so, per <a href="https://pytorch.org/docs/stable/generated/torch.argmin.html" rel="noopener nofollow ugc">example from the docs</a> we have</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.randn(4, 4)
&gt;&gt;&gt; a
tensor([[ 0.1139,  0.2254, -0.1381,  0.3687],
        [ 1.0100, -1.1975, -0.0102, -0.4732],
        [-0.9240,  0.1207, -0.7506, -1.0213],
        [ 1.7809, -1.2960,  0.9384,  0.1438]])
&gt;&gt;&gt; torch.argmin(a)
tensor(13)
</code></pre>
<p>But wouldn’t it be more natural to return <code>tensor([3,1])</code>? In particular case, I have two distinct sets of points A and B, and I look for the indices of a pair (a,b) ∈ A × B. I can measure pairwise distances using <code>cdist</code>, but a <code>42</code>-like answer is of little use. Sure the indices can be inferred, but doing so looks messy and superfluous.<br>
Is there any better solution? And also, are there any reason for delivering <code>argmin</code>/<code>argmax</code> this way beyond backward compatibility and seeking in an effectively linear array?</p>
Maximum efficiency on Numpy vs Pytorch vs GPU

vishak_bharadwaj: <p>Any general guidelines both space and time wise as to which frameworks are best suited in the various stages of ML</p>
<p>Are there any guidelines as to why transforms, for example, shouldn’t happen on the GPU but on Pytorch CPU. If Pytorch CPU is slower than Numpy, then why not code the transforms up in Numpy.</p>

ptrblck: <p>I would claim it all depends on your use case and there is no silver bullet.</p>
<p>By default you would want to use the GPU to train the model and the CPU to load and process the data. Using multiple workers in the <code>DataLoader</code> the data loading and processing will be applied in the background, while the GPU is busy with training the model.<br>
<code>torchvision.transforms</code> are using <code>PIL</code> as the default backend for image transformations, which uses numpy arrays internally.</p>
<p>However, if you need differentiable transformations (e.g. via Kornia), you would have to apply them on tensors and cannot use numpy anymore (or would need to write the backward functions manually).</p>
<p>Also, on “bigger” systems, you might see that the CPU is not fast enough feeding the GPUs, which is visible in starving GPUs.<br>
In that case you could use e.g. <a href="https://github.com/NVIDIA/DALI">NVIDIA DALI</a> to apply the transformations on the GPU, which could give you an overall performance boost.</p>

vishak_bharadwaj: <p>That’s what I needed. Thanks!</p>