PyTorch 1.9.0. cause RuntimeError: expected scalar type Half but found Float While PyTorch 1.7.1 does not cause any problem

Hussein: <p>Hello<br>
I am using Google Colab for training contrastive learning model. All the code is from this notebook</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/github/facebookresearch/moco/blob/colab-notebook/colab/moco_cifar10_demo.ipynb#scrollTo=dvuxcmejkKt8">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20211001-060054-RC00_400164144" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/github/facebookresearch/moco/blob/colab-notebook/colab/moco_cifar10_demo.ipynb#scrollTo=dvuxcmejkKt8" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/github/facebookresearch/moco/blob/colab-notebook/colab/moco_cifar10_demo.ipynb#scrollTo=dvuxcmejkKt8" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>When I use PyTorch 1.9.0+cu102 and Torchvision 0.10.0+cu102 I got this error (RuntimeError: expected scalar type Half but found Float ) BUT when I use PyTorch 1.7.1 and Torchvision 0.8.2 everything is fine. Thanks for the help</p>
<p>#################################################################################<br>
RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
376     print(vars(args))<br>
377<br>
 → 378     main(args)<br>
379<br>
380</p>
<p>3 frames<br>
 in main(args)<br>
356     # training loop<br>
357     for epoch in range(epoch_start, args.epochs + 1):<br>
 → 358         train_loss = train(model, train_loader, optimizer, epoch, args)<br>
359         results[‘train_loss’].append(train_loss)<br>
360         <span class="hashtag">#test_acc_1</span> = test(model.encoder_q, memory_loader, test_loader, epoch, args)</p>
<p> in train(net, data_loader, train_optimizer, epoch, args)<br>
170             if args.grad_accum==True:<br>
171                 if ((i + 1) % args.accum_iter == 0) or (i + 1 == len(data_loader)):<br>
 → 172                     scaler.scale(loss).backward()<br>
173                     scaler.step(train_optimizer)<br>
174                     scaler.update()</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)<br>
253                 create_graph=create_graph,<br>
254                 inputs=inputs)<br>
 → 255         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
256<br>
257     def register_hook(self, hook):</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)<br>
147     Variable.<em>execution_engine.run_backward(<br>
148         tensors, grad_tensors</em>, retain_graph, create_graph, inputs,<br>
 → 149         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag<br>
150<br>
151</p>
<p>RuntimeError: expected scalar type Half but found Float<br>
#################################################################################</p>
<hr>
<h2>
<a name="the-loss-and-the-training-loop-are-as-the-following-1" class="anchor" href="#the-loss-and-the-training-loop-are-as-the-following-1"></a>The loss  and the training loop are as the following:</h2>
<p>def contrastive_loss(self, im_q, im_k):</p>
<pre><code>    # compute query features

    q = self.encoder_q(im_q)  # queries: NxC

    #print('q.dtype',q.dtype)

    q = nn.functional.normalize(q, dim=1)  # already normalized

    #print('q.dtype',q.dtype)

    # compute key features

    with torch.no_grad():  # no gradient to keys

        # shuffle for making use of BN

        im_k_, idx_unshuffle = self._batch_shuffle_single_gpu(im_k)

        

        k = self.encoder_k(im_k_)  # keys: NxC

        #print('k.dtype',k.dtype)

        k = nn.functional.normalize(k, dim=1)  # already normalized

        #print('k.dtype',k.dtype)

        # undo shuffle

        k = self._batch_unshuffle_single_gpu(k, idx_unshuffle)

        #print('k.dtype',k.dtype)

    # compute logits

    # Einstein sum is more intuitive

    # positive logits: Nx1

    l_pos = torch.einsum('nc,nc-&gt;n', [q, k]).unsqueeze(-1)

    #print('l_pos.dtype',l_pos.dtype)

    # negative logits: NxK

    l_neg = torch.einsum('nc,ck-&gt;nk', [q, self.queue.clone().detach()])

    #print('l_neg.dtype',l_neg.dtype)

    

    # logits: Nx(1+K)

    logits = torch.cat([l_pos, l_neg], dim=1)

    #print('logits.dtype',logits.dtype)

    # apply temperature

    logits /= self.T

    #print('logits.dtype',logits.dtype) 

    # labels: positive key indicators

    labels = torch.zeros(logits.shape[0], dtype=torch.long).cuda() #.to(self.device)

    #print('labels.dtype',labels.dtype)

    ###

    #labels = labels.type(torch.LongTensor).cuda()

    ###

    loss = nn.CrossEntropyLoss().cuda()(logits, labels)            #.to(self.device)(logits, labels)

    #print('loss.dtype',loss.dtype)

    return loss, q, k

def forward(self, im1, im2):

    """

    Input:

        im_q: a batch of query images

        im_k: a batch of key images

    Output:

        loss

    """

    # update the key encoder

    with torch.no_grad():  # no gradient to keys

        self._momentum_update_key_encoder()

    # compute loss

    if self.symmetric:  # asymmetric loss

        loss_12, q1, k2 = self.contrastive_loss(im1, im2)

        loss_21, q2, k1 = self.contrastive_loss(im2, im1)

        loss = loss_12 + loss_21

        print('loss.dtype',loss.dtype)

        k = torch.cat([k1, k2], dim=0)

    else:  # asymmetric loss

        loss, q, k = self.contrastive_loss(im1, im2)

    self._dequeue_and_enqueue(k)

    return loss
</code></pre>
<hr>
<h1>
<a name="train-for-one-epoch-2" class="anchor" href="#train-for-one-epoch-2"></a>train for one epoch</h1>
<p>def train(net, data_loader, train_optimizer, epoch, args):</p>
<pre><code>i = 0



scaler = torch.cuda.amp.GradScaler(enabled=args.use_amp)

net.train()

adjust_learning_rate(train_optimizer, epoch, args)

total_loss, total_num, train_bar = 0.0, 0, tqdm(data_loader)

for clip_1, clip_2 in train_bar:

    clip_1 = clip_1.cuda(non_blocking=True)   #.to(args.device) 

    clip_2 = clip_2.cuda(non_blocking=True)   #.to(args.device)

    

    #print('clip_1.shape', clip_1.shape)

    #print('clip_2.shape', clip_2.shape)

    with torch.cuda.amp.autocast(enabled=True):

        loss = net(clip_1, clip_2)

        #print('loss.dtype',loss.dtype)

            

    if args.grad_accum==True and args.batch_size&gt;1:

        loss = loss / args.accum_iter

    #print('loss',loss)

    #print(loss.shape)

    if args.grad_accum==False:

        train_optimizer.zero_grad()

        #loss.backward()

        #train_optimizer.step()

        scaler.scale(loss).backward()

        scaler.step(train_optimizer)

        scaler.update()

    else:

        if args.grad_accum==True:

            if ((i + 1) % args.accum_iter == 0) or (i + 1 == len(data_loader)):

                scaler.scale(loss).backward()

                scaler.step(train_optimizer)

                scaler.update()

                # zero the parameter gradients

                train_optimizer.zero_grad()

   

    total_num += data_loader.batch_size

    total_loss += loss.item() * data_loader.batch_size

    train_bar.set_description('Train Epoch: [{}/{}], lr: {:.6f}, Loss: {:.4f}'.format(epoch, args.epochs, train_optimizer.param_groups[0]['lr'], total_loss / total_num))

    

    i+=1

    if i%30==0:

         torch.save({'epoch': epoch, 'state_dict': net.state_dict(), 'optimizer' : train_optimizer.state_dict(),}, args.results_dir + '/model_last.pth')

return total_loss / total_num
</code></pre>
<hr>

Hussein: <p>I am sorry, I forgot to mention that I used a different encoder than the one in the Colab Notebook. I used S3dg encoder with  videos as input not images but I don’t think its affect anything because the encoder outputs 128 embedding vector anyways.</p>
<p>Thanks</p>

ptrblck: <p>The notebook doesn’t show an error, so could you post a minimal, executable code snippet to reproduce the issue, please?</p>

Hussein: <p>Thanks ptrblck. Please follow this link. It is the complete notebook.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1wy7NK83P0Ebe5BpZHcTBDLcyR67xABON?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20211001-060054-RC00_400164144" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1wy7NK83P0Ebe5BpZHcTBDLcyR67xABON?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1wy7NK83P0Ebe5BpZHcTBDLcyR67xABON?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>You need to have the following folders, one for the videos of the UCF and the other one is for<br>
the split’s information.</p>
<p>self.videos_path=’/content/drive/My Drive/UCF-101 Dataset/Videos/UCF-101’<br>
self.splits_path=’/content/drive/My Drive/UCF-101 Dataset/Splits’</p>
<p>Many thanks my friend !!!<br>
Hussein A Hassan</p>

ptrblck: <p>Could you please post the input shapes to make the model executable or is the dataset necessary to reproduce the issue?</p>

Hussein: <p>Hello ptrblck</p>
<p>Thank you very much for your time. It is much appreciated !!!<br>
I modified the notebook so it can be executed directly and that should reproduce the error without the<br>
need for the dataset.</p>
<p>The input for each batch is one video tensor so the shape of the batch is (1, 3, 64, 224, 224).</p>
<p>Many Thanks !!!<br>
Hussein</p>

ptrblck: <p>Thanks for the notebook!<br>
I can reproduce the issue in Colab using the older <code>1.9.0</code> release, while <code>1.9.1</code> works fine (also locally).<br>
I’ve used</p>
<pre><code class="lang-auto">!pip install torch==1.9.1+cu111 https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>in the first cell to update PyTorch.</p>

Hussein: <p>Many thanks to you ptrblck, this helps me a lot.</p>