Load batch to GPU problem in pytorch using BERT model

username: Yorgos_Pantis
cooked: <p>I want to implement the BERT model for a classification task based on this tutorial <a href="https://skimai.com/fine-tuning-bert-for-sentiment-analysis/" class="inline-onebox" rel="noopener nofollow ugc">Tutorial: Fine-tuning BERT for Sentiment Analysis - by Skim AI</a></p>
<p>Training step</p>
<pre><code class="lang-auto">def train(model, train_dataloader, val_dataloader=None, epochs=4, evaluation=False):
    """
    Train the BertClassifier model.
    """
    # Start training loop
    print("Start training...\n")
    for epoch_i in range(epochs):
        # =======================================
        #               Training
        # =======================================
        # Print the header of the result table
        print(f"{'Epoch':^7} | {'Batch':^7} | {'Train Loss':^12} | {'Val Loss':^10} | {'Val Acc':^9} | {'Elapsed':^9}")
        print("-"*70)

        # Measure the elapsed time of each epoch
        t0_epoch, t0_batch = time.time(), time.time()

        # Reset tracking variables at the beginning of each epoch
        total_loss, batch_loss, batch_counts = 0, 0, 0

        # Put the model into the training mode
        model.train()

        # For each batch of training data...
        for step, batch in enumerate(train_dataloader):
            batch_counts +=1
            # Load batch to GPU
            b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)

            # Zero out any previously calculated gradients
            model.zero_grad()

            # Perform a forward pass. This will return logits.
            logits = model(b_input_ids, b_attn_mask)

            # Compute loss and accumulate the loss values
            loss = loss_fn(logits, b_labels)
            batch_loss += loss.item()
            total_loss += loss.item()

            # Perform a backward pass to calculate gradients
            loss.backward()

            # Clip the norm of the gradients to 1.0 to prevent "exploding gradients"
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            # Update parameters and the learning rate
            optimizer.step()
            scheduler.step()

            # Print the loss values and time elapsed for every 20 batches
            if (step % 20 == 0 and step != 0) or (step == len(train_dataloader) - 1):
                # Calculate time elapsed for 20 batches
                time_elapsed = time.time() - t0_batch

                # Print training results
                print(f"{epoch_i + 1:^7} | {step:^7} | {batch_loss / batch_counts:^12.6f} | {'-':^10} | {'-':^9} | {time_elapsed:^9.2f}")

                # Reset batch tracking variables
                batch_loss, batch_counts = 0, 0
                t0_batch = time.time()

        # Calculate the average loss over the entire training data
        avg_train_loss = total_loss / len(train_dataloader)

        print("-"*70)
        # =======================================
        #               Evaluation
        # =======================================
        if evaluation == True:
            # After the completion of each training epoch, measure the model's performance
            # on our validation set.
            val_loss, val_accuracy = evaluate(model, val_dataloader)

            # Print performance over the entire training data
            time_elapsed = time.time() - t0_epoch
            
            print(f"{epoch_i + 1:^7} | {'-':^7} | {avg_train_loss:^12.6f} | {val_loss:^10.6f} | {val_accuracy:^9.2f} | {time_elapsed:^9.2f}")
            print("-"*70)
        print("\n")
    
    print("Training complete!")

</code></pre>
<p>After training the model I want to refit it again in both training and validation sets. I should mention that the texts_train and texts_valid are lists.</p>
<pre><code class="lang-auto"># Train our model on the entire training data
# Concatenate the train set and the validation set
full_train_data = torch.utils.data.ConcatDataset([texts_train, texts_valid])
full_train_sampler = RandomSampler(full_train_data)
full_train_dataloader = DataLoader(full_train_data, sampler=full_train_sampler, batch_size=32)

# Train the Bert Classifier on the entire training data
set_seed(42)
bert_classifier, optimizer, scheduler = initialize_model(epochs=2)
train(model=bert_classifier, train_dataloader=full_train_dataloader, val_dataloader=None, epochs=2, evaluation=False)
</code></pre>
<p>And I have the following error</p>
<pre><code class="lang-auto">AttributeError: 'str' object has no attribute 'to'
</code></pre>
<p>See below the full error<br>
<a href="https://i.stack.imgur.com/WPMie.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/WPMie.png" alt="enter image description here" width="689" height="252"></a></p>

username: ptrblck
cooked: <p>Based on the error message it seems that <code>batch</code> contains a <code>str</code> object (besides tensors or other objects), which can’t be pushed to the GPU. Could you check the content of <code>batch</code> and call <code>to(device)</code> only on tensors?</p>

username: Yorgos_Pantis
cooked: <p>Thank you so much for your help. Is it possible to be more specific as for the coding part? I am a begginer in pytorch, thus I’m not so familiar with the concepts.</p>

username: ptrblck
cooked: <p>Check what types are returned by the <code>DataLoader</code> via:</p>
<pre><code class="lang-python">for batch in full_train_dataloader:
    for b in batch:
        print(type(b))
</code></pre>
<p>and make sure they are all tensors.<br>
Based on the error message:</p>
<pre><code class="lang-python">tuple(t.to(device) for t in batch)
</code></pre>
<p>crashes since <code>t</code> is a <code>str</code> object while a tensor is expected.</p>

username: Yorgos_Pantis
cooked: <p>The output is the following</p>
<pre><code class="lang-auto">Input: for batch in full_train_dataloader:
    for b in batch:
        print(type(b))

Output:

&lt;class 'torch.Tensor'&gt;
</code></pre>
<pre><code class="lang-auto">Input: tuple(t.to(device) for t in batch)
Output:
(tensor([[  101,  3183,  6656,  1288,   121,  2981,  3070,   349,  3824,  7192,
            613,   273, 11811,  4858,   435,   358,   708,  3234, 24109,  2504,
          11753,   121,   435,   349,   802,  7192,   344,   345,  3865, 18340,
          13825, 12820,  9857,   358, 24860,   273,  3712,   273,  1463,  6286,
           1873,   774,   588,  9857, 10423,  1571,   373,   121,   509,  2947,
           6461,  2489,  1287,  2550,   530,   346,   433,  6302,   247,   281,
            280,  1250,   614,  1742,   414, 22267,   247, 11883,   515,  1288,
            239,   440,  1508,  1383,   355,  8489,  1723,  4385,   558,  8518,
            278,  1354,   683,   273, 12948,   239,   273, 18340,   414,  2359,
          12948,   770,   267,   564,  9857,  1050,  3234,   749,  4624,   278,
            344, 10240,  2746,  1622,  2201,   278,   121,   374,   344,   345,
            349, 22321,   278,  1393, 29187,  1475,  3102,   273,  3186, 11811,
           4858, 14588,   247,  6911,  8089,   483,  4651,   278,   247,  6911,
            237,   261,   281,  1423,  6373,   278,  1294,  1604, 16231,   121,
            344,   102,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0],
         [  101,  4570,  6509,   273,   435,   349,   518,   939, 13508,   569,
           4999,  1678,   267,   381,  2981,  1372,  7511,   278,  1060,   247,
            281,   349,  6171,  4905,  6029,   769, 13056,  2447,   497,   433,
           1136,  3815, 24109,  2504, 10765,   483,   278,   121,  2344,   236,
            437,  2359,  1792,   566,  6424,  5409,   661,   273,  1354,   344,
          11797,   267,   668,   273,  2416,  1413,   281,  2928,   410,   273,
            269,   350, 13715,   273,  1678,  4385,   421,   346,  5101,  4201,
           4961,   281, 17700,   390, 13958,   273,  2330,  6040,   273,   121,
            247,   351,  2108,   269, 18687,   278,   691,  1436,   281,  1413,
           4872,  3744, 11811,   468,   344,   346,  1423, 10760,  3001,   381,
            367,   643,   832,  1946, 10986,   278,  9038, 11072,  1413,   281,
           1857,   904,   643,  8092,  6654,  9857,   121,   100,   121,   518,
          21391,   389,   121, 21391,  1032,  6874,  5532,   344,  3468,  5532,
            382,   350,  1183,   273,   352, 13120,   273,   121,  9038,   433,
            344, 23006,   378,   387,   102,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0],
         [  101,   358,  4668, 10009,   781, 10290,   273,  1200, 16760,   121,
            363,   433, 10373,  2658,   269,   929,   351,   273,  9656,   344,
          12547,   269,  2378,  9483,   236,   257, 15158,  6167,   404,  3156,
          14553,   387, 17238,  8850,   742,   278,  3885, 14229,  7789,   344,
            452,  4824,  6357,  9857, 29526,   373,   352,  6636,   278,   247,
           7151,  1270,   121,   247,   281,   441,  5646,  3679,   605,   744,
            437,   868,   868,  2669,  5595,   278, 16372,   265,  6686,   814,
            344,   433,  2927,   605,   518,   358, 22972,   373, 17621,   278,
           1294,  1403,  3058,   420,   404,   606,   344,  5463, 34931,  2313,
            362, 17426,  4430,   273,   352,  4791,   421,   355,   278,  1643,
            624,   742,   278,   344,  1995,   546,   421,   345,   349,  3664,
            613,   273,  1678,   421, 16940,   483,   344, 11989,   285, 26157,
            273,  3815,  2411, 16514,  4113,   273,   344, 28701,   121,   346,
            433, 17426,  4430,   281,   355,   278,   351,   273,  1643,   624,
            452,   466,   269,  5779, 18123,   281,   102,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0],
         [  101, 19828,  3998, 14696,  9968,   904,   381,   367,  3343,  1571,
           4858,   247,  8009,   278, 22267,  1802, 19828,   269,   285,   414,
          28672,  1191,  5758,  2179,  9038,  3815, 17657,   483,   278, 16252,
            934,   121,   655,   247,   281,   273, 29478,  1032,  4569,  1721,
            345, 19828,  3998,   239,   271,  1742,   494,   382,   362,   713,
            513,   273,  1821,   729,  1112,   410,  1199,   255,  3573,   373,
            643,   376, 31409,   273,   121,   247,   281,  2947,   440, 27967,
           3815,   924,   273,  4569,  3810,  1825,   273, 16252,   934,   239,
            100,   595,  7314,  1571,   267, 10580,   546,   464, 11367,   433,
           1698,  9857,   358,  4067,   350,  4020,   273,   358,  1995,  3344,
           7385,   255,  3573,   373,   121,  2273,  2947,   355,  1821,  1340,
            990,   358,   248,  2351,  1894,   464,   350,   358,   255,  3573,
            373, 12263,   273,   437,  7312,   827,   121,   355,   351,   273,
           2947,  1821,  1340,   990, 14397, 16907,   273,   357,  2706,   483,
            362,  5075,   273,   355,   102,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0]], device='cuda:0'),
 tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
         [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
         [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
         [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], device='cuda:0'),
 tensor([9, 9, 3, 9], device='cuda:0'))

</code></pre>

username: ptrblck
cooked: <p>This single batch seems to contain the expected tensors, so you would either have to check the entire dataset or verify where the <code>str</code> is coming from in your code.</p>