Running with no end when usin the pipeline Trainer for the transformers

MABtt: <p>Hello,</p>
<p>I have been exploring the pipelines of transformers with a dataset of tweets.<br>
After I charged the data and prepare it with train_test_split of scikit learn, I prepared everything as the code below:</p>
<pre><code class="lang-auto"># load the tokenizer
tokenizer = CamembertTokenizerFast.from_pretrained("camembert-base", do_lower_case=True)

# tokenize the dataset, truncate when passed `max_length = 512`, 
# and pad with 0's when less than `max_length`
train_encodings = tokenizer(train_texts, truncation=True, padding=True, max_length=512)
valid_encodings = tokenizer(valid_texts, truncation=True, padding=True, max_length=512)

#load the model and pass to CUDA after importing all the packages 
model = CamembertForSequenceClassification.from_pretrained(model_name, num_labels=len(label_dict)).to(device)

training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=4,              # total number of training epochs
    per_device_train_batch_size=10,  # batch size per device during training
    per_device_eval_batch_size=4,   # batch size for evaluation
    #warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    load_best_model_at_end=True,     # load the best model when finished training 
    logging_steps=100,               # log &amp; save weights each logging_steps
    evaluation_strategy="steps",     # evaluate each `logging_steps`
    learning_rate=2e-5
)

trainer = Trainer(
    model="camembert-base",              # the instantiated Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=valid_dataset,          # evaluation dataset
    compute_metrics=compute_metrics,     # prepare the function to computes metrics of interest
)
</code></pre>
<p>At this point I have no error at all I check every tensor and every output? all go.<br>
But when I come to the final step of training, the model keeps running with no end</p>
<pre><code class="lang-auto"># train the model
trainer.train()
</code></pre>
<p>and display this…</p>
<pre><code class="lang-auto">***** Running training *****
  Num examples = 61601
  Num Epochs = 4
  Instantaneous batch size per device = 10
  Total train batch size (w. parallel, distributed &amp; accumulation) = 10
  Gradient Accumulation steps = 1
  Total optimization steps = 24644
Exception in thread Thread-15:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/parallel_loader.py", line 139, in _loader_worker
    _, data = next(data_iter)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "&lt;ipython-input-15-37abd243a789&gt;", line 7, in __getitem__
    item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}
  File "&lt;ipython-input-15-37abd243a789&gt;", line 7, in &lt;dictcomp&gt;
    item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}
IndexError: list index out of range
</code></pre>
<p>I couldn’t interpret the exception nor the error.<br>
Can anyone help me through this, please?</p>

ptrblck: <p>The error points towards an invalid indexing operation in:</p>
<pre><code class="lang-python">item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}
</code></pre>
<p>which fails in the data loading step.<br>
I would recommend to iterate the <code>Dataset</code> and <code>DataLoader</code> for a full epoch and make sure all samples can be loaded properly.</p>

MABtt: <p>Thank you for the update but I think there is more to do than that because I found out that the process of tokenizing takes more than 2H just to tokenize a dataset of 60K tweets which is a very bad this to do.</p>
<pre><code class="lang-auto">train_encodings = tokenizer(train_texts, truncation=True, padding=True, max_length=512)
valid_encodings = tokenizer(valid_texts, truncation=True, padding=True, max_length=512)
</code></pre>
<p>Can you tell me if there is another way to tokenize the data with less time, please?<br>
All the methods that I found and worked with the TrainingArguments and the Trainer do a simple <code>tokenizer</code> for a small data and wok well but with my data, it takes more than 2H <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"></p>
<hr>
<p>EDIT: It would be VERY helpful if you could tell me more about your proposed solution, I didn’t get what do you mean by</p>
<blockquote>
<p>I would recommend to iterate the <code>Dataset</code> and <code>DataLoader</code> for a full epoch and make sure all samples can be loaded properly.</p>
</blockquote>

ptrblck: <p>I don’t know if there are faster tokenizer, but would check what other repositories are using (e.g. HuggingFace).</p>
<aside class="quote no-group" data-username="MABtt" data-post="3" data-topic="132028">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/eb8c5e/48.png" class="avatar"> MABtt:</div>
<blockquote>
<p>It would be VERY helpful if you could tell me more about your proposed solution, I didn’t get what do you mean by</p>
</blockquote>
</aside>
<p>Something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">for data, target in dataset:
    print(data.shape)

for data, target in loader:
    print(data.shape)
</code></pre>
<p>This would iterate the <code>Dataset</code> and <code>DataLoader</code> for an entire epoch and could be used to isolate the indexing issue.</p>

MABtt: <p>Hello,</p>
<blockquote>
<p>I don’t know if there are faster tokenizer, but would check what other repositories are using (e.g. HuggingFace).</p>
</blockquote>
<p>Thank you for your reply. I appreciate your efforts <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>This would iterate the <code>Dataset</code> and <code>DataLoader</code> for an entire epoch and could be used to isolate the indexing issue.</p>
</blockquote>
<p>It still running very slow which is not very practical so I am going to change my strategy!<br>
Thanks anyway!</p>

ptrblck: <p>The test of iterating the <code>Dataset</code> and/or the <code>DataLoader</code> is only a debugging step to isolate the failure and you wouldn’t need to use it in your actual training script.</p>

MABtt: <p>Yes yes, I figured that out <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"><br>
I change all my script! and got more problems now haha<br>
Thanks anyway <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>