text: 
  
  TITLE
  Jupyter Notebook Kernel crashes when tokenizing large dataset
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  I am using 2 setups, my personal laptop and a cluster.
  
  My laptop has this environment : 
  
   `transformers` version: 4.1.1
  - Platform: Darwin-20.2.0-x86_64-i386-64bit
  - Python version: 3.7.6
  - PyTorch version (GPU?): 1.7.1 (False)
  - Tensorflow version (GPU?): 2.2.0 (False)
  - Using GPU in script?: No
  - Using distributed or parallel set-up in script?: No
  
  The cluster has this : 
  
  - `transformers` version: 4.1.1
  - Platform: Linux-3.10.0-1160.6.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
  - Python version: 3.7.1
  - PyTorch version (GPU?): 1.7.1 (False)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: No
  - Using distributed or parallel set-up in script?: No
  
  ## Information
  
  Model I am using (Bert, XLNet ...): DistilBERT
  
  The problem arises when using:
  * [ ] the official example scripts: (give details below)
  * [X] my own modified scripts: (give details below)
  
  I followed [this example](https://huggingface.co/transformers/custom_datasets.html?highlight=custom%20datasets), but I modified the dataset part to include the one that I am using, which is described below. 
  
  The tasks I am working on is:
  * [ ] an official GLUE/SQUaD task: (give the name)
  * [X] my own task or dataset: (give details below)
  
  http://help.sentiment140.com/for-students
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  1. Use this script
  ```python
  from transformers import DistilBertTokenizerFast
  from sklearn.model_selection import train_test_split
  import pandas as pd
  
  # Run these first :
  # $ wget http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip
  # $ unzip trainingandtestdata.zip -d ./data
  # $ rm trainingandtestdata.zip
  
  
  def get_data(path):
      # Read the dataset
      df = pd.read_csv(path, encoding='ISO-8859-1', header=None, nrows=None)
  
      # Keep only the label and the text, replace 4 with 1
      # Note: there are actually no neutral labels in the train dataset
      df = df[[0, 5]].replace(2, 1).replace(4, 1)
  
      # Rename
      df = df.rename(columns={0: "label", 5: "text"})
  
      return df
  
  
  dftrain = get_data('data/training.1600000.processed.noemoticon.csv')
  dftest = get_data('data/testdata.manual.2009.06.14.csv')
  
  X_train = dftrain['text'].to_list()
  y_train = dftrain['label'].to_list()
  X_test = dftest['text'].to_list()
  y_test = dftest['label'].to_list()
  
  # Comment this to use full dataset
  # _,X_train, _, y_train = train_test_split(X_train, y_train, test_size=0.05, random_state=1)
  X_train, X_val, y_train, y_val = train_test_split(
      X_train, y_train, test_size=0.25, random_state=1)
  
  tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')
  train_encodings = tokenizer(X_train, truncation=True, padding=True)
  
  # To check the memory used
  with open('output.txt', 'w') as f:
      print(str(train_encodings))
  ```
  2. python tokenize_test.py
  
  ## Description
  
  In total, I tried this script in 4 different settings : 
  1. Personal laptop as a python script
  2. Personal laptop in a Jupyter Notebook
  3. Cluster as a python script
  4. Cluster in a Jupyter Book
  
  In case 2, 4, the kernel died. I assume it is because of memory error since case 3 was killed as a process because I exceeded the memory usage. However, case 1 worked flawlessly. The training dataset is about 90 MB in total and weighs 1.6 GB after tokenization. My personal laptop has 16 GB of RAM and I reserve 4 GB of RAM in the cluster. The memory that ought to be used is clearly below the limit yet I still get memory issues. Maybe is there a memory leak in a specific version somewhere?
  