Jupyter Notebook Kernel crashes when tokenizing large dataset
Author: lthiet
Pull request: no
Labels: name: wontfix

## Environment info
I am using 2 setups, my personal laptop and a cluster.

My laptop has this environment : 

 `transformers` version: 4.1.1
- Platform: Darwin-20.2.0-x86_64-i386-64bit
- Python version: 3.7.6
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

The cluster has this : 

- `transformers` version: 4.1.1
- Platform: Linux-3.10.0-1160.6.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.7.1
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): DistilBERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

I followed [this example](https://huggingface.co/transformers/custom_datasets.html?highlight=custom%20datasets), but I modified the dataset part to include the one that I am using, which is described below. 

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

http://help.sentiment140.com/for-students

## To reproduce

Steps to reproduce the behavior:

1. Use this script
```python
from transformers import DistilBertTokenizerFast
from sklearn.model_selection import train_test_split
import pandas as pd

# Run these first :
# $ wget http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip
# $ unzip trainingandtestdata.zip -d ./data
# $ rm trainingandtestdata.zip


def get_data(path):
    # Read the dataset
    df = pd.read_csv(path, encoding='ISO-8859-1', header=None, nrows=None)

    # Keep only the label and the text, replace 4 with 1
    # Note: there are actually no neutral labels in the train dataset
    df = df[[0, 5]].replace(2, 1).replace(4, 1)

    # Rename
    df = df.rename(columns={0: "label", 5: "text"})

    return df


dftrain = get_data('data/training.1600000.processed.noemoticon.csv')
dftest = get_data('data/testdata.manual.2009.06.14.csv')

X_train = dftrain['text'].to_list()
y_train = dftrain['label'].to_list()
X_test = dftest['text'].to_list()
y_test = dftest['label'].to_list()

# Comment this to use full dataset
# _,X_train, _, y_train = train_test_split(X_train, y_train, test_size=0.05, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=1)

tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')
train_encodings = tokenizer(X_train, truncation=True, padding=True)

# To check the memory used
with open('output.txt', 'w') as f:
    print(str(train_encodings))
```
2. python tokenize_test.py

## Description

In total, I tried this script in 4 different settings : 
1. Personal laptop as a python script
2. Personal laptop in a Jupyter Notebook
3. Cluster as a python script
4. Cluster in a Jupyter Book

In case 2, 4, the kernel died. I assume it is because of memory error since case 3 was killed as a process because I exceeded the memory usage. However, case 1 worked flawlessly. The training dataset is about 90 MB in total and weighs 1.6 GB after tokenization. My personal laptop has 16 GB of RAM and I reserve 4 GB of RAM in the cluster. The memory that ought to be used is clearly below the limit yet I still get memory issues. Maybe is there a memory leak in a specific version somewhere?


Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

i still see it's a problem.. running tokenizer

We can't really help you debug your script, you should probably ask for help on[ the forum ](https://discuss.huggingface.co/)