binary classification does not work with a large amount of data
Author: saeedrafieyan
Pull request: no

I'm trying to use binary classification, and I used a code like Minimal Start code in the simpletransformers.ai. I can get f1, tp> 0 with a small sample of my data (around 200K rows), but surprisingly, when I'm trying to apply the model to a whole dataset (2.6M rows) or less (500K rows), the evaluation of the model is not working very well. it returns mcc=0, tp=0, f1=0 . This is the case if the model works properly with fewer data and can predict correctly.

my code is here:

```
from simpletransformers.classification import ClassificationModel, ClassificationArgs
import pandas as pd
import logging
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score as f1
import torch

logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)

dataset = pd.read_csv(r"C:\Users\**.csv", encoding="utf-8")#, header=None)
dataset['labels'] = (def_dataset['labels'].astype(int))

train, test = train_test_split(dataset, train_size=0.8)

model_args = ClassificationArgs(num_train_epochs=1, train_batch_size=1, save_eval_checkpoints=False,
                                save_steps=2000000, overwrite_output_dir=True,
                                output_dir=r'C:\Users\***\test\output',
                                save_model_every_epoch=True,
                                )
cuda_available = torch.cuda.is_available()

# Create a ClassificationModel
model = ClassificationModel(
    "bert", "HooshvareLab/bert-fa-base-uncased", args=model_args, use_cuda=cuda_available
)

# Train the model
model.train_model(train)

# Evaluate the model
result, model_outputs, wrong_predictions = model.eval_model(test, f1=f1)
```

These are the results obtained with a semi-large amount of data(>=500K):

```
{'mcc': 0.0,
 'tp': 0,
 'tn': 77052,
 'fp': 0,
 'fn': 22948,
 'auroc': 0.5,
 'auprc': 0.22948,
 'f1': 0.0,
 'eval_loss': 1.0871533093261718}
```

and this is what I get with fewer data(200K):
```
{'mcc': 0.6321070718937202,
 'tp': 6193,
 'tn': 28748,
 'fp': 1925,
 'fn': 3134,
 'auroc': 0.9218176063608271,
 'auprc': 0.7718176609516296,
 'f1': 0.7100028661507596,
 'eval_loss': 0.31948030271530153}
```

The only difference between these two results is the size of the dataset.

I'm using windows 10 and Nvidia Quadro RTX 5000
I also tried on google colab, But the problem persisted.

How can I solve this problem?

Comments:
It won't be possible for us to answer issues with another library. Also please use the forum to such [questions](https://discuss.huggingface.co/). Thank you!

Thank you for your answer.

I asked this question here because the simple transformers are working on the transformers library. It is just an interface of it, so I thought my question relevant to this repo!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.