Repository: huggingface/transformers

Issue:
[examples/text-classification] `do_predict` for the test set of local datasets
# 🚀 Feature request

It seems that `run_glue.py` has the train set and the validation set management for local CSV/JSON files, but it doesn't have args for managing the test set of the local datasets.

https://github.com/huggingface/transformers/blob/7a9f1b5c99e9a5d1772649d029acdf5160419239/examples/text-classification/run_glue.py#L90-L95

I think the script is intended to be used not only for the train/validation but the test, as `glue` tasks test sets are downloaded as shown in https://huggingface.co/docs/datasets/loading_datasets.html#selecting-a-configuration.
It has the `--do_predict` option for the test sets.

If there is no particular reason for not having the ability to read the test set in the local dataset, would it be ok for me to add the feature?
Or is there some intention behind this implementation?

## Motivation

I'd like to train, validate, and test my own local dataset.

## Your contribution

I think some modifications like the below may help to add the feature.

``` python
    test_file: Optional[str] = field(
        default=None, metadata={"help": "A csv or a json file containing the test data."}
    )
```

``` python
        datasets = load_dataset(
            "csv", data_files={"train": data_args.train_file, "validation": data_args.validation_file, "test": data_args.test_file}
        )
```

``` python
    # if data_args.task_name is not None:
    #     test_dataset = datasets["test_matched" if data_args.task_name == "mnli" else "test"]
    test_dataset = datasets["test_matched" if data_args.task_name == "mnli" else "test"]
```

Thank you in advance.



Discussion:
As long as it's kept simple (we want short and focused example scripts, so that they are easy to understand and tweak), I don't mind adding this feature. Feel free to open a PR with your suggestions!
Thanks, I'll try a modification as simple as possible, and if it can fix this issue without making the example difficult to understand, I’ll open a PR!

Resolving patch:
diff --git a/examples/text-classification/run_glue.py b/examples/text-classification/run_glue.py
--- a/examples/text-classification/run_glue.py
+++ b/examples/text-classification/run_glue.py
@@ -93,6 +93,7 @@ class DataTrainingArguments:
     validation_file: Optional[str] = field(
         default=None, metadata={"help": "A csv or a json file containing the validation data."}
     )
+    test_file: Optional[str] = field(default=None, metadata={"help": "A csv or a json file containing the test data."})
 
     def __post_init__(self):
         if self.task_name is not None:
@@ -102,10 +103,12 @@ def __post_init__(self):
         elif self.train_file is None or self.validation_file is None:
             raise ValueError("Need either a GLUE task or a training/validation file.")
         else:
-            extension = self.train_file.split(".")[-1]
-            assert extension in ["csv", "json"], "`train_file` should be a csv or a json file."
-            extension = self.validation_file.split(".")[-1]
-            assert extension in ["csv", "json"], "`validation_file` should be a csv or a json file."
+            train_extension = self.train_file.split(".")[-1]
+            assert train_extension in ["csv", "json"], "`train_file` should be a csv or a json file."
+            validation_extension = self.validation_file.split(".")[-1]
+            assert (
+                validation_extension == train_extension
+            ), "`validation_file` should have the same extension (csv or json) as `train_file`."
 
 
 @dataclass
@@ -205,16 +208,33 @@ def main():
     if data_args.task_name is not None:
         # Downloading and loading a dataset from the hub.
         datasets = load_dataset("glue", data_args.task_name)
-    elif data_args.train_file.endswith(".csv"):
-        # Loading a dataset from local csv files
-        datasets = load_dataset(
-            "csv", data_files={"train": data_args.train_file, "validation": data_args.validation_file}
-        )
     else:
-        # Loading a dataset from local json files
-        datasets = load_dataset(
-            "json", data_files={"train": data_args.train_file, "validation": data_args.validation_file}
-        )
+        # Loading a dataset from your local files.
+        # CSV/JSON training and evaluation files are needed.
+        data_files = {"train": data_args.train_file, "validation": data_args.validation_file}
+
+        # Get the test dataset: you can provide your own CSV/JSON test file (see below)
+        # when you use `do_predict` without specifying a GLUE benchmark task.
+        if training_args.do_predict:
+            if data_args.test_file is not None:
+                train_extension = data_args.train_file.split(".")[-1]
+                test_extension = data_args.test_file.split(".")[-1]
+                assert (
+                    test_extension == train_extension
+                ), "`test_file` should have the same extension (csv or json) as `train_file`."
+                data_files["test"] = data_args.test_file
+            else:
+                raise ValueError("Need either a GLUE task or a test file for `do_predict`.")
+
+        for key in data_files.keys():
+            logger.info(f"load a local file for {key}: {data_files[key]}")
+
+        if data_args.train_file.endswith(".csv"):
+            # Loading a dataset from local csv files
+            datasets = load_dataset("csv", data_files=data_files)
+        else:
+            # Loading a dataset from local json files
+            datasets = load_dataset("json", data_files=data_files)
     # See more about loading any type of standard or custom dataset at
     # https://huggingface.co/docs/datasets/loading_datasets.html.
 
@@ -325,7 +345,7 @@ def preprocess_function(examples):
 
     train_dataset = datasets["train"]
     eval_dataset = datasets["validation_matched" if data_args.task_name == "mnli" else "validation"]
-    if data_args.task_name is not None:
+    if data_args.task_name is not None or data_args.test_file is not None:
         test_dataset = datasets["test_matched" if data_args.task_name == "mnli" else "test"]
 
     # Log a few random samples from the training set:


Test patch: