Not able to fine tune language model
Author: ghoshmithun
Pull request: no

I am trying to fine tune a language model using sagemaker huggingface API

I am using the code

```
import sagemaker
from sagemaker.huggingface import HuggingFace

# gets role for executing training job
role = sagemaker.get_execution_role()

hyperparameters = {
    'model_name_or_path':'EleutherAI/gpt-neo-1.3B',
    'output_dir':'/opt/ml/model' ,
    'data_dir': container_data_train,
    'output_dir': container_model_dir
    # add your remaining hyperparameters
    # more info here https://github.com/huggingface/transformers/tree/v4.4.2/examples/language-modeling
}

# git configuration to download our fine-tuning script
git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.4.2'}

container_data_train = '/opt/ml/input/data/training'
container_data_test = '/opt/ml/input/data/testing'
container_model_dir = '/opt/ml/model'

# creates Hugging Face estimator
huggingface_estimator = HuggingFace(
    entry_point='run_mlm.py',
    source_dir='./examples/language-modeling',
    instance_type='ml.p3.2xlarge',
    instance_count=1,
    role=role,
    git_config=git_config,
    transformers_version='4.4.2',
    pytorch_version='1.6.0',
    py_version='py36',
    hyperparameters = hyperparameters
)

# starting the train job
huggingface_estimator.fit({'train':'s3://train-data-gpt/'})
```

The input data is there. It is a .txt file with english sentences in each line. However after any number of try I am getting error as 

**File "run_mlm.py", line 170, in __post_init__
    raise ValueError("Need either a dataset name or a training/validation file.")
ValueError: Need either a dataset name or a training/validation file.**

However the training job was launched as here is the log:

```
Training Env:

{
    "additional_framework_parameters": {},
    "channel_input_dirs": {
        "train": "/opt/ml/input/data/train"
    },
    "current_host": "algo-1",
    "framework_module": "sagemaker_pytorch_container.training:main",
    "hosts": [
        "algo-1"
    ],
    "hyperparameters": {
        "output_dir": "/opt/ml/model",
        "model_name_or_path": "EleutherAI/gpt-neo-1.3B",
        "data_dir": "/opt/ml/input/data/training"
    },
    "input_config_dir": "/opt/ml/input/config",
    "input_data_config": {
        "train": {
            "TrainingInputMode": "File",
            "S3DistributionType": "FullyReplicated",
            "RecordWrapperType": "None"
        }
    },
    "input_dir": "/opt/ml/input",
    "is_master": true,
    "job_name": "huggingface-pytorch-training-2021-05-24-06-11-02-967",
    "log_level": 20,
    "master_hostname": "algo-1",
    "model_dir": "/opt/ml/model",
    "module_dir": "s3://sagemaker-us-west-2-928825629101/huggingface-pytorch-training-2021-05-24-06-11-02-967/source/sourcedir.tar.gz",
    "module_name": "run_mlm",
    "network_interface_name": "eth0",
    "num_cpus": 8,
    "num_gpus": 1,
    "output_data_dir": "/opt/ml/output/data",
    "output_dir": "/opt/ml/output",
    "output_intermediate_dir": "/opt/ml/output/intermediate",
    "resource_config": {
        "current_host": "algo-1",
        "hosts": [
            "algo-1"
        ],
        "network_interface_name": "eth0"
    },
    "user_entry_point": "run_mlm.py"
}

Environment variables:

SM_HOSTS=["algo-1"]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={"data_dir":"/opt/ml/input/data/training","model_name_or_path":"EleutherAI/gpt-neo-1.3B","output_dir":"/opt/ml/model"}
SM_USER_ENTRY_POINT=run_mlm.py
SM_FRAMEWORK_PARAMS={}
SM_RESOURCE_CONFIG={"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"}
SM_INPUT_DATA_CONFIG={"train":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}}
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=["train"]
SM_CURRENT_HOST=algo-1
SM_MODULE_NAME=run_mlm
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=8
SM_NUM_GPUS=1
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://sagemaker-us-west-2-928825629101/huggingface-pytorch-training-2021-05-24-06-11-02-967/source/sourcedir.tar.gz
SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"train":"/opt/ml/input/data/train"},"current_host":"algo-1","framework_module":"sagemaker_pytorch_container.training:main","hosts":["algo-1"],"hyperparameters":{"data_dir":"/opt/ml/input/data/training","model_name_or_path":"EleutherAI/gpt-neo-1.3B","output_dir":"/opt/ml/model"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"train":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"huggingface-pytorch-training-2021-05-24-06-11-02-967","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-us-west-2-928825629101/huggingface-pytorch-training-2021-05-24-06-11-02-967/source/sourcedir.tar.gz","module_name":"run_mlm","network_interface_name":"eth0","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"eth0"},"user_entry_point":"run_mlm.py"}
SM_USER_ARGS=["--data_dir","/opt/ml/input/data/training","--model_name_or_path","EleutherAI/gpt-neo-1.3B","--output_dir","/opt/ml/model"]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_CHANNEL_TRAIN=/opt/ml/input/data/train
SM_HP_OUTPUT_DIR=/opt/ml/model
SM_HP_MODEL_NAME_OR_PATH=EleutherAI/gpt-neo-1.3B
SM_HP_DATA_DIR=/opt/ml/input/data/training
PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages

Invoking script with the following command:

/opt/conda/bin/python3.6 run_mlm.py --data_dir /opt/ml/input/data/training --model_name_or_path EleutherAI/gpt-neo-1.3B --output_dir /opt/ml/model
```

@philschmid @sgugger 

Comments:
Hey @ghoshmithun, 
When you are using the `examples/` / "use sagemaker" with custom data on s3 `huggingface_estimator.fit({'train':'s3://train-data-gpt/'})`. 
You need to provide the hyperparameter `train_file` with the path to your file from s3. In your case, this would be `/opt/ml/input/data/train/my_train_file.csv`. 
[reference to `train_file` parameter defined in the `run_mlm.py`](https://github.com/huggingface/transformers/blob/6da129cb3152d93c425aab08a92d68c99e09d252/examples/pytorch/language-modeling/run_mlm.py#L114)  
[documentation for language-modelling](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling#language-model-training)

P.S. Not sure if `masked language modeling` is the preferred task for `GPT-NEO`. I think it is`causal language modeling` as for `GPT-2`



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.