using run_squad.py for predict and specifying config_name as path, config.json not found
Author: mfeblowitz
Pull request: no

## 🐛 Bug

<!-- Important information -->

The situation is that, when running a predict-only task and specifying 1) an explicit path for a fine-tuned albert model and 2) specifying a specific path to the corresponding config.json file, run_squad attempts to seek the config file in the location of the --output_dir.

It appears that, when specifying config_name as a full path, the specification is ignored and run_squad looks in the specified output_dir location for config.json. Since I am automatically generating several output dirs during the course of running my pipeline, it is not convenient nor sensible for me to also copy the config file to each result directory.

To be concrete, I am trying to specify that the config file should come from the training output model directory:

--config_name /home/.../albert_models/squad2/config.json
and that the eval/predict output should go to a result directory:

--output_dir /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0
When I do this I see the below.

Model I am using (Bert, XLNet....): albert, fine-tuned for SQuAD2 using albert xxl

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [X] the official example scripts: run_squad.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [2] an official GLUE/SQUaD task: SQuAD2
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. in transformers, run_squad.py, with arguments below

Also tried without --cache_dir, which also had the same effect.

The only thing that worked was to  use the model dir == the output dir, but that placed my outputs into the model dir, which is not acceptable (should be unaltered by predict-only tasks).

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
run_squad.py arguments:
```
 --model_type albert
--model_name_or_path  /home/.../albert_models/squad2
--cache_dir /home/.../transformers/cache_dir/v2.0-albert
--config_name  /home/.../albert_models/squad2/config.json
--do_eval
--do_lower_case
--predict_file /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0/questionnaire.json
--train_file None
--per_gpu_train_batch_size 2
--per_gpu_eval_batch_size 24
--learning_rate 3e-05
--num_train_epochs 2.0
'--max_seq_length 128
--doc_stride 128
--version_2_with_negative
--output_dir /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0/
```
Partial stack trace:
 ```
12/12/2019 19:18:24 - INFO - __main__ -   Evaluate the following checkpoints: ['/home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0'']                                                                                                                      Traceback (most recent call last):                                                                                                            File "/home/.../transformers/transformers/configuration_utils.py", line 134, in from_pretrained                                 resolved_config_file = cached_path(config_file, cache_dir=cache_dir, force_download=force_download, proxies=proxies)                      
File "/home/.../transformers/transformers/file_utils.py", line 182, in cached_path raise EnvironmentError("file {} not found".format(url_or_filename))
OSError: file /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0/config.json' not found                                                                                                                                                                    During handling of the above exception, another exception occurred: 
Traceback (most recent call last):                                                                                                          
  File "/home/.../transformers/transformers/configuration_utils.py", line 134, in from_pretrained                             
    resolved_config_file = cached_path(config_file, cache_dir=cache_dir, force_download=force_download, proxies=proxies)                    
  File "/home/.../transformers/transformers/file_utils.py", line 182, in cached_path                                          
    raise EnvironmentError("file {} not found".format(url_or_filename))                                                                     
OSError: file /home/.../pipeline_results/session_id_1234/output/workdirs/0.0.0'/config.json not found  
```
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
Expected that overriding the model_name_or_path with a direct path, and the config_name with another path (different from output_dir) would override the defaults.

## Environment

* OS: ubuntu 18.04.1
* Python version: 3.7.5
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 2.2.1, installed from source
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
@LysandreJik - Note changes above.

I believe the issue stems from the fact that the model cannot be evaluated unless it has been trained. A workaround is to specify the `model_name_or_path` to be the same as the `output_dir` so that it loads that when evaluating, but it isn't the best user experience.

I'm thinking of forcing the evaluation to load the model from `model_name_or_path` rather than from `output_dir` when there is no `do_train` argument is specified. What do you think?

That's exactly the situation, and exactly the behavior that I believe would work.
The training takes many hours (~20?) and I only want to do it once. Then I cache that in a central location and all of my many predict runs use that. 

Let me know if the latest commit (c8ed1c8) fixes this issue.

Well... no and yes. The "no" is that I had trouble with dependencies when running that commit ( issues with missing "past" module). The "yes" is that I was able to slot in the fix to the 2.1.1 version and the fix worked.

Not sure what my problems are with running both from master and from c8ed1c8. But at least there's a way forward. For unattended clone and operate, I'd either need the patch to be applied to 2.1.1, or I'll need some guidance about dependency issues with c8ed1c8. 

Thanks!

When you install from master (`pip install git+https://github.com/huggingface/transformers`) you run into issues? Do you mind copying the error message along with your software versions?

We'll push a patch (2.2.2) later today.

Works great. Don't know what was going wrong with my prior attempts. 

Thanks!!


Glad to hear that. Feel free to re-open if you have similar issues.