Calling Inference API returns input text
Author: gstranger
Pull request: no

- `transformers` version: 4.4.0dev0
- Platform: MACosx
- Python version: 3.7
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Library:

@patrickvonplaten
@LysandreJik
@sgugger


## Information

Model I am using (Bert, XLNet ...): TransformerXL

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Upload private model to hub
2. Follow the tutorial around calling the Inference API (pasted below)

I've trained a `TransformerXLLMHeadModel` and am using the equivalent tokenizer class `TransfoXLTokenizer` on a custom dataset. I've saved both of these classes and have verified that loading the directory using the Auto classes succeeds and that the model and tokenizer are usable. When attempting to call the Inference API, I only get back my input text.

### Specific code from tutorial
```python
import json
import requests

API_TOKEN = "api_1234"
API_URL = "https://api-inference.huggingface.co/models/private/model"
headers = {"Authorization": f"Bearer {API_TOKEN}"}

def query(payload):
    data = json.dumps(payload)
    response = requests.request("POST", API_URL, headers=headers, data=data)
    return json.loads(response.content.decode("utf-8"))

data = query({"inputs": "Begin 8Bars ", "temperature": .85, "num_return_sequences": 5})
# data = [{'generated_text': 'Begin First '}]
```

## Expected behavior

When calling the Inference API on my private model I would expect it to return additional output rather than just my input text.


Comments:
Could you try to add `"max_length": 200` to your payload ? (also cc @Narsil )

Hi @gstranger ,

Can you reproduce the problem locally ? It could be that your model simply produces EOS token with high probability (leading to having the exact same prompt as output)

If not, do you mind telling us your username by DM `api-entreprise@huggingface.co` so we can investigate this issue ?

Hello @Narsil ,

When I run the same model locally, without either `max_length` or `min_length` I receive additional output, typically it will generate about 20-40 tokens. Also @patrickvonplaten when I add either that parameter or the `min_length` parameter the model still returns in less than a second with the same input text. I've sent an email with additional information. 

Hi @gstranger,

It does seem like a `max_length`. Your config defines it as `20` which is not long enough for the prompt that gets automatically added (because it's a transfo-xl model): https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines/text_generation.py#L23

You can override both `max_length` and `prefix` within the config to override the default `transfo-xl` behavior (depending on how it was trained it might lead to significant perf boost, or loss).

By default the API will read the config first. Replying also with more information by email with more information. Just sharing here so that the community can get help too.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.