Turn of do_sample for T0pp in Inference API
Author: feyzaakyurek
Pull request: no

Hi,

I'm trying to use the Inference API with T0pp. I would like to set `do_sample=False` but it doesn't let me specify `parameters` as in the example [here](https://api-inference.huggingface.co/docs/python/html/detailed_parameters.html#summarization-task).

Code:

    API_URL = "https://api-inference.huggingface.co/pipeline/text2text-generation/bigscience/T0pp"
    headers = {"Authorization": f"Bearer {API_TOKEN}"}

    payload = {"inputs": "Titanic was built in", "parameters": {"do_sample":False}}
    data = json.dumps(payload)
    response = requests.request("POST", API_URL, headers=headers, data=data)
    ans = json.loads(response.content.decode("utf-8"))
    print(ans)

Output I'm getting:

    {'error': 'Parameters are not accepted for this specific model'}

My environment:

    - `transformers` version: 4.12.5
    - Platform: Linux-4.14.254-llgrid-10ms-x86_64-with-debian-buster-sid
    - Python version: 3.7.11
    - PyTorch version (GPU?): not installed (NA)
    - Tensorflow version (GPU?): not installed (NA)
    - Flax version (CPU?/GPU?/TPU?): not installed (NA)
    - Jax version: not installed
    - JaxLib version: not installed
    - Using GPU in script?: No
    - Using distributed or parallel set-up in script?: No

Thank you!

Comments:
cc @Narsil 

Hi @feyzaakyurek ,

Thanks. Currently (and in the short term) this won't be available.

`bigscience/T0pp` is a very large model (40Go) meaning we deploy it in a very specific way. Here we used `flax` and it's deployed on a TPU for instance. This enables quite fast inference speed but it has drawbacks, for instance, `flax` is much less dynamic than `pytorch` when it comes to inference (the graph needs to be compiled to get those fast inference times).

This is not something that will never be available, we started some work to make the pipelines (they power the API) support `flax` in a more integrated way. You can track progress here: https://github.com/huggingface/transformers/pull/14356. Hopefully we can bridge the gap and enable that later.

But the truth is that flax has made different choices than Pytorch, which means we might not support exactly the same arguments for both frameworks. or the same flexibility. But we'll try to.

Thank you so much for your prompt response, I understand. 

I'm on Pro Plan for the Inference API right now and does your answer suggest that there is no difference between Pro plan and one of the organization plans in terms of which processor is being used (i.e. it's always TPU)? Would the inference be any faster if I were to switch to Pay as you go?

https://huggingface.co/pricing

Thank you!

Hi @feyzaakyurek ,

If you are a paying user then we should probably start a discussion with api-enterprise@huggingface.co.
This model if part of the large model family which don't get deployed automatically and we usually start a discussion to understand your needs first. But this is definitely a good option where we can consider deploying with  custom parameters.

What I mentioned concerned `bigscience/T0pp` as currently deployed for everyone.

Thank you, have reached out to them.