Return raw outputs in TextClassificationPipeline
Author: LysandreJik
Pull request: yes

Currently the `TextClassificationPipeline` does a softmax over the output values when `num_labels > 1`, and does a sigmoid over the output when `num_labels == 1`.

As seen in https://github.com/huggingface/transformers/issues/8259, this may be problematic when systems depend on a different output range.

Adding a flag `return_raw_outputs` that will not apply the sigmoid or softmax in that case.

closes #8259

Comments:
Will that flag be configurable on a model-level for the hosted Inference API, and if yes, how?

Through the configuration, as specified in the original issue would be the easiest way to control it through the API. Haven't thought about it this far however.

The other option would be to disable the `sigmoid` over the output and let the user handle that instead, but this would break existing models on the inference API such as the DialogRPT model.

Here's a proposal @julien-c (see latest commit).

The current `TextClassificationPipeline` has a task set to `""` as it is not initialized from `pipeline(task)`.
This proposal puts the default `task` as `"text-classification"` for the `TextClassificationPipeline`. Since the pipeline will fetch the `task_specific_params` according to the given task, it can therefore fetch the following configuration option (which can of course already be set in the configuration on S3):

```py
config = GPT2Config.from_pretrained("microsoft/DialogRPT-updown", task_specific_params={"text-classification": {"return_raw_outputs": True}})
model = GPT2ForSequenceClassification.from_pretrained("microsoft/DialogRPT-updown", config=config)
```

This will be used as the default value for the pipeline, and allow users to specify pipeline-specific arguments directly in their model configuration.

The proposal was only made for the `TextClassificationPipeline`, but it would need to be made to all other pipelines to stay coherent. Let me know if this is an approach you would be interested in.


pinging @Narsil and @mfuntowicz for their thoughts on this (we have ways to get configurable params on the API, just wanted to make sure we could use them for this)

I've also run into this issue recently when uploading some multi-label text classification models to the HuggingFace hub, where it seems like the default activation for multiple classes is a softmax.

Having it return raw outputs is definitely useful, however, it still wouldn't show the scores I would expect in the inference api - in our case using a sigmoid over the results to allow more than one positive label?

Would it be useful to also have an argument like `output_nonlinearity` for the desired activation function e.g. sigmoid/softmax?


Hi @laurahanu, thanks for your proposal. I'll try and see how to best integrate that in this PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hello, was just wondering if there are any updates on allowing the user to choose which function to run on the outputs? It seems like in the current documentation, the pipeline would still only run a sigmoid over the result if there is one label.

> If multiple classification labels are available (model.config.num_labels >= 2), the pipeline will run a softmax over the results. If there is a single label, the pipeline will run a sigmoid over the result.


Sorry for taking a long time to merge this; I wonder if we can't re-use the recently introduced `problem_type` instead, wdyt @sgugger @abhi1thakur?

Yes, this flag is there for this reason specifically :-) 

Thanks for reopening this and having another look at it! Do you have a timeline in mind for when this would be merged/available to use with the model hub?

Hey @laurahanu, the PR should now be ready for merging. I'm pinging two team members for review; this should be merged in the coming days and deployed on the API shortly after.

Great, thank you @LysandreJik!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@LysandreJik is this ready to be merged?