Request on application/pipeline for Text Regression
Author: wanyu2018umac
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

The Text Regression task aims at delivering scalar output for each input sequence, which is widely used in NLP applications.

In my own use, I'm working on the research topic named translation evaluation, including metric and quality estimation (QE) tasks. Here are some available baselines:
- [BLEURT](https://aclanthology.org/2020.acl-main.704/)
- [COMET](https://aclanthology.org/2020.emnlp-main.213/)
- [TransQuest](https://aclanthology.org/2020.coling-main.445/)

Those models are designed on available pretrained language models (PLMs, like BERT, mBERT and XLM-R). Besides, those translation evaluation models consist of a multi-layer perceptron (MLP) at the end, which receives pooled hidden states and deliver a scalar output as the final prediction to describe how well the generated translation output express the semantic of referred sentence (source sentence for QE task, and reference for metric).

The Text Regression task is mostly related to Text Classification, where two things raise my concern:
- **The number of linear modules in MLP** Available Text Classification pipeline only makes one or two linear module available inside MLP, whereas modern approaches involve multiple linear modules (sometimes more than 2), and activate functions are arranged between any adjacent two of them.
- **Output of model** The output of Text Classification is a discrete output, e.g. 0 or 1 for binary classification. The output of Text Regression is a scalar.

So if it's possible, I wish that both features can be well supported:
- Increase the adaptability of related MLP module design, so that more models can be uploaded to Huggingface Hub and easy to be used. Coding can be referred to [COMET github repository](https://github.com/Unbabel/COMET/blob/c772b679e20725e6cc79b2107d50594f9ea7a4ae/comet/modules/feedforward.py);
- Create a new task named Text Regression, where it meets the demand of related translation evaluation approaches, as well as other applications in the forthcoming. I think the source code should be very easy to be implemented via transferring many things from Text Classification part.

Finally, that would be very appreciated if Text Regression model can benefit from the MLP with multiple linear modules.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

Recently I'm working on the repository of my own work, which is also a model-based translation evaluation approach like COMET and TransQuest. Personally I want to make my approach public and easy to use.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md -->

I'm very glad to submit a PR and help develop the related functionalities.


Comments:
@Narsil 

Here are related links on this Request:
[Huggingface Discuss](https://discuss.huggingface.co/t/demand-on-text-regression-pipeline-application/15935)
[Huggingface Doc](https://github.com/huggingface/hub-docs/issues/65)

Tagging @ricardorei, the 1st author and main developer of [COMET](https://github.com/Unbabel/COMET) -- he is also interested in HF <> COMET interoperability

Yes, I would like to work on this.

It would be useful to make COMET models and BLEURT models available through hugging face.

Also, it would help people develop new metrics!

Maybe a first step would be to add COMET/BLEURT as a model? 

 


Hi Pinging a core maintainer for feasibility 

@LysandreJik .

**Adding a model is a task on its own and is a necessary preliminary step to adding it to a pipeline.**

Pipelines work on any model that implement `AutoModelForXXX`  (XXX being `TextRegression` maybe here), because all those models will have a fixed API to be called on (regardless of implementations details like number of MLP in the heads).

Then adding a pipeline is described here: https://huggingface.co/docs/transformers/add_new_pipeline
I can help once we've started that stage.

> Output of model The output of Text Classification is a discrete output, e.g. 0 or 1 for binary classification. The output of Text Regression is a scalar.

This is actually not true, the output of a text classification model is a `N` classes tensor (floats) that is usually `softmaxed` and the max is then taken (but that comes later, the model really outputs N floats).



Hi all,

Great thanks to @gante @ricardorei !

Thanks to @Narsil for pointing out my misunderstanding on Text Classification pipeline.

So... it seems that the first step should be adding this TextRegression pipeline into huggingface? Then @ricardorei and I can prepare our own models to upload to huggingface. I can try developing TextRegression pipeline recently.






> So... it seems that the first step should be adding this TextRegression pipeline into huggingface? Then @ricardorei and I can prepare our own models to upload to huggingface. I can try developing TextRegression pipeline recently.

The first step is actually getting the models into `transformers` and adding the corresponding AutoModelForXXX.
I am not 100% familiar with that process, but I think what you did in documenting where the models come from, the research associated is a good first step.

I wonder if what was implemented here wasn't sufficient for the purpose: https://github.com/huggingface/transformers/pull/8328

The text classification pipeline adapts to the number of labels, and adapts either a sigmoid, a softmax, or no function at all. Is it insufficient for what you want to do here, and if so, can it be adapted to support your use case in a way that may be leveraged by other users?

Hi thanks to @Narsil @LysandreJik, I check the doc on recent topics about SequenceClassification. Take XLMRoBERTa as an example, I noticed that, the output of ```XLMRoBERTaForSequenceClassification``` can contain the logits before classification. I think the ```logits``` in this output format can suit my use.

The difference lies in the MLP of ```XLMRoBERTaForSequenceClassification```, as it inherits ```RobertaClassificationHead``` class. It contains two linear layers, and it takes the hidden states at the first place as input. For some modern approaches (like COMET), the number of linear layers may be not two, and the pooling strategy can be average pooling rather than first-place pooling.

Besides the functionalities of used PLM, I think for my model named ```XXX```, I have several solutions:

1. All I need to do additionally is to provide a new MLP class named ```XXXClassificationHead``` for the functionalities of my model. The ```XXX``` model can be applied in existing ```TextClassification``` pipeline, and I can directly collect the logits of output as the real output;
2. Design a new pipeline aside from ```TextClassification```. The new pipeline named ```TextRegression``` can receive related model classes named like ```XXXForSequenceRegression```;
3. ```TextClassification``` and ```TextRegression``` can somewhat share a great content. Maybe ```TextRegression``` can inherit from ```TextClassification```, or design a new parent class (maybe ```TextLogitsPrediction```) where ```TextClassification``` and ```TextRegression``` can both inherit from.

If we want a faster implementation, choice 1 is the best.
For a more precise definition in the forthcoming research and application, I think the difference between text classification and text regression can not be ignored, where choices 2&3 are recommended. We should not use the concept of classification instead of regression as this is a misdirection.


Hi all @Narsil @gante @LysandreJik,

Sorry for my late response, I've been working on something else recently.

I've uploaded the model files of mine, named [UniTE-UP](https://huggingface.co/ywan/unite-up/) and [UniTE-MUP](https://huggingface.co/ywan/unite-mup/), and I'm implementing the UniTEForSequenceClassification class for my models.

But here, if I want to use the from_pretrained method to initialize the model, like:
```
from transformers import UniTEForSequenceClassification
unite_up_model = UniTEForSequenceClassification.from_pretrained('unite-up')
unite_mup_model = UniTEForSequenceClassification.from_pretrained('unite-mup')
```

How should I collect related materials to make those commands work?

Great thanks!



Hi all,

I've uploaded the functionalities of my own model in the pull request. 

Also, I've uploaded my model named UniTE, and you can check the [paper link](https://arxiv.org/abs/2204.13346) and [existing repo](https://github.com/NLP2CT/UniTE). Personally, I can use my model like this:

```
# import models (take UniTE-MUP as an example)
>>> from transformers import UniTEForSequenceClassification, UniTETokenizerFast
>>> tokenizer = UniTETokenizerFast.from_pretrained('ywan/unite-mup')
>>> model = UniTEForSequenceClassification.from_pretrained('ywan/unite-mup')

# construct sources (src), references (ref) and hypotheses (hyp)
>>> src = tokenizer(['你好！', '很高兴认识你！'], return_tensors='pt', padding=True)
>>> ref = tokenizer(['Hello!', 'Nice to meet you!'], return_tensors='pt', padding=True)
>>> hyp = tokenizer(['Hi!', 'Nice to see you!'], return_tensors='pt', padding=True)

# evaluating with different input formats
>>> model(hyp=hyp, src=src).cpu().tolist()
[0.714469850063324, 0.6583192944526672]
>>> model(hyp=hyp, ref=ref).cpu().tolist()
[0.746547281742096, 0.7588061094284058]
>>> model(hyp=hyp, src=src, ref=ref).cpu().tolist()
[0.6857070326805115, 0.7172597050666809]
```

What should I need to do next? Great thanks!

@Narsil @LysandreJik @gante 

+1 on interest in application/pipeline (plus documenting!!!) Text Regression. While preparing for a workshop with movie reviews (1-5), I found this code:

```python
AutoModelForSequenceClassification.from_pretrained('model_name', num_labels=1, ignore_mismatched_sizes=True)
```

The word 'regression', and an example link should be on the same page as [AutoModelForSequenceClassification](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForSequenceClassification). Try searching [the main docs](https://huggingface.co/docs) for 'regression', or 'ignore_mismatched_sizes' and you will find very little to go on.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Text regression is a special case of text classification. Hence the existing classes for text classification are sufficient to train and do predictions for regression problems. As @mapmeld described, the right way to train a regression model is by passing `num_labels=1` to the `AutoModelForSequenceClassification.from_pretrained` method.

Coming to the pipeline for inferencing, it can be easily created like this
`pipe = pipeline('text-classification', model=trained_model_dir, function_to_apply="none")`

Remember to set the value of the parameter `function_to_apply` as `"none"` and not None.