Add a new pipeline for the Relation Extraction task.
Author: xegulon
Pull request: no

# 🚀 Feature request

Add a new pipeline option for the Relation Extraction task : `nlp = pipeline('relation-extraction')`

## Motivation

Relation Extraction between named entities is a well-known NLP task. For example, when you get entities relative to medications (let's say our entity types are DRUG and FORM (tablet, capsule, etc.)), you want to know which FORM entity goes with which DRUG entity, etc.

Reference: https://portal.dbmi.hms.harvard.edu/projects/n2c2-2018-t2/
This task is not limited to the biomedical domain.

## Your contribution

I still need to play more with the HF API to contribute !

But, as I see it, the pipeline would return a list of dictionaries, each dictionary representing an identified relation in the text.

The relation extraction model would probably sit on top of the NER model.

There are implementations of such models [here](https://nlpprogress.com/english/relationship_extraction.html).


Comments:
We have a voluntarily generic `token-classification` pipeline that should be suited for this, no?

> We have a voluntarily generic `token-classification` pipeline that should be suited for this, no?

As far as I understood, `token-classification` is just an alias for `ner` (in the source code, we can observe: `NerPipeline = TokenClassificationPipeline`). 

The relation extraction part would be to classify pairs of entities (given by the `ner`/`token-classification` part of the pipeline) to a set of relation classes, such as `IS_CEO_OF_ORG`. 

I don't think it is possible to do this for now. Thanks for the reply!

This seems to be a popular repo for RE: https://github.com/thunlp/OpenNRE

For now, there's only 1 model that is capable of performing relation extraction out-of-the-box, and that's [LUKE](https://huggingface.co/transformers/model_doc/luke.html#overview). You can use `LukeForEntityPairClassification` to classify the relationship between two entities in a sentence:

```
from transformers import LukeTokenizer, LukeForEntityPairClassification

tokenizer = LukeTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred")
model = LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred")

text = "Beyoncé lives in Los Angeles."
entity_spans = [(0, 7), (17, 28)]  # character-based entity spans corresponding to "Beyoncé" and "Los Angeles"
inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])
```

However, relation extraction is a task that is solved in many different ways. So it's not straightforward to define a generic pipeline for it, in which you can plug different models.

Thanks @NielsRogge for your answer. I have 3 questions then:

1. What do you mean exactly by:
> relation extraction is a task that is solved in many different ways.

because the task of RE itself is quite standardized, isn't it?

2. Is the LUKE model you showed me usable with *any* dataset? If yes, which format of the dataset is needed?

3. Wouldn't be good to choose *one* approach (maybe SpanBERT?, cf [this](https://kr2ml.github.io/2020/papers/KR2ML_12_paper.pdf)) and implement it in the HF `pipeline`?

> 1. What do you mean exactly by:
> 
> > relation extraction is a task that is solved in many different ways.

NER is always solved in the same way (in the Transformers library, at least), namely by placing a token classification head on top of the final hidden states of the tokens. However, relation extraction can be solved in many ways. LUKE for example has a very specific way, namely it considers a word sequence (tokens) and an entity sequence (entities), and it places a linear layer on top of the concatenation of the entity tokens. Another model, like [R-BERT](https://arxiv.org/abs/1905.08284) for example, does it differently. From the paper: "(...) We apply the average operation to get a vector representation for each of the two target entities. Then after an activation operation (i.e. tanh), we add a fully connected layer to each of the two vectors (...)":

![image](https://user-images.githubusercontent.com/48327001/119819182-61f7f280-bef0-11eb-8b46-6eafd5cf11cc.png)

In other words, as every relation extraction model does it in a different way, it's not straightforward to define a general pipeline for it. 

> 2. Is the LUKE model you showed me usable with _any_ dataset? If yes, which format of the dataset is needed?

Yes, you can use it with any dataset. I fine-tuned it myself on a custom dataset. You just need to prepare a csv file with 4 columns: sentence, entity 1, entity 2, relationship. I will prepare a notebook that illustrates how you can do it easily. 

> 3\. Wouldn't be good to choose _one_ approach (maybe SpanBERT?, cf [this](https://kr2ml.github.io/2020/papers/KR2ML_12_paper.pdf)) and implement it in the HF `pipeline`?

A pipeline is meant to be used for several models, I don't think it's nice to have a pipeline that only works for a single model.

Thanks for all your answers @NielsRogge!

I understand it better now. In fact, the way you do it makes me think of the QA task, but here the context is replaced by the entity spans, and the output is the one of a `SequenceClassification` task.

Looks pretty good for becoming the standard :wink:!

@xegulon here's a notebook that illustrates how to fine-tune `LukeForEntityPairClassification` on a custom dataset for relation extraction: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LUKE/Supervised_relation_extraction_with_LukeForEntityPairClassification.ipynb

Thanks a lot @NielsRogge ! 
Hoping to see `pipeline('relation-classification')` and `pipeline('joint-ner-and-re')` someday ;) !

> @xegulon here's a notebook that illustrates how to fine-tune `LukeForEntityPairClassification` on a custom dataset for relation extraction: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LUKE/Supervised_relation_extraction_with_LukeForEntityPairClassification.ipynb

Thanks a lot @NielsRogge for this notebook. You saved me a lot of time! 
I have a doubt, a statement we're trying to annotate is:
Mukesh Ambani married Nita Ambani in 1985 and they have two sons, Akash and Anant, and a daughter, Isha.

There are multiple entities in one sentence and different relations between them.
How should i go about incorporating this in my dataset?

1. The sentence column will have the above statement multiple times until all relations and entities are captured. The entity and label columns will change as per entities.
2. Making this a multi label problem -- (which is more tricky)

Would love to know your approach on this. Thanks!

> How should i go about incorporating this in my dataset?

I think you need to create several training examples for this single sentence. Each training example should be <sentence, entity 1, entity 2, relationship>. So indeed, option 1 is what I would do. 

There are other approaches to relation extraction, in which one applies a binary classifier to each possible pair of entities (an example is [this paper](https://www.sciencedirect.com/science/article/abs/pii/S095741741830455X?via%3Dihub)). However, LUKE doesn't work that way.

> I think you need to create several training examples for this single sentence. Each training example should be <sentence, entity 1, entity 2, relationship>. So indeed, option 1 is what I would do.

I have gone ahead with the LUKE approach.

The [TACRED dataset](https://nlp.stanford.edu/pubs/zhang2017tacred.pdf) has 79.5% relation labels as 'no_relation'.
This seems logical because not every sentence consists of relations and also reduces false positives. (my model will be tested against newspaper articles, blogs, wiki text, etc)

I have two doubts:
1. Whilst making a custom dataset (like the one in your notebook) should we also include sentences that have no relations between entities? What percentage of no_relation labels would you suggest for our custom dataset ?

2. How should we go about labelling this sentence: (such sentences are common in news articles or excerpts from interviews)

"Pep Guardiola was unhappy with the passing during the game."

This has only one entity (entity1 = PERSON). Do we consider this sentence since entity2 would be empty?
We have been discarding these as of now.


> Whilst making a custom dataset (like the one in your notebook) should we also include sentences that have no relations between entities? What percentage of no_relation labels would you suggest for our custom dataset ?

Yes, for sure. In that way, you can let the model learn that there are also a lot of sentences where there's no relationship between 2 entities. Probably, the percentage of no_relation labels depends on your domain, but it will probably be the most occuring class.

> How should we go about labelling this sentence: (such sentences are common in news articles or excerpts from interviews)

I don't think you need to add sentences that only have a single entity, you can simply discard these.