[WIP] Add Few Shot Named Entity Recognition (FSNER) model
Author: sayef
Pull request: yes

# What does this PR do?

- This PR adds a new model FSNER (few shot named entity recognition) which has been implemented and trained based on the paper: [Example-Based Named Entity Recognition](https://arxiv.org/abs/2008.10570) from the researches in Microsoft Dynamics 365 AI.
- It includes only FSNERModel, not any other derivations, i.e., MaskedLM, ForQuestionAnswering etc.
- Doc strings are also updated, but not sure how it would appear visually.
- No additional tests are included.


<details><summary>Usage Example</summary>
<p>


```
from transformers import FSNERModel, FSNERTokenizerFast, FSNERTokenizer

device = 'cpu'
fsner_model = FSNERModel.from_pretrained("sayef/fsner-bert-base-uncased").to(device)
fsner_tokenizer = FSNERTokenizer.from_pretrained("sayef/fsner-bert-base-uncased")

# size of query and supports must be same. If you want to find all the entitites in one particular query, just repeat same query n times where n is the size of supports (or entities).


query = [
    'KWE 4000 can reach with a maximum speed from up to 450 P/min an accuracy from 50 mg',
    'I would like to order a computer from eBay.',
]

# each list in supports are the examples of one entity type

supports = [
        [
           'Horizontal flow wrapper [E] Pack 403 [/E] features the new retrofit-kit „paper-ON-form“',
           '[E] Paloma Pick-and-Place-Roboter [/E] arranges the bakery products for the downstream tray-forming equipment'
,
           'Finally, the new [E] Kliklok ACE [/E] carton former forms cartons and trays without the use of glue',
           'We set up our pilot plant with the right [E] FibreForm® [/E] configuration to make prototypes for your marketi
ng tests and package validation',
           'The [E] CAR-T5 [/E] is a reliable, purely mechanically driven cartoning machine for versatile application fiel
ds'
        ],
        [
            "[E] Walmart [/E] is a leading e-commerce company",
            "I recently ordered a book from [E] Amazon [/E]",
            "I ordered this from [E] ShopClues [/E]",
            "Fridge can be ordered in [E] Amazon [/E]",
            "[E] Flipkart [/E] started it's journey from zero"
        ]
   ]


def tokenize(x):
    return fsner_tokenizer(x, padding='max_length', max_length=384, truncation=True, return_tensors="pt")

W_query = tokenize(query).to(device)
W_supports = tokenize([s for support in supports for s in support]).to(device)

start_prob, end_prob = fsner_model.get_start_end_token_scores(W_query, W_supports)

output = fsner_tokenizer.extract_entity_from_scores(query, W_query, start_prob, end_prob, thresh=0.50)

print(output)

```


</p>
</details>




## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@LysandreJik @stas00 @sgugger

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Hi,

Thanks for your contribution! Looking at the code, I'm not sure whether there's a need to add an entire new model for it that includes a modeling file, tokenizer, etc., as the model itself is a just a BERT model, and one can just use `BertTokenizer` to prepare data for the model. The only differences are the `get_start_end_token_scores` and `extract_entity_from_scores` 
methods.

So personally, I'd opt to:
- upload the model's weights to the hub under the "microsoft" namespace. I see you've already uploaded them under your own name, so we can transfer or copy them for you.
- add a Colab notebook or script under the [research_projects](https://github.com/huggingface/transformers/tree/master/examples/research_projects) directory, that illustrates how FSNER works. 

Thanks for your kind reviews and replies. I would like to start with @NielsRogge 's comment. 

- The `get_start_end_token_scores `  method is actually the few shot prediction method. The training method is almost similar (which is not yet included), returning the entity start/end span probability and off-course the proposed loss function etc. Because of the extra special tokens i.e. [E], and [/E] the tokenizer has some modifications inside. Well, end users also can do that by themselves.

- On the other hand, the `extract_entity_from_scores` method is responsible for choosing the best spans from the start/end probabilities like answer span selection process in question answering task.

- At the end, we actually achieve a fine-tuned BERT model ready for few-shot named entity recognition task. 

Now, what I am confused about is, how you maintain/support a new model which is not a variation of transformer model, rather uses pretrained BERT model and fine-tunes on new task and data, for example, BertForQuestionAnswering. It would be best, in my opinion, if we could work on something like that, BertForFSNER or something similar.

It's totally okay for me to support the model in any format, i.e., new model, colab script or what I discussed above.

For your comprehension of the model, I am attaching the class I wrote when I started implementing the proposed architecture.

<details><summary>FSNER Prototype Code</summary>
<p>

```python
class FSNER(nn.Module):
    def __init__(self, model_name='bert-base-uncased'):
        super(FSNER, self).__init__()
        # declare bert tokenizer
        self.tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
        # add special tokens for enity boundaries
        self.tokenizer.add_special_tokens({'additional_special_tokens': ['[E]','[/E]']})
        
        # get enitity start and end token ids
        self.start_token_id, self.end_token_id = tuple(self.tokenizer.convert_tokens_to_ids(['[E]','[/E]']))
        
        # declare bert model
        self.bert = BertModel.from_pretrained(model_name, return_dict=True)
                
        # resize model token embeddings
        self.bert.resize_token_embeddings(len(self.tokenizer))

        # cosine sim
        self.cos = torch.nn.CosineSimilarity(3, 1e-08)
        
        # softmax
        self.softmax = torch.nn.Softmax(dim=1)
        
    def BERT(self, **inputs):
        return self.bert(**inputs).last_hidden_state
    
    def VectorSum(self, token_embeddings):
        return token_embeddings.sum(2, keepdim=True)
        
    def Atten(self, q_rep, S_rep, T=1):
        return self.softmax(T*self.cos(q_rep, S_rep))
    
    def tokenize(self, x):
        return self.tokenizer(x, padding='max_length', max_length=384, truncation=True, return_tensors="pt", return_offsets_mapping=True)
    
    def save(self):
        self.bert.save_pretrained('./fsner-bert-base-uncased/')

    def forward(self, W_query, W_supports):
        
        q = self.BERT(**W_query)
        S = self.BERT(**W_supports)
        
        # reshape from (batch_size, 384, 784) to (batch_size, 1, 384, 784)
        q = q.view(q.shape[0], -1, q.shape[1], q.shape[2])
        
        # reshape from (batch_size*n_exaples_per_entity, 384, 784) to (batch_size, n_exaples_per_entity, 384, 784)
        S = S.view(q.shape[0], -1, S.shape[1], S.shape[2])
                
        q_rep = self.VectorSum(q)
        S_rep = self.VectorSum(S)
        
        s_start = S[(W_supports['input_ids'] == self.start_token_id).view(S.shape[:3])].view(S.shape[0], -1, 1, S.shape[-1])
        s_end = S[(W_supports['input_ids'] == self.end_token_id).view(S.shape[:3])].view(S.shape[0], -1, 1, S.shape[-1])
        
        atten = self.Atten(q_rep, S_rep)
        
        P_start = torch.sum(atten * torch.einsum("bitf,bejf->bet", q, s_start), dim=1)
        P_end = torch.sum(atten * torch.einsum("bitf,bejf->bet", q, s_end), dim=1)
        
        
        return P_start, P_end
    
    def decode(self, ids, skip_special_tokens=True):
        return self.tokenizer.decode(ids, skip_special_tokens=skip_special_tokens) 
```
As you can see, the forward method takes in BERT pretrained model and does some calculations to provide span predictions. 
</p>
</details>



The [research_projects folder](https://github.com/huggingface/transformers/tree/master/examples/research_projects) is typically used for models that introduce a new technique or model based BERT. Examples are Performer, LXMERT, etc. 

As mentioned there, they are not actively maintained, one just needs to specify a requirements.txt file, together with a script or Colab notebook. So perhaps you can make a Colab notebook in which you define the `nn.Module` as shown in your prototype above, and illustrate how the model works to perform few-shot NER. You can also fill in the README of that folder as you like.

Does that work for you?



> The [research_projects folder](https://github.com/huggingface/transformers/tree/master/examples/research_projects) is typically used for models that introduce a new technique or model based BERT. Examples are Performer, LXMERT, etc.
> 
> As mentioned there, they are not actively maintained, one just needs to specify a requirements.txt file, together with a script or Colab notebook. So perhaps you can make a Colab notebook in which you define the `nn.Module` as shown in your prototype above, and illustrate how the model works to perform few-shot NER. You can also fill in the README of that folder as you like.
> 
> Does that work for you?

- Yeah, that works for me. I just want to keep the trained model's weights and tokenizer under my namespace, since they are not officially from Microsoft.  And I also plan to add other BERT variation based fsner. So, I would prefer to keep those under my namespace, if that's not an issue for you.
- So, should I/you close this PR and open a new PR with the suggested procedures you mentioned above?

> I just want to keep the trained model's weights and tokenizer under my namespace, since they are not officially from Microsoft. And I also plan to add other BERT variation based fsner. So, I would prefer to keep those under my namespace, if that's not an issue for you.

Makes sense!

> So, should I/you close this PR and open a new PR with the suggested procedures you mentioned above?

Yes, indeed. You can perhaps take a look at other research projects to get some inspiration :) 

Thanks for your help. Will talk to you in other PR.