Repository: flairNLP/flair

Issue:
TransformerWordEmbeddings does not support Transformer-XL model
**Describe the bug**
`TransformerWordEmbeddings` does not support Transformer-XL model.

**To Reproduce**
```python
from flair.embeddings import TransformerWordEmbeddings
from flair.data import Sentence

model = TransformerWordEmbeddings("transfo-xl-wt103", "all", "mean")
model.embed(Sentence("tiago"))
```

Output is:
```python
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    model.embed(Sentence("tiago"))
  File "/home/tiago/.local/lib/python3.8/site-packages/flair/embeddings/base.py", line 60, in embed
    self._add_embeddings_internal(sentences)
  File "/home/tiago/.local/lib/python3.8/site-packages/flair/embeddings/token.py", line 861, in _add_embeddings_internal
    self._add_embeddings_to_sentences(batch)
  File "/home/tiago/.local/lib/python3.8/site-packages/flair/embeddings/token.py", line 972, in _add_embeddings_to_sentences
    hidden_states = self.model(input_ids, attention_mask=mask)[-1]
  File "/home/tiago/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'attention_mask'
```

**Expected behavior**
Expected to support as it looks like it should be the class to concentrate all Transformer based models.

**Screenshots**
not relevant.

**Environment (please complete the following information):**
 - OS [e.g. iOS, Linux]: Linux
 - Version [e.g. flair-0.3.2]: 0.6.1


Discussion:

Resolving patch:
diff --git a/flair/embeddings/token.py b/flair/embeddings/token.py
--- a/flair/embeddings/token.py
+++ b/flair/embeddings/token.py
@@ -9,7 +9,7 @@
 
 import torch
 from bpemb import BPEmb
-from transformers import XLNetTokenizer, T5Tokenizer, GPT2Tokenizer, AutoTokenizer, AutoConfig, AutoModel
+from transformers import TransfoXLTokenizer, XLNetTokenizer, T5Tokenizer, GPT2Tokenizer, AutoTokenizer, AutoConfig, AutoModel
 
 import flair
 import gensim
@@ -864,6 +864,8 @@ def __init__(
             self.begin_offset = 0
         if type(self.tokenizer) == GPT2Tokenizer:
             self.begin_offset = 0
+        if type(self.tokenizer) == TransfoXLTokenizer:
+            self.begin_offset = 0
 
     def _add_embeddings_internal(self, sentences: List[Sentence]) -> List[Sentence]:
         """Add embeddings to all words in a list of sentences."""
@@ -984,10 +986,13 @@ def _add_embeddings_to_sentences(self, sentences: List[Sentence]):
             sequence_length = len(sentence)
             input_ids[s_id][:sequence_length] = sentence
             mask[s_id][:sequence_length] = torch.ones(sequence_length)
-
+        
         # put encoded batch through transformer model to get all hidden states of all encoder layers
-        hidden_states = self.model(input_ids, attention_mask=mask)[-1]
-        # make the tuple a tensor; makes working with it easier.
+        if type(self.tokenizer) == TransfoXLTokenizer:
+            hidden_states = self.model(input_ids)[-1]
+        else:
+            hidden_states = self.model(input_ids, attention_mask=mask)[-1]   # make the tuple a tensor; makes working with it easier.
+
         hidden_states = torch.stack(hidden_states)
 
         sentence_idx_offset = 0


Test patch: