Repository: flairNLP/flair

Issue:
Is TransformerDocumentEmbeddings sentence length is limited to 512 tokens? 
Hello, 

In the class _TransformerDocumentEmbeddings__ in the method __add_embeddings_to_sentences_ there is an indication that the sentence is truncated up to 512 tokens (max_length=512). 

> 
             # subtokenize sentences

            for sentence in sentences:
                # tokenize and truncate to 512 subtokens (TODO: check better truncation strategies)
                subtokenized_sentence = self.tokenizer.encode(sentence.to_tokenized_string(),
                                                              add_special_tokens=True,
                                                              max_length=512,
                                                              truncation=True,
                                                              )

So, is only 512 tokens are taken into account in order to calculate the sentence embeddings ? Or the whole document is divided on the sequences of 512 tokens and then all the document is parsed to get the final embedding? I mean, in the case if the document is longer than 512 tokens.

Another question, how the token is determined, is it a character, word, sub-word ? 

Thank for your help :)
And thanks for a developer team, very useful library :) 


Discussion:
I wanted to ask the same question, since `allow_long_sentences=True` was added as a flag to `TransformerWordEmbeddings`, is it possible to transfer this behaviour to TransformerDocumentEmbeddings?
I'm generating DocumentEmbeddings using `TransformerDocumentEmbeddings` in order to represent the Sentence using the CLS-Token and with `DocumentPoolEmbeddings` and `TransformerWordEmbeddings` in order to represent the sentence using mean pooling. It would be very helpful to be able to use longer sequences with both strategies.
For `TransformerDocumentEmbeddings` the text is truncated to 512 (usually) subtokens. It is not trivial to add `allow_long_sentences`.
I think we should allow to set a user-defined value (constructor?!). because e.g. the [LongFormer](https://arxiv.org/abs/2004.05150) model supports a sequence length up to 4096 subtokens.

Resolving patch:
diff --git a/flair/embeddings/document.py b/flair/embeddings/document.py
--- a/flair/embeddings/document.py
+++ b/flair/embeddings/document.py
@@ -110,12 +110,14 @@ def _add_embeddings_to_sentences(self, sentences: List[Sentence]):
 
             # subtokenize sentences
             for sentence in sentences:
-                # tokenize and truncate to 512 subtokens (TODO: check better truncation strategies)
+
+                # tokenize and truncate to max subtokens (TODO: check better truncation strategies)
                 subtokenized_sentence = self.tokenizer.encode(sentence.to_tokenized_string(),
                                                               add_special_tokens=True,
-                                                              max_length=512,
+                                                              max_length=self.tokenizer.model_max_length,
                                                               truncation=True,
                                                               )
+
                 subtokenized_sentences.append(
                     torch.tensor(subtokenized_sentence, dtype=torch.long, device=flair.device))
 


Test patch: