Repository: jina-ai/jina
Issue: Transformer encoder is broken

**Describe the bug**
The transformer with `distilbert-base-cased` is broken. It might be related to the latest change of #764 

```
ERROR:transformers.tokenization_utils:Using eos_token, but it is not set yet.
```

**Describe how you solve it**
<!-- copy past your code/pull request link -->

- - -
<!-- Optional, but really help us locate the problem faster -->

**Environment**
<!-- Run `jina --version-full` and copy paste the output here -->

**Screenshots**
<!-- If applicable, add screenshots to help explain your problem. -->



┆Issue is synchronized with this [Jira Bug](https://jinaai.atlassian.net/browse/JINACORE-149) by [Unito](https://www.unito.io/learn-more)


Changed files:
jina/executors/encoders/nlp/transformer.py, tests/unit/executors/encoders/nlp/test_mock_transformer.py

diff --git a/jina/executors/encoders/nlp/transformer.py b/jina/executors/encoders/nlp/transformer.py
index e98e025275e..392b732d9a9 100644
--- a/jina/executors/encoders/nlp/transformer.py
+++ b/jina/executors/encoders/nlp/transformer.py
@@ -80,18 +80,19 @@ class BaseTransformerEncoder(BaseEncoder):
         :return: an ndarray in size `B x D`
         """
         try:
-            self.tokenizer.pad_token = self.tokenizer.eos_token
+            if self.tokenizer.pad_token is None:
+                self.tokenizer.pad_token = self.tokenizer.eos_token
+                if self.tokenizer.pad_token is None:
+                    self.tokenizer.add_special_tokens({'pad_token': '[PAD]'})
             ids_info = self.tokenizer.batch_encode_plus(data,
                                                         max_length=self.max_length,
                                                         truncation=self.truncation_strategy,
-                                                        pad_to_max_length=True,
-                                                        padding='max_length')
+                                                        pad_to_max_length=True)
         except ValueError:
             self.model.resize_token_embeddings(len(self.tokenizer))
             ids_info = self.tokenizer.batch_encode_plus(data,
                                                         max_length=self.max_length,
-                                                        pad_to_max_length=True,
-                                                        padding='max_length')
+                                                        pad_to_max_length=True)
         token_ids_batch = self.array2tensor(ids_info['input_ids'])
         mask_ids_batch = self.array2tensor(ids_info['attention_mask'])
         with self.session():
diff --git a/tests/unit/executors/encoders/nlp/test_mock_transformer.py b/tests/unit/executors/encoders/nlp/test_mock_transformer.py
index ce033c7c6b6..db1a59851c1 100644
--- a/tests/unit/executors/encoders/nlp/test_mock_transformer.py
+++ b/tests/unit/executors/encoders/nlp/test_mock_transformer.py
@@ -1,9 +1,7 @@
-import os
 from unittest.case import TestCase
 from unittest.mock import patch
 
 import numpy as np
-import pytest
 
 from jina.executors.encoders.nlp.transformer import TransformerTorchEncoder, TransformerTFEncoder
 
@@ -33,7 +31,7 @@ class MockTFModel(MockPtModel):
         seq_length = input_ids.shape[1]
         return tf.repeat(tf.reshape(tf.range(seq_length), (1, seq_length, 1)), batch_size, axis=0), None
 
-@pytest.mark.skip('TODO: fix whatever wrong with transformers')
+
 class TransformerEncoderWithMockedModelTestCase(TestCase):
     """To skip weights downloading and model initialization part replaces the actual model with dummy version"""
     texts = ["Never gonna run around", "and desert you"]