Not Implemented Error 
Author: 4rshdeep
Pull request: no
Labels: name: wontfix

name: TensorFlow

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): TFBert

Language I am using the model on (English, Chinese ...): Any

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

## To reproduce
 I am trying to run the Bert model on a sequence of sentences. 
Steps to reproduce the behavior:
```
import tensorflow as tf
from transformers import BertTokenizer, TFBertModel
inputs = tf.keras.Input(shape=(50, 64), dtype='int32')
model = TFBertModel.from_pretrained('bert-base-uncased')
outputs = tf.keras.layers.TimeDistributed(model)(inputs)

```
I get a not implemented error, 
NotImplementedError                       Traceback (most recent call last)
<ipython-input-5-631f3cd2e8b2> in <module>
----> 1 outputs = tf.keras.layers.TimeDistributed(model)(inputs)

The same code works fine for 
```
inputs = tf.keras.Input(shape=(10, 128, 128, 3))
conv_2d_layer = tf.keras.layers.Conv2D(64, (3, 3))
outputs = tf.keras.layers.TimeDistributed(conv_2d_layer)(inputs)
outputs.shape
```


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I should be able to run the bert model on the sequence of sentences. 

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.0.0
- Platform: Linux-5.0.0-1028-azure-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Comments:
This is likely due because you are using a wrong version of Tensorflow. Can you run `transformers-cli env`, as the template suggests, and place the result here?

Can you post the full stack trace? It's hard to debug this way.

@BramVanroy for your reference

```
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-2-db5a2030f0d4> in <module>
      3 inputs = tf.keras.Input(shape=(50, 64), dtype='int32')
      4 model = TFBertModel.from_pretrained('bert-base-uncased')
----> 5 outputs = tf.keras.layers.TimeDistributed(model)(inputs)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    840                     not base_layer_utils.is_in_eager_or_tf_function()):
    841                   with auto_control_deps.AutomaticControlDependencies() as acd:
--> 842                     outputs = call_fn(cast_inputs, *args, **kwargs)
    843                     # Wrap Tensors in `outputs` in `tf.identity` to avoid
    844                     # circular dependencies.

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/wrappers.py in call(self, inputs, training, mask)
    254       y = self.layer(inputs, **kwargs)
    255       # Shape: (num_samples, timesteps, ...)
--> 256       output_shape = self.compute_output_shape(input_shape).as_list()
    257       output_shape = self._get_shape_tuple(
    258           (-1, input_length), y, 1, output_shape[2:])

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/wrappers.py in compute_output_shape(self, input_shape)
    208     child_input_shape = tensor_shape.TensorShape([input_shape[0]] +
    209                                                  input_shape[2:])
--> 210     child_output_shape = self.layer.compute_output_shape(child_input_shape)
    211     if not isinstance(child_output_shape, tensor_shape.TensorShape):
    212       child_output_shape = tensor_shape.TensorShape(child_output_shape)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py in compute_output_shape(self, input_shape)
    710   def compute_output_shape(self, input_shape):
    711     if not self._is_graph_network:
--> 712       return super(Network, self).compute_output_shape(input_shape)
    713 
    714     # Convert any shapes in tuple format to TensorShapes.

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in compute_output_shape(self, input_shape)
    637                                       'layer (%s).' % self.__class__.__name__)
    638       return nest.map_structure(lambda t: t.shape, outputs)
--> 639     raise NotImplementedError
    640 
    641   @doc_controls.for_subclass_implementers

NotImplementedError: 
```

Could this be caused by the fact that TFBertModel returns a tuple (hidden states, pooled output)? In @4rshdeep's case, would we expect TimeDistributed to return a shape of (batch_size, 50, 768), or (batch_size, 50, 64, 768)?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm having the same error with using the data Streaming app on HuggingFace.