Issue 1:
🐛 Bug

Information

Model I am using TFBertEncoder:

Language I am using the model on English:

The problem arises when using:

my own modified scripts: (give details below)

The tasks I am working on is:

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

When I use, TFBertEncoder, I get an error.

Here is my code.

import tensorflow as tf
import numpy as np
from transformers.modeling_tf_bert import BertConfig, TFBertEncoder

print(tf.__name__, tf.__version__)
input_a = tf.keras.layers.Input(shape=(91, 128))

config = BertConfig()
config.hidden_size = 128
config.num_attention_heads = 4

# config.output_attentions = False
# config.output_hidden_states = False

head_mask = [None for _ in range(config.num_hidden_layers)]
encoder_output = TFBertEncoder(config=config)([input_a, None, head_mask])[0]
print(encoder_output.shape)

test_out = tf.keras.layers.Dense(128)(encoder_output)
print(test_out.shape)

Expected behavior

Here is the error:

(None, 91, 128)
2020-06-03 11:18:10.160647: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Failed precondition: Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
[[{{node output_23/dense/BiasAdd/ReadVariableOp}}]]
Traceback (most recent call last):
File "D:/python/tx/TEST.py", line 16, in <module>
a = tf.keras.layers.Dense(128)(encoder_output)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 720, in __call__
base_layer_utils.create_keras_history(inputs)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 187, in create_keras_history
_, created_layers = _create_keras_history_helper(tensors, set(), [])
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
layer_inputs, processed_ops, created_layers)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
layer_inputs, processed_ops, created_layers)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
layer_inputs, processed_ops, created_layers)
[Previous line repeated 5 more times]
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 247, in _create_keras_history_helper
constants[i] = backend.function([], op_input)([])
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\backend.py", line 3727, in __call__
outputs = self._graph_fn(*converted_inputs)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1551, in __call__
return self._call_impl(args, kwargs)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1591, in _call_impl
return self._call_flat(args, self.captured_inputs, cancellation_manager)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1692, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 545, in call
ctx=ctx)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
[[node output_23/dense/BiasAdd/ReadVariableOp (defined at /python/tx/TEST.py:16) ]] [Op:__inference_keras_scratch_graph_5205]

Function call stack:
keras_scratch_graph

Environment info

transformers version: 2.3.0 (in conda list)

Platform:

Python version:3.7

PyTorch version (GPU?):

Tensorflow version (GPU?):TF2.1.0(GPU)

Using GPU in script?:

Using distributed or parallel set-up in script?:No

Issue 2:
🐛 Bug

Information

Model I am using TFBertEncoder:

Language I am using the model on English:

The problem arises when using:

my own modified scripts: (give details below)

The tasks I am working on is:

my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

When I use, TFBertEncoder, I get an error.

Here is my code.

import tensorflow as tf
import numpy as np
from transformers.modeling_tf_bert import BertConfig, TFBertEncoder

print(tf.__name__, tf.__version__)
input_a = tf.keras.layers.Input(shape=(91, 128))

config = BertConfig()
config.hidden_size = 128
config.num_attention_heads = 4

# config.output_attentions = False
# config.output_hidden_states = False

head_mask = [None for _ in range(config.num_hidden_layers)]
encoder_output = TFBertEncoder(config=config)([input_a, None, head_mask])[0]
print(encoder_output.shape)

test_out = tf.keras.layers.Dense(128)(encoder_output)
print(test_out.shape)

Expected behavior

Here is the error:

(None, 91, 128)
2020-06-03 11:18:10.160647: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Failed precondition: Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
[[{{node output_23/dense/BiasAdd/ReadVariableOp}}]]
Traceback (most recent call last):
File "D:/python/tx/TEST.py", line 16, in <module>
a = tf.keras.layers.Dense(128)(encoder_output)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 720, in __call__
base_layer_utils.create_keras_history(inputs)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 187, in create_keras_history
_, created_layers = _create_keras_history_helper(tensors, set(), [])
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
layer_inputs, processed_ops, created_layers)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
layer_inputs, processed_ops, created_layers)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
layer_inputs, processed_ops, created_layers)
[Previous line repeated 5 more times]
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 247, in _create_keras_history_helper
constants[i] = backend.function([], op_input)([])
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\backend.py", line 3727, in __call__
outputs = self._graph_fn(*converted_inputs)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1551, in __call__
return self._call_impl(args, kwargs)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1591, in _call_impl
return self._call_flat(args, self.captured_inputs, cancellation_manager)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1692, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 545, in call
ctx=ctx)
File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
[[node output_23/dense/BiasAdd/ReadVariableOp (defined at /python/tx/TEST.py:16) ]] [Op:__inference_keras_scratch_graph_5205]

Function call stack:
keras_scratch_graph

Environment info

transformers version: 2.3.0 (in conda list)

Platform:

Python version:3.7

PyTorch version (GPU?):

Tensorflow version (GPU?):TF2.1.0(GPU)

Using GPU in script?:

Using distributed or parallel set-up in script?:No