TFLongformer Error: TypeError: __init__() missing 1 required positional argument: 'last_hidden_state'
Author: SapirWeissbuch
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
    
- `transformers` version: 3.5.0
- Platform: Linux-4.19.129-aufs-1-x86_64-with-debian-10.1
- Python version: 3.7.3
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help
 Longformer/Reformer: @patrickvonplaten


## Information

Model I am using (Bert, XLNet ...): TFLongformer

The problem arises when using:
* [x] my own modified scripts: Scripts below.

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: TriviaQA converted to SQuAD format

Hi, I’m getting an error when trying to fine-tune TFLongformer on TriviaQA using a simple architecture that is discussed [here](https://keras.io/examples/nlp/text_extraction_with_bert/) (I want to use TFLongformer instead of BERT and formatted-TriviaQA instead of SQuAD).

The error I'm getting is:
```
Traceback (most recent call last):                                                                                        [29/1207]
  File "model_for_issues.py", line 102, in <module>                                                                                
    model = build_model()                                                                                                          
  File "model_for_issues.py", line 65, in build_model                                                                              
    embedding = encoder(input_ids, attention_mask=attention_mask).last_hidden_state                                                
  File "path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base$
layer.py", line 926, in __call__                                                                                                   
    input_list)                                                                                                                    
  File "path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base$
layer.py", line 1117, in _functional_construction_call                                                                             
    outputs = call_fn(cast_inputs, *args, **kwargs)                                                                                
  File "path/to/project/env/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api
.py", line 258, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

    path/to/project/env/lib/python3.7/site-packages/transformers/modeling_tf_longformer.py:1
760 call  *
        outputs = self.longformer(inputs, **kwargs)
    path/to/project/env/lib/python3.7/site-packages/transformers/modeling_tf_longformer.py:1
509 call  *
        encoder_outputs = self.encoder(
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_laye
r.py:926 __call__  **
        input_list)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_laye
r.py:1147 _functional_construction_call
        outputs)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_laye
r.py:2568 _set_connectivity_metadata
        outputs = nest.pack_sequence_as(outputs, outputs_copy)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/util/nest.py:570 pack_
sequence_as
        return _pack_sequence_as(structure, flat_sequence, expand_composites)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/util/nest.py:533 _pack
_sequence_as
        return sequence_fn(structure, packed)
    path/to/project/env/lib/python3.7/site-packages/tensorflow/python/util/nest.py:152 _sequ
ence_like
        d = instance_type()

    TypeError: __init__() missing 1 required positional argument: 'last_hidden_state'
```

_Remark: I changed the path to the working directory to “path/to/project” for my privacy_

## To reproduce

Steps to reproduce the behavior:

1. Convert TriviaQA data to SQuAD format using this script Longformer’s github https://github.com/allenai/longformer/blob/master/scripts/cheatsheet.txt#L29
2. Use this code to re-format the data to the format I work with:
```python
""" Usage:
    TriviaQA_formatting.py [--in=INPUT_FILE] [--out=OUTPUT_FILE] [--debug]
"""
# External imports
import logging
import pdb
from pprint import pprint
from pprint import pformat
from docopt import docopt
from pathlib import Path
from tqdm import tqdm
import json
# Local imports
import numpy as np
#----
if __name__ == "__main__":

    # Parse command line arguments
    args = docopt(__doc__)
    inp_fn = Path(args["--in"]) if args["--in"] else None
    out_fn = Path(args["--out"]) if args["--out"] else Path("./tmp.out")
        # Determine logging level
    debug = args["--debug"]
    if debug:
        logging.basicConfig(level = logging.DEBUG)
    else:
        logging.basicConfig(level = logging.INFO)
    # Start computation
    data = []
    with open(inp_fn, 'r') as f:
        print("started loading")
        data = json.load(f)["data"]
        print("ended loading")
    data_dict = {}
    contexts  = [entry["paragraphs"][0]["context"] for entry in tqdm(data)]
    questions = [entry["paragraphs"][0]["qas"][0]["question"] for entry in tqdm(data)]
    answer_texts = [entry["paragraphs"][0]["qas"][0]["aliases"] for entry in tqdm(data)]
    start_indices = [None] * len(data)
    end_indices = [None] * len(data)
    for index, entry in enumerate(data):
        # taking the first answer
        answers = entry["paragraphs"][0]["qas"][0]["answers"]
        start_indices[index] = answers[0]["answer_start"] if answers else -1
        end_indices[index] = (answers[0]["answer_start"] + len(answers[0]["text"])) if answers else -1
    data_dict = {
        "context": contexts,
        "question": questions,
        "start_index": start_indices,
        "end_index": end_indices,
        "answer_texts": answer_texts,
    }
    with open(out_fn, 'w+') as f:
        json.dump(data_dict, f)
    # End
    logging.info("DONE")
```
Scripts used to run this formatting code:
```
python TriviaQA_formatting.py --in=squad-wikipedia-train-4096.json --out=formatted_wiki_train_4096.json
```


3. Run this script for tokenization and training model:

```python
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from transformers import TFLongformerModel,LongformerConfig, LongformerTokenizer, LongformerTokenizerFast
from longformer_encoding import *
import numpy as np
from tqdm import tqdm
import json

def encode_sentence(s, tokenizer):
    s = s + tokenizer.sep_token
    return list(tokenizer.encode(s))

def pad_to_max_length(t, max_len, padding_value):
    t = t[:, :max_len]
    paddings = [[0, 0], [0, max_len-tf.shape(t)[1]]]
    return tf.pad(t, paddings, 'CONSTANT', constant_values=padding_value)

def longformer_encode(texts1, texts2, answer_end, tokenizer, max_len=4096):
    num_examples = len(texts1)
    sentence1 = tf.ragged.constant([encode_sentence(tokenizer.cls_token+s, tokenizer)
        for s in tqdm(texts1)])
    sentence2 = tf.ragged.constant([
        encode_sentence(s, tokenizer)
        for s in tqdm(texts2)])
    input_word_ids = tf.concat([sentence1, sentence2], axis=-1)
    attention_mask = tf.ones_like(input_word_ids).to_tensor()
    type_s1 = tf.ones_like(sentence1)
    type_s2 = tf.zeros_like(sentence2)
    global_attention_mask = tf.concat(
       [type_s1, type_s2], axis=-1).to_tensor()

    sentence2_start_index = [len(s1) for s1 in sentence1]
    # get indices of examples to ignore:
    valid_sample_indices = []
    for i in tqdm(range(num_examples)):
        if sentence2_start_index[i] + answer_end[i] <=  max_len:
            valid_sample_indices.append(i)
re

    inputs = {
        'input_ids': pad_to_max_length(input_word_ids.to_tensor(), max_len, tokenizer.pad_token_id),
        'attention_mask': pad_to_max_length(attention_mask, max_len, 0),
        'global_attention_mask': pad_to_max_length(global_attention_mask, max_len, 0),
    }

    return inputs, valid_sample_indices

def read_data_file(fn):
    with open(fn, 'r') as f:
        data = json.load(f)
    answer_exists_slice = np.ndarray.flatten(np.argwhere(np.array(data["end_index"]) != -1))
    return {key:np.ndarray.tolist(np.array(value)[answer_exists_slice]) for (key,value) in data.items()}

def build_model( max_len=4096):

    config = LongformerConfig()
    LongformerModel = TFLongformerModel(config=config)
    encoder = LongformerModel.from_pretrained('allenai/longformer-base-4096',return_dict=True)
    input_ids = layers.Input(shape=(max_len,), dtype=tf.int32, name="input_ids")
    attention_mask = layers.Input(shape=(max_len,), dtype=tf.int32, name="attention_mask")
    global_attention_mask = layers.Input(shape=(max_len,), dtype=tf.int32, name="global_attention_mask")
    embedding = encoder(input_ids, attention_mask=attention_mask).last_hidden_state
    start_logits = layers.Dense(1, name="start_logit", use_bias=False)(embedding)
    start_logits = layers.Flatten()(start_logits)

    end_logits = layers.Dense(1, name="end_logit", use_bias=False)(embedding)
    end_logits = layers.Flatten()(end_logits)

    start_probs = layers.Activation(keras.activations.softmax)(start_logits)
    end_probs = layers.Activation(keras.activations.softmax)(end_logits)

    model = keras.Model(
        inputs=[input_ids, attention_mask],
        outputs=[start_probs, end_probs],
    )
    loss = keras.losses.SparseCategoricalCrossentropy(from_logits=False)
    optimizer = keras.optimizers.Adam(lr=5e-5)
    model.compile(optimizer=optimizer, loss=[loss, loss])
    model.summary()
    return model

# Preprocessing
train_fn = "data/formatted_wiki_train_4096.json"
data = read_data_file(train_fn)
print("One data example:\n Question: {}\ncontext: {}\nstart_index: {}".format(data['question'][0], data['context'][0][:200], data["start_index"][0]))
tokenizer = LongformerTokenizerFast.from_pretrained('allenai/longformer-base-4096')
inputs, valid_sample_indices = longformer_encode(data["question"], data["context"], data["end_index"], tokenizer, 4096)
valid_samples_slice = np.array(valid_sample_indices)
clean_inputs = {key:np.array(value)[valid_samples_slice] for (key,value) in inputs.items()}
clean_data = {key:np.array(value)[valid_samples_slice] for (key,value) in data.items()}
clean_data["encoded_input"] = clean_inputs


x_train = clean_data["encoded_input"]
y_train = [clean_data["start_index"], clean_data["end_index"]]


model = build_model()
model.fit(
x_train,
y_train,
epochs=3,
verbose=2,
batch_size=1,
    )
```

###### Trainig sample example: 
```
One data example:
 Question: Ownership of which worldwide publishing concern gave Conrad Black control of the Daily Telegraph?
context: Conrad Moffat Black , Baron Black of Crossharbour , ( born 25 August 1944 ) is a Canadian-born British former newspaper publisher and author . He is a non-affiliated life peer . 
 
 Black controlled H
start_index: 199
```


## Expected behavior

The model should start training.



Comments:
Hey @SapirWeissbuch - it's very hard for us to debug such long and user-specific examples that also require external files. 

I'd recommend that you either:

1) Narrow down the error much more and give us a 10-liner to reproduce the error (also making sure that it is in fact an error and not a wrong pre-processing step)
OR
2) Ask your question on the forum https://discuss.huggingface.co/
OR
3) Make a google colab (also keeping it to an absolute minimum of lines) to reproduce your error which makes it easier for us to quickly debug the problem.

We are only 4 people maintaining the library and sadly don't find the time to debug such longer issues. Hope you understand and looking forward for a pin-pointed error description :-) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@SapirWeissbuch , I'm also facing this issue, were you able to resolve this? I get this error when running training for "allenai/longformer-base-4096" model with distributed training