text: 
  
  TITLE
  TFLongformer Error: TypeError: __init__() missing 1 required positional argument: 'last_hidden_state'
  
  COMMENTS
  2
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  <!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
       Don't forget to fill out the missing fields in that output! -->
      
  - `transformers` version: 3.5.0
  - Platform: Linux-4.19.129-aufs-1-x86_64-with-debian-10.1
  - Python version: 3.7.3
  - PyTorch version (GPU?): not installed (NA)
  - Tensorflow version (GPU?): 2.3.1 (True)
  - Using GPU in script?: Yes
  - Using distributed or parallel set-up in script?: No
  
  
  ### Who can help
   Longformer/Reformer: @patrickvonplaten
  
  
  ## Information
  
  Model I am using (Bert, XLNet ...): TFLongformer
  
  The problem arises when using:
  * [x] my own modified scripts: Scripts below.
  
  The tasks I am working on is:
  * [x] an official GLUE/SQUaD task: TriviaQA converted to SQuAD format
  
  Hi, I’m getting an error when trying to fine-tune TFLongformer on TriviaQA using a simple architecture that is discussed [here](https://keras.io/examples/nlp/text_extraction_with_bert/) (I want to use TFLongformer instead of BERT and formatted-TriviaQA instead of SQuAD).
  
  The error I'm getting is:
  ```
  Traceback (most recent call last):                                                                                        [29/1207]
    File "model_for_issues.py", line 102, in <module>                                                                                
      model = build_model()                                                                                                          
    File "model_for_issues.py", line 65, in build_model                                                                              
      embedding = encoder(input_ids, attention_mask=attention_mask).last_hidden_state                                                
    File "path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base$
  layer.py", line 926, in __call__                                                                                                   
      input_list)                                                                                                                    
    File "path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base$
  layer.py", line 1117, in _functional_construction_call                                                                             
      outputs = call_fn(cast_inputs, *args, **kwargs)                                                                                
    File "path/to/project/env/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api
  .py", line 258, in wrapper
      raise e.ag_error_metadata.to_exception(e)
  TypeError: in user code:
  
      path/to/project/env/lib/python3.7/site-packages/transformers/modeling_tf_longformer.py:1
  760 call  *
          outputs = self.longformer(inputs, **kwargs)
      path/to/project/env/lib/python3.7/site-packages/transformers/modeling_tf_longformer.py:1
  509 call  *
          encoder_outputs = self.encoder(
      path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_laye
  r.py:926 __call__  **
          input_list)
      path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_laye
  r.py:1147 _functional_construction_call
          outputs)
      path/to/project/env/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_laye
  r.py:2568 _set_connectivity_metadata
          outputs = nest.pack_sequence_as(outputs, outputs_copy)
      path/to/project/env/lib/python3.7/site-packages/tensorflow/python/util/nest.py:570 pack_
  sequence_as
          return _pack_sequence_as(structure, flat_sequence, expand_composites)
      path/to/project/env/lib/python3.7/site-packages/tensorflow/python/util/nest.py:533 _pack
  _sequence_as
          return sequence_fn(structure, packed)
      path/to/project/env/lib/python3.7/site-packages/tensorflow/python/util/nest.py:152 _sequ
  ence_like
          d = instance_type()
  
      TypeError: __init__() missing 1 required positional argument: 'last_hidden_state'
  ```
  
  _Remark: I changed the path to the working directory to “path/to/project” for my privacy_
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  1. Convert TriviaQA data to SQuAD format using this script Longformer’s github https://github.com/allenai/longformer/blob/master/scripts/cheatsheet.txt#L29
  2. Use this code to re-format the data to the format I work with:
  ```python
  """ Usage:
      TriviaQA_formatting.py [--in=INPUT_FILE] [--out=OUTPUT_FILE] [--debug]
  """
  # External imports
  import logging
  import pdb
  from pprint import pprint
  from pprint import pformat
  from docopt import docopt
  from pathlib import Path
  from tqdm import tqdm
  import json
  # Local imports
  import numpy as np
  #----
  if __name__ == "__main__":
  
      # Parse command line arguments
      args = docopt(__doc__)
      inp_fn = Path(args["--in"]) if args["--in"] else None
      out_fn = Path(args["--out"]) if args["--out"] else Path("./tmp.out")
          # Determine logging level
      debug = args["--debug"]
      if debug:
          logging.basicConfig(level = logging.DEBUG)
      else:
          logging.basicConfig(level = logging.INFO)
      # Start computation
      data = []
      with open(inp_fn, 'r') as f:
          print("started loading")
          data = json.load(f)["data"]
          print("ended loading")
      data_dict = {}
      contexts  = [entry["paragraphs"][0]["context"] for entry in tqdm(data)]
      questions = [entry["paragraphs"][0]["qas"][0]["question"] for entry in tqdm(data)]
      answer_texts = [entry["paragraphs"][0]["qas"][0]["aliases"] for entry in tqdm(data)]
      start_indices = [None] * len(data)
      end_indices = [None] * len(data)
      for index, entry in enumerate(data):
          # taking the first answer
          answers = entry["paragraphs"][0]["qas"][0]["answers"]
          start_indices[index] = answers[0]["answer_start"] if answers else -1
          end_indices[index] = (answers[0]["answer_start"] + len(answers[0]["text"])) if answers else -1
      data_dict = {
          "context": contexts,
          "question": questions,
          "start_index": start_indices,
          "end_index": end_indices,
          "answer_texts": answer_texts,
      }
      with open(out_fn, 'w+') as f:
          json.dump(data_dict, f)
      # End
      logging.info("DONE")
  ```
  Scripts used to run this formatting code:
  ```
  python TriviaQA_formatting.py --in=squad-wikipedia-train-4096.json --out=formatted_wiki_train_4096.json
  ```
  
  
  3. Run this script for tokenization and training model:
  
  ```python
  import tensorflow as tf
  from tensorflow import keras
  from tensorflow.keras import layers
  from transformers import TFLongformerModel,LongformerConfig, LongformerTokenizer, LongformerTokenizerFast
  from longformer_encoding import *
  import numpy as np
  from tqdm import tqdm
  import json
  
  def encode_sentence(s, tokenizer):
      s = s + tokenizer.sep_token
      return list(tokenizer.encode(s))
  
  def pad_to_max_length(t, max_len, padding_value):
      t = t[:, :max_len]
      paddings = [[0, 0], [0, max_len-tf.shape(t)[1]]]
      return tf.pad(t, paddings, 'CONSTANT', constant_values=padding_value)
  
  def longformer_encode(texts1, texts2, answer_end, tokenizer, max_len=4096):
      num_examples = len(texts1)
      sentence1 = tf.ragged.constant([encode_sentence(tokenizer.cls_token+s, tokenizer)
          for s in tqdm(texts1)])
      sentence2 = tf.ragged.constant([
          encode_sentence(s, tokenizer)
          for s in tqdm(texts2)])
      input_word_ids = tf.concat([sentence1, sentence2], axis=-1)
      attention_mask = tf.ones_like(input_word_ids).to_tensor()
      type_s1 = tf.ones_like(sentence1)
      type_s2 = tf.zeros_like(sentence2)
      global_attention_mask = tf.concat(
         [type_s1, type_s2], axis=-1).to_tensor()
  
      sentence2_start_index = [len(s1) for s1 in sentence1]
      # get indices of examples to ignore:
      valid_sample_indices = []
      for i in tqdm(range(num_examples)):
          if sentence2_start_index[i] + answer_end[i] <=  max_len:
              valid_sample_indices.append(i)
  re
  
      inputs = {
          'input_ids': pad_to_max_length(input_word_ids.to_tensor(), max_len, tokenizer.pad_token_id),
          'attention_mask': pad_to_max_length(attention_mask, max_len, 0),
          'global_attention_mask': pad_to_max_length(global_attention_mask, max_len, 0),
      }
  
      return inputs, valid_sample_indices
  
  def read_data_file(fn):
      with open(fn, 'r') as f:
          data = json.load(f)
      answer_exists_slice = np.ndarray.flatten(np.argwhere(np.array(data["end_index"]) != -1))
      return {key:np.ndarray.tolist(np.array(value)[answer_exists_slice]) for (key,value) in data.items()}
  
  def build_model( max_len=4096):
  
      config = LongformerConfig()
      LongformerModel = TFLongformerModel(config=config)
      encoder = LongformerModel.from_pretrained('allenai/longformer-base-4096',return_dict=True)
      input_ids = layers.Input(shape=(max_len,), dtype=tf.int32, name="input_ids")
      attention_mask = layers.Input(shape=(max_len,), dtype=tf.int32, name="attention_mask")
      global_attention_mask = layers.Input(shape=(max_len,), dtype=tf.int32, name="global_attention_mask")
      embedding = encoder(input_ids, attention_mask=attention_mask).last_hidden_state
      start_logits = layers.Dense(1, name="start_logit", use_bias=False)(embedding)
      start_logits = layers.Flatten()(start_logits)
  
      end_logits = layers.Dense(1, name="end_logit", use_bias=False)(embedding)
      end_logits = layers.Flatten()(end_logits)
  
      start_probs = layers.Activation(keras.activations.softmax)(start_logits)
      end_probs = layers.Activation(keras.activations.softmax)(end_logits)
  
      model = keras.Model(
          inputs=[input_ids, attention_mask],
          outputs=[start_probs, end_probs],
      )
      loss = keras.losses.SparseCategoricalCrossentropy(from_logits=False)
      optimizer = keras.optimizers.Adam(lr=5e-5)
      model.compile(optimizer=optimizer, loss=[loss, loss])
      model.summary()
      return model
  
  # Preprocessing
  train_fn = "data/formatted_wiki_train_4096.json"
  data = read_data_file(train_fn)
  print("One data example:\n Question: {}\ncontext: {}\nstart_index: {}".format(data['question'][0], data['context'][0][:200], data["start_index"][0]))
  tokenizer = LongformerTokenizerFast.from_pretrained('allenai/longformer-base-4096')
  inputs, valid_sample_indices = longformer_encode(data["question"], data["context"], data["end_index"], tokenizer, 4096)
  valid_samples_slice = np.array(valid_sample_indices)
  clean_inputs = {key:np.array(value)[valid_samples_slice] for (key,value) in inputs.items()}
  clean_data = {key:np.array(value)[valid_samples_slice] for (key,value) in data.items()}
  clean_data["encoded_input"] = clean_inputs
  
  
  x_train = clean_data["encoded_input"]
  y_train = [clean_data["start_index"], clean_data["end_index"]]
  
  
  model = build_model()
  model.fit(
  x_train,
  y_train,
  epochs=3,
  verbose=2,
  batch_size=1,
      )
  ```
  
  ###### Trainig sample example: 
  ```
  One data example:
   Question: Ownership of which worldwide publishing concern gave Conrad Black control of the Daily Telegraph?
  context: Conrad Moffat Black , Baron Black of Crossharbour , ( born 25 August 1944 ) is a Canadian-born British former newspaper publisher and author . He is a non-affiliated life peer . 
   
   Black controlled H
  start_index: 199
  ```
  
  
  ## Expected behavior
  
  The model should start training.
  
  