text: 
  
  TITLE
  why set masked_bias as -10000 in GPT2
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Information
  
  `masked_bias` is set as `-10000` in GPT2, why not `-inf`?
  
  https://github.com/huggingface/transformers/blob/e43f3b6190cfd98a38912411b8bc8ecbb6629280/src/transformers/models/gpt2/modeling_gpt2.py#L133
  
  ## openai/gpt-2
  
  In [openai/gpt2](https://github.com/openai/gpt-2/blob/a74da5d99abaaba920de8131d64da2862a8f213b/src/model.py#L88), the bias is set as `-1e10` 
  ```py
  w = w*b - tf.cast(1e10, w.dtype)*(1-b)
  ```
  ## Other implementation, such as bert, transformer
  
  https://github.com/huggingface/transformers/blob/82498cbc37d5c15520c7bddde5d804c804eee498/src/transformers/models/bart/modeling_bart.py#L81
  
  <!-- A clear and concise description of what you would expect to happen. -->
  