hi there today we'll look at big bird transformers for longer sequences by manil zaire and guru garuganesh at al of google research so this paper on a high level proposes to replace the quadratic attention mechanism in transformers by a mix of random attention windowed attention and selective global attention therefore achieving a complexity of linear memory requirement instead of quadratic memory requirement and as a result of that they can process longer sequences than traditional transformers like bert and achieve better results in some nlp