A Theoretical Review on "Rethinking Attention with Performers (ICLR 2021 Oral)"
Channel: SotAdemia
Date: 2021-05-29

I make some time to make a theoretical review on an interesting work from Choromanski et al. with the title of “rethinking attention with performers.”
I assume that you have read the abstract and skim the paper, rather to focus on their theoretical analyses. This part 1 will review:

1) preliminaries on attention mechanism
2) generalized kernelizable attention
3) work-through of the proofs of Lemma 1 and 2.