multi-headed attention transformers are awesome even though there are N squared and the length of the document these all-to-all comparisons can be done almost for free in a modern GPU GPUs changed all sorts of things right you can do a thousand by thousand matrix multiply as fast as you can do a ten by two in a lot of cases because they have so much parallelism they have so much bandwidth that but a fixed latency for every operation so you can do these massive massive multiplies almost for free in a lot of cases so doing things