blue dimension on one word versus another then you can see how it knows that that token is to the left or right of the other and because it has this at all these different wavelengths it can look across the entire document at kind of arbitrary scales to see whether one idea is before or after another the key thing is that this is how the system understands position and isn't just a bag of words for Fortran when doing the attention okay so transformers there's the two key innovations as positional encoding and