what they do is they take the inputs they use word Tyvek to calculate some vector for each input token and then onto that onto that embedding they add a bunch of sine and cosines of different frequencies starting at just pi and then stretching out longer and longer and longer and if you look at the whole thing it looks like this and what this does is it lets the model reason about the relative position of any tokens right so if you can kind of imagine that the model can say if the orange dimension is slightly higher than the