critical innovation that I think is incredibly clever so without this positional encoding attention mechanisms are just bags of words right there's nothing seeing what the difference is between work to live or live to work right there they're just all positions they're all equivalent positions you're just going to compute some score for each of them so what they did is they took a lesson from Fourier theory and added in a bunch of sines and cosines as extra dimensions sorry not as extra dimensions but onto the the word embeddings so going back so