bi-directional context within a self attention model and the way they do this is they change the objective so they're no longer doing this like standard you know maximum likelihood training on like just the data distribution they use this kind of proxy tasks called mast language modeling so again you know at the bottom here we could see left-right LM is like the cat sat on the and then you blink out a word and it's supposed to predict math right language modeling would be we'd go the other way around and same at the onset