self attention architecture from scratch on a supervised test they always use bi-directional or almost always instead of no mess mess self attention matrices and so this turns out to have a huge boost on glue so that that bump I believe between GP t1 and Burt was like they went from I will GPT one had like an average of 78 or something or sorry excuse me I think this got reworked and sorry we excluded WN a lie it was like a bump of like five five plus percent so it basically got a double the Headroom on gbg1 and they show with very careful