they also do better data generation it turned out that the original bird kind of from a simplicity perspective cache the masking so they only actually masked the sequences once and they always print mask location and so you can simply change that to an online setting where you keep sampling the mask and that helps with overfitting and they also use a more flexible vocab scheme these kind of a full BP scheme that can do kind of full utf-8 byte sequences so you can handle any string at least with the standard byte sequence representation and then