character level vocabulary and it kind of just merges the two most common pairs of characters at a time so you might have T and H be the most common pair of words or characters and then you'll combine them into a new token called you have th and you'll merge that and you'll resub stitute it in all of your words and then you'll run this loop again and so if you run this and kind of just keep merging and merging and merging it learns basically a tree of merges that quickly pop out words full words like the and you know common endings like IMG