associative memory every module in there is an associative memory and you put 40 layers of those with hundreds of millions of parameters you train this on billions of words of text and you train it in the following way you take a window of a few hundred words you take out 15% of the words and you train the machine to just predict the missing words now the machine cannot do a perfect job at this so what it outputs is for each word there are missing it that puts a probability vector whose size is the size of the of the