and you can see the corresponds if you remember the previous plot the correspondence is pretty one to one with how much blue and green uh heads there are as in contrast to how much red uh and orange ones there are so here you have lots of blue ones and you can see that the error kind of goes up and interestingly here you have more meta-stable states at the end but still the perplexity goes up uh more so i guess you can only really replace the red ones with the averaging so this is always averaging in one particular layer