experiments are pretty pretty pretty interesting and here they have they do these are the last experiments for this paper um they do an interesting experiment where they actually replace the attention heads by simply an average mechanism and later they actually replace them by gaussians but in this case they simply average and they show that look if i replace layer one with just averaging the perplexity doesn't rise that much right so it's pretty good um even if i replace an entire layer here with averaging uh it it perplexity goes more up