we we can make up for not having a fully connected graph by simply running multiple layers so their sparsity is first of all you have this random attention which i believe changes from sequence to sequence but stays within or among the layers of the same sequence then you have the window attention with the reasoning so the reasoning behind the random attention is that if you have a randomly connected graph the path lengths are on average logarithmic so you can route information efficiently the reasoning behind the window