resnet i've done a paper on resnet resnet basically takes the input pipes it through straight and adds to it whatever comes out of this operation okay that's a residual block now usually this thing here would be convolutions and convolutions and they are now replaced by these multi-head axial attention you can see there is a multi-headed tension in the height and there is a multi-head attention in the width and that gives us the property that every node can send around information to every other node in two steps