Calculate gram matrix as a whole