Speed up some of the multiplications for sparse input