Fix matrix addto in case where padding is applied