matrix/mat64: use math.Hypot for 2-norm

This brings the implementation into line with gonum/floats and is more
careful.

Fixes issue 10.