compute complex norm in a way that does not overflow. add test for this

cf. Numerical Recipes, Chapter 5.5