Fix quirks in rotation_matrix

As discussed in more detail on the astropy list, the  implementation of `rotation_matrix` for arbitrary axes had some rounding and sign convention issues.  This PR fixes that, updates `angle_axis` to be more consistent with it, and adds a few tests of these functions.