[math] more cleanup, e.g. rmat_[transp]_vmult

for consistency always call the fuctions x_rmat_[transp]_vmult,
FLOAT_RMAT_VECT3_MUL and FLOAT_RMAT_VECT3_TRANSP_MUL are removed.