tests: use fabsf (float), not fabs (double) in epsilon comparisons of floats

Fixes a mistake in commit f68eb7c