Fix compilation with uclibc.

uclibc does not have full math.h support built in so we need to explicitly link to libm in that case. Specifically isnan and isinf would be missing and not compile in this case.