Don't always link to libm.

Forgot to remove this.