Move libm dependency to the correct places

Issue #61