Revert "libm build workaround"

This reverts commit 174d2aa74be5b73f4c17e1fefaea4db397c3d319.