Revert "Factor out math headers"

This reverts commit e0a93dc0dfca3c8d5bdd82e82a4a158d001378ae.