Revert "apply O3 to a larger part of the build"

This reverts commit 571f0037905f61c8be9f79658a25a62bac6b2ee4.