Revert "Improve how we set LTO"

This reverts commit a3cf7b8945c8fade6f7c6de55b30c21613212b52.