Revert "try LTO for clang too"

This reverts commit b401906d82b510909f3ebc151fbf91b1c96edbd9.