Fix XFAIL condition from r208840

The XFAILs in r208840 were too general. They were meant to only apply
to testing when use_system_lib was set.