Actually configure allowed failures properly

Oops.