Only automatically include validations when enabled

This is a follow up to #16024.