Add a maximum number of failures per change to report

More than this is considered to be a broken change rather than a potential race