Add keepgoing flag (and option) for better continuous testing support

(pair programmed with @HBehrens)