@galderz just skimmed over it - will we be able to see which one of the two failed? Do we get some kind of message "Now testing with/without compression" ? If not, can we at least make a logger.info() or something like that?

From the point of reusability I'd say let's use your version, definitely better to just have a single class for the tests.