I believe it does break when trying to use the binary on java6, for 
example.

I would consider running two profiles, and executing the tests twice in 
both, or something like that.