This test is not relevant to the source lookup features you test. I believe this assertion is a direct result of the runtime detector doing its job and is a requirement for all other tests. So it should be asserted once the runtimes have been initialized (in init()), you could also directly check it contains "jboss-as-7.1.1.Final" there.