Make sure tests will work after #6205

Pkg.generate() will start creating a test/runtests.jl so need the tests
to cope with that