Don't install directories that we find

If we are including something because it is a sub-directory of a
package that we want, then we shouldn't try to install it.  We should
only be trying to install things that are part of the import chain
from the test code that we want to run.