Only add build sub-directory if it is actually used

Otherwise the search will cascade down one level too many.
Only add parent directories for actual variants

Otherwise we'll inadvertedly add the root directory and will probably
find the system implementation underneath there.