Fixed #22056 -- Omitted non-package directories when collecting tests

This allows us to ignore empty dirs or other unrelated dirs.
Thanks Sebastian Pawluś for the report and the initial patch.