Make sure directories aren't getting tested

Fixes #3910