Properly detect empty and missing directories