Refactor of readdir to not fail as much

Fixes #15279
Based on #13338