Don't keep the full info of all children in memory

Improves the memory footprint for the file scanner (30MB -> 5MB for my local test)

Fixes #10954 for the most part

Note, I'm working on additional (less significant) improvements for this issue but those require larger changes and depend on #10993 and are thus not suited for backporting to stable7

cc @DeepDiver1975 @PVince81 @th3fallen