PERF: Do far fewer checks and use a lot less memory when loading more