But here you are iterating over newWorldObjects (n) and in lostWorldObjects.erase you would do the same -> n^2. 

But if you can confirm that only the WorldTreeQuery/VoxelTreeQuery really take performance here we could delay further optimization until every object has a scanner.