Erase destroyed entites after a safe time has elapsed