Only track relevant entities for collision detection