Reduced the loops in removeFromCollisionGrid