Great point @WilkAndy !  I almost forgot about that case!

@jwpeterson you need to check to see if the Elem is still "active".... if it's not then search it's children (and their children if necessary) to try to find the point.

@WilkAndy - Just to be clear I was actually referring to the mesh moving because of displacements...