Change the value type of BeingPointedBy to DenseSet

The old type was vector, and it took linear time to find an delete an
element.