Index active pointers by the invocation ID

so that we can quickly remove all pointers tagged with a specific
invocation ID