There are way too many obsticles to using the context system for this. I've actually found that core uses this same pattern in the annotation:

Constraint plugins restricted by typed data type:
```
/**
 * Validation constraint for the entity changed timestamp.
 *
 * @Plugin(
 *   id = "EntityChanged",
 *   label = @Translation("Entity changed", context = "Validation"),
 *   type = {"entity"}
 * )
 */
```

Field Formatter and Widget plugins use the same thing.

I think rather than keying based on 'entity' that if the 'type' value isn't in the annotation, then it is assumed to be allowed for all entity types.