Ensure we can safely remove a single target-action pair

This removes the secondary targets tracking set in favor of a more
expensive aggregation of targets, but the previous implementation
could've been some premature optimization.