Some more changes to how audience is calculated

It must be some subset of the scopes because the scopes along with
the audience are interpreted by the resource server.