Patent ID: 11960526
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving a query spoken by a user and captured by a user device associated with the user, the query requesting identification of an entity, the query including one or more terms that reference an entity type associated with the entity and does not include any terms that uniquely identify the entity; and
in response to receiving the query spoken by the user:
processing, using a speech recognition engine, the query spoken by the user to identify the entity type associated with the entity;
obtaining environmental data comprising at least one of audio data captured by a microphone associated with the user device or image data captured by an image capture device associated with the user device, the audio data representing ambient audio present in an environment of the user device while a media item is playing in the environment, and the image data representing an image of the environment while the media item is playing in the environment;
processing, using a content recognition engine, the environmental data to identify a particular segment of a plurality of segments of the media item that is playing in the environment of the user device when the at least one of the audio data or the image data is captured;
identifying, based on one or more timestamps associated with the identified particular segment, supplemental information for the media item that identifies one or more entities that actually appear in or are heard in the identified particular segment;
identifying as the entity requested by the query, a particular entity of the identified one or more entities based on the entity type; and
providing a response to the query spoken by the user that identifies the identified particular entity.