ARTIFICIAL INTELLIGENCE ASSISTANT CONTEXT RECOGNITION SERVICE

A computer-implemented method includes activating ACR (Automatic Content Recognition) functionalities through a voice command in an audio file received by a virtual assistant, processing the audio file to improve the audio file's quality, providing context information to the virtual assistant, locating supplemental information associated with the context information, and presenting a response to the voice command based on the supplemental information and context information.

BACKGROUND

1. Technical Field

An objective of the example implementations is provide a method of producing recommendations to a user in response to a voice command based on the voice command and the associated non-voice command audio data.

2. Related Art

Artificial intelligence assistants are limited by a command string that users must learn or teach the AI assistant through trial and error. In some cases, users may use a colloquial term, synonym, or pronoun that the artificial intelligence assistant is unable to process. For example, when a user asks an AI assistant, “What is this?” the AI assistant is unable to associate the pronoun “this” to process the command string without additional information. However, the audio stream combined with the command commonly includes additional sounds that can be used to process the command.

SUMMARY

An objective of the example implementations is to provide a process that, in combining the features of a Virtual Assistant (powered by Artificial Intelligence features) and an Automatic Content Recognition (ACR) Engine based on audio fingerprinting, can enrich the user experience by providing information and direct purchasing options on the content (i.e., television advertisements, songs, movies/television series) that the user is exposed to at a given moment. Such content can be played in any media source (television, computer, media player, videogame console, phone, etc.).

Different use cases are provided that use the AI Assistant-ACR Engine combination to obtain information about a product, brands, and their ratings. It is another object of this process to be able to precisely and accurately provide the Virtual Assistant with context around the media consumed so that the assistant can quickly respond to users' inquiries.

DETAILED DESCRIPTION

The following detailed description provides further details of the figures and example implementations of the present specification. Terms used throughout the description are provided as examples and are not intended to be limiting. For example, the use of the term “automatic” may involve fully automatic or semi-automatic implementations involving user or operator control over certain aspects of the implementation, depending on the desired implementation of one of ordinary skill in the art practicing implementations of the present application.

According to the present example implementation, one or more related art problems may be resolved. For example, but not by way of limitation, media content is generated and provided to a user via a device. An online application that is running on a device that is configured to receive an audio signal senses an audio input from the user. The audio input may be, but is not limited to a query from the user. In the query, the user may include a pronoun, but may exclude the noun associated with the query. In this situation, the example implementation will apply content ingestion and fingerprint extraction techniques, as well as data ingestion operations, to provide the ACR content database with the necessary information.

The ACR content database then applies one or more algorithms to determine the context and provide the information associated with the noun for which the pronoun was provided. While the foregoing description refers to a noun in the concept of a query in the English language, the present example implementations are not limited thereto, and other situations in which a portion of a query and other query structures may be substituted therefore without departing from the inventive scope. Further, queries may be performed in other languages with other structures, and similar results may be obtained in those languages by the example implementations.

Accordingly, the example implementations may permit a more natural and does user friendly approach to processing user queries, especially for those users who would typically use pronouns in their natural conversations and questions, and for which it would be unusual or awkward to use something other than the pronoun, such as “this” or the like, as explained in the further details below.

Technical Description

An audio-based Automatic Content Recognition (ACR) runs on any device with a compatible operating system (i.e., smart speaker, smartphone, smart watch, smart TV, etc.). This technology uses the device's microphone to securely and privately collect media exposure in real time. The ACR engine encrypts and compresses audio recorded by an input device such as a microphone and either matches content on the device or sends a small “fingerprint” of data for servers to decipher. In both cases, a content database made of previously ingested content fingerprints is required.

The database is populated with coded strings of binary digits (generated by a mathematical algorithm) that uniquely identifies original audio signals (called digital audio fingerprints). Fingerprints can be generated by applying a cryptographic hash function to an input (in this case, audio signals). They are designed to be one-way functions, that is, functions which are infeasible to invert. Moreover, only a fraction of the audio is used to create the fingerprints. The combination of these two methodologies enables the possibility of storing digital fingerprints securely and in a privacy preserving manner, for example but not by way of limitation, without infringing copyright law.

According to an example implementation, in environment100shown inFIG. 1, the ACR system110is integrated with an artificial intelligence virtual assistant to provide context aware query for environmental parameters. According to an example implementation, the context service can receive an audio stream105including a command string. The context service analyzes the audio stream to separate the command string from the rest of the audio data. The remaining audio data other than the command string is analyzed to conduct queries using an ACR database135to match environmental sounds via fingerprinting. By identifying environmental sounds from audio data in an audio stream with a command string, the context service is able to provide additional inputs into the artificial intelligence engine to process the command string. For example, a user may provide a command string to an artificial intelligence assistant while a television, radio, home appliance, or other person in the room also is included as environmental sound in the audio stream.

A virtual assistant is a software agent that can perform tasks or services based on scheduling activities (e.g., pattern analysis, machine-learning, etc.) or detecting triggers (e.g., a voice command, video analysis, sensor data, etc.). Virtual assistants may include various types of interfaces to interact with, for example:Text (online chat), especially in an instant messaging application or other applicationVoice, for example, with Amazon Alexa on the Amazon Echo device, or Siri on an iPhoneBy taking and/or uploading images, as in the case of Samsung Bixby on the Samsung Galaxy S8

The Virtual Assistant-ACR Engine combination can receive input from hardware (e.g., a microphone), a file, or data stream. Described herein is a service that provides improved functionality with Voice Enabled assistants.

Technical Details

An audio-based ACR engine can include a microphone in order to capture users' media exposure. A client-side ACR Engine technology is described that is compatible with the operating system and proprietary requirements that power the Virtual Assistant. For example, for an ACR engine to work on a device running Siri, the ACR engine will have to be compatible with the correspondent iOS version as well as with the developer guidelines defined by Apple.

Basic Functionality

As shown inFIG. 2andFIG. 3, after a user makes a query at305regarding the content being consumed, the ACR Engine running on a Virtual Assistant sentences (e.g., listens for) content at115, extracts fingerprints at120from that content, and sends the fingerprints to an ACR content database135. The ACR Engine also sends ingested data at125to the ACR content database135. The Virtual Assistant then takes the processed information and gives a response to the user's query at330and130.

Server SideAs shown in environment200, content (i.e., television advertisements, YouTube promotions, songs, etc.) is ingested and fingerprinted at205.Fingerprints are saved in a database at210.Each content is tagged either manually or automatically with relevant metadata and information at215, for example:Advertisements and promotions can include, for example, Brand, Category, Parent Company, Product, etc.Songs information can include, for example, Title, Band, Album, Ratings, etc.Movie/series trailers can include, for example, Episode and Season Number, Rating, etc.

Client SideAs shown in environment300, the ACR Engine captures surrounding audio and transforms it into digital fingerprints at310.The audio fingerprints are matched against a content database made out of fingerprints at315. This database can be hosted in the device or in a server.If the database is hosted on a server, the ACR Engine will use the Virtual Assistant's network capabilities to send them to such server for the matching process to take place.

ResultsOnce the content has been matched (the fingerprints from the client side have a correspondence in the database), a result is generated at320.Such result will include the metadata and information the content was assigned at the ingestion phase115.Results are sent back to the Virtual Assistant at325, now ready to share them directly with the user or process and merge them with any other available datasets.

Implementation and Result Examples

1. A user is watching a commercial break on live TV/DVR/OTT.The user asks the Virtual Assistant, “What is this ad about?”The Virtual Assistant activates the ACR functionalities (capturing audio, sending fingerprints, generating results) and answers the question by providing information on the product and brand (i.e., “This is a Nike commercial featuring LeBron James's new shoes.”). Such information was included in the ACR Engine response.The user asks the Virtual Assistant, “What kind of product is this?”The Virtual Assistant activates the ACR functionalities (capturing audio, sending fingerprints, generating results) and answers the question by providing information about the category. Such information was included in the ACR Engine response.The user asks the Virtual Assistant, “How is this product rated?”The Virtual Assistant activates the ACR functionalities (capturing audio, sending fingerprints, generating results). The result from the ACR engine includes the product and brand names (i.e., “Nike Zoom 3”). The Virtual Assistant processes that information and:1. If the response includes a link to a review-enabled site where the product is available (i.e., Amazon, Target, Google), answers the user's question by providing information on the product reviews; OR2. Pulls extra data from other datasets (i.e., Amazon website), and answers the user's question by providing information on the product reviews.The user asks the Virtual Assistant, “I want to buy this product.”The Virtual Assistant activates the ACR functionalities (capturing audio, sending fingerprints, generating results). The result from the ACR Engine includes the product and brand names (i.e., “Nike Zoom 3”). The Virtual Assistant processes that information and:1. If the response includes a link to a store where the product is available, answers the user's question by providing a purchasing option; OR2. Pulls extra data from other datasets (i.e., Amazon website) and answers the user's question by providing a direct purchase option.2. A user is watching any video content where a song is being played.The user asks the Virtual Assistant, “Which song is this?”The Virtual Assistant activates the ACR functionalities (capturing audio, sending fingerprints, generating results). The result form the ACR Engine includes information about the song (i.e., “Wonderwall, by Oasis), that the Assistant uses to answer the user's question.The user asks the Virtual Assistant, “Buy this song.”The Virtual Assistant activates the ACR functionalities (capturing audio, sending fingerprints, generating results). The result from the ACR Engine includes information about the song (i.e., Wonderwall, by Oasis). The Virtual Assistant processes that information and:1. If the response includes a link to a store where the product is available, answers the user's question by providing a purchase option; OR2. Pulls extra data from other databases (i.e., iTunes) and answers the user's question by providing a direct purchase option.3. A user watches a movie/series promotion.The user asks the Virtual Assistant, “Which movie/series is this?”The Virtual Assistant activates the ACR functionalities (capturing audio, sending fingerprints, generating results). The result from the ACR Engine includes information about the song (i.e., Baywatch, with Dwayne Johnson), that the Assistant uses to answer the user's question.The user asks the Virtual Assistant, “Rent this movie/series.”The Virtual Assistant activates the ACR functionalities (capturing video, sending fingerprints, generating results). The result from the ACR Engine includes information about the song (i.e., Baywatch). The Virtual Assistant processes that information and:1. If the response includes a link to a store where the product is available, answers the user's question by providing a purchasing option; OR2. Pulls extra data from other datasets (i.e., iTunes), and answers the user's question by providing a direct purchase option.

Associated media file (i.e., audio file) stored in the database may be associated with other information and/or content for providing various services. In some example applications, the media file may be a song or music (Music M). Music M may be associated with availability and/or purchase information, for example, but not limited to, where, when, and how to buy Music M, the purchase price, associated promotions, etc., in the database. The media files may be provided to one or more media sources to promote one or more services. A service can be any service, such as an advertisement of products and/or services, a purchase/sell opportunity, a request for more information, etc. For example, Music M may be made available to broadcasters, radio stations, Internet streaming providers, TV broadcasting stations, sports bars, restaurants, etc. For example, information associated with songs are provided and an option may be given to select one or more of the provided songs to download, listen to, purchase, etc.

The above examples are not intended to be limiting. Further information may be used to perform additional queries, as would be understood by those skilled in the art. However, in all of the example implementations, it is important to note that the main contribution of the ACR Engine is providing the Virtual Assistant with context on the user's media exposure. Until now, virtual assistants need a specific query in order to operate properly (i.e., “What's the rating of Baywatch?”) rather than a generic query (i.e., “What's this movie's rating?”). The ACR engine acts as an intermediate layer that makes the interaction between user and Virtual Assistant smoother.

TABLE 1Example Use Cases. The above examples are not intended to be limiting.With the Context Provided byUser QueryWithout Previous Contextthe ACR EngineWhat's the brand in this ad?The Virtual Assistant doesn'tACR Engine provides a resultknow which ad the user isto the Virtual Assistant, whichreferring to.uses it to query into otherdatasets that are analyzed.Response: “Nike, an Americanathletic footwear and apparelcompany founded in 1964.”Buy this song.The Virtual Assistant doesn'tACR Engine provides a resultknow which song the user isto the Virtual Assistant, whichasking about.uses it to query into otherdatasets that are analyzed.Response: “Wonderwall, byOasis, is ready to be purchasedand added to your library.

According to an example implementation of a use case, shown in environment400inFIG. 4, the following may occur with the present example implementations associated with the inventive concept:

A method comprising:A virtual assistant activates the ACR functionalities (capturing audio, sending fingerprints, generating results) at405;Receives an audio file comprising a voice command;Improves the quality of the audio file for processing including at410:Separating the voice command from remaining audio data in the audio file;Analyzing the audio data to identify one or more audio signals;Querying a content recognition system for each of the one or more audio signals;The result from the ACR Engine includes the context information (e.g., product and brand information such as “Nike Zoom 3”);The Virtual Assistant processes that information at415and in response to receiving a match for one or more audio signals,Locates supplemental information associated with the context information, for example,If the response includes a link to a store where the product is available, answers the user's question by providing a purchasing option;Sends a request to a third party or resource or searches public and proprietary resources to pull extra data from other datasets (i.e., iTunes); andProvides the user a response to the command string at420based on the context information associated with one of the environmental inputs, for example:Supplementing pronouns with context information and extra data from third party resources;And answers the user's question by providing a direct purchase option.

According to other implementations, the context service can be integrated with using an Artificial Intelligence Assistant and an ACR Engine to provide users with content recommendations including or in addition to purchasing options on the content being consumed.

FIG. 5shows an example environment suitable for some example implementations. Environment500includes devices505-555, and each is communicatively connected to at least one other device via, for example, network560(e.g., by wired and/or wireless connections). Some devices may be communicatively connected to one or more storage devices530and545. Devices505-555may include, but are not limited to, a computer505(e.g., a laptop computing device), a mobile device510(e.g., a smartphone or tablet), a television515, a device associated with a vehicle520, a server computer525, computing devices535-540, wearable technologies with processing power (e.g., smart watch)550, smart speaker555, and storage devices530and545.

Example implementations may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes, or it may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium.

A computer-readable storage medium may involve tangible mediums such as, but not limited to optical disks, magnetic disks, read-only memories, random access memories, solid state devices and drives, or any other types of tangible or non-tangible media suitable for storing electronic information. A computer-readable signal medium may include mediums such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Computer programs can involve pure software implementations that involve instructions that perform the operations of the desired implementation.

FIG. 6shows an example computing environment with an example computing device suitable for implementing at least one example embodiment. Computing device1005in computing environment1000can include one or more processing units, cores, or processors1010, memory1015(e.g., RAM, ROM, and/or the like), internal storage1020(e.g., magnetic, optical, solid state storage, and/or organic), and I/O interface1025, all of which can be coupled on a communication mechanism or bus1030for communicating information. Processors1010can be general purpose processors (CPUs) and/or special purpose processors (e.g., digital signal processors (DSPs), graphics processing units (GPUs), and others).

In some example embodiments, computing environment1000may include one or more devices used as analog-to-analog converters, digital-to-analog converters, and/or radio frequency handlers.

Computing device1005can be communicatively coupled to external storage1045and network1050for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configuration. Computing device1005or any connected computing device can be functioning as, providing services of, or referred to as a server, client, thin server, general machine, special-purpose machine, or another label.

Computing device1005can be used to implement techniques, methods, applications, processes, or computer-executable instructions to implement at least one embodiment (e.g., a described embodiment). Computer-executable instructions can be retrieved from transitory media and stored on and retrieved from non-transitory media. The executable instructions can be originated from one or more of any programming, scripting, and machine languages (e.g., C, C++, Java, Visual Basic, Python, Perl, JavaScript, and others).

Processor(s)1010can execute under any operating system (OS) (not shown), in a native or virtual environment. To implement a described embodiment, one or more applications can be deployed that include logic unit1060, application programming interface (API) unit1065, input unit1070, output unit1075, media identifying unit1080, and inter-communication mechanism1095for the different units to communicate with each other, with the OS, and with other applications (not shown). For example, media identifying unit1080, media processing unit1085, and context information processing unit1090may implement one or more processes described above. The described units and elements can be varied in design, function, configuration, or implementation and are not limited to the descriptions provided.

In some examples, logic unit1060may be configured to control the information flow among the units and direct the services provided by API unit1065, input unit1070, output unit1075, media identifying unit1080, media processing unit1085, and media pre-processing unit to implement an embodiment described above. For example, the flow of one or more processes or implementations may be controlled by logic unit1060alone or in conjunction with API unit1065.

As is known in the art, the operations described above can be performed by hardware, software, or some combination of hardware and software. Various aspects of the example implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which if executed by a processor, would cause the processor to perform a method to carry out implementations of the present application.

Further, some example implementations of the present application may be performed solely in hardware, whereas other example implementations may be performed solely in software. Moreover, the various functions described can be performed in a single unit, or the functions can be spread out across a number of components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions can be stored on the medium in a compressed and/or encrypted format.

The example implementations may have various differences and advantages over related art. For example, but not by way of limitation, as opposed to instrumenting web pages with JavaScript as known in the related art, text and mouse (i.e., pointing) actions may be detected and analyzed in video documents. Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and/or components of the described example implementations may be used singly or in any combination. It is intended that the specification and example implementations be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.