Patent Description:
<CIT> describes methods, systems, and computer program products that enable users to provide explicit declarations that are used to generate recommendations for the users. An explicit declaration is received from a user of a user device. The explicit declaration is configured to influence a subsequent recommendation. The words of the explicit declaration are processed to generate a record. A recommendation rule is generated based on the generated record. The recommendation rule is executed to generate a recommendation for the user. The generated recommendation is provided to the user.

<CIT> describes a multimedia processing method comprising: converting multimedia items into versions of multiple compression levels for storage within a media catalog; and estimating the likelihood of a user consuming each multimedia item based on a consumption history of items within said media catalog and said user; computing the multimedia items to be downloaded and a per-item compression level in dependence on resource constraints at a client device of said user and said estimated consumption likelihoods; downloading a compressed version of the multimedia items; and storing a record of consumed downloaded items.

<CIT> describes a computer implemented method and system for processing a query may include receiving a query from a user device; categorizing the query to identify one or more content sources; formatting the query according to one or more content source specifics for the one or more content sources; transmitting the formatted query for the one or more content sources to the one or more content sources; merging results in response to the formatted query from the one or more content sources based at least in part on one or more factors; and formatting the results for delivering to the user device.

At least one aspect of the disclosure is directed to a system configured to allocate content sources in a voice activated packet-based computer network environment. The system can include a data processing system that includes one or more processors. A natural language processor component, executed by a data processing system, can receive, at an interface of a data processing system, data packets that can include an input audio signal. The input audio signal can be detected by a sensor of a client computing device and transmitted from the client device to the data processing system. The natural language processor component can parse the input audio signal to identify a request and a trigger keyword corresponding to the request. A direct action application programming interface can generate, based on at least one of the request and the trigger keyword, a first action data structure. A content selector component, executed by the data processing system, can select a digital component based on at least one of the request and the trigger keyword. A source allocation component can identify a plurality of candidate content sources after selecting the digital component. Each of the plurality of content sources may include a respective instance of the digital component. The source allocation component can determine a characteristic for each of the plurality of candidate content sources and for the instances of the digital component. The source allocation component can select a content source from the plurality of candidate content sources based on the characteristics for each of the plurality of candidate content sources and the instances of the digital components. The client computing device can request an instance of the content item from the content source.

At least one aspect of the disclosure is directed to a method to allocate content sources in a voice activated packet-based computer network environment. The method may include receiving, by a natural language processor component executed by a data processing system, data packets that can include an input audio signal. The input audio signal can be detected by a sensor of a client device and transmitted to the data processing system by the client device. The method may include parsing, by the natural language processor component, the input audio signal to identify a request and a trigger keyword corresponding to the request. The method may include generating, by a direct action application programming interface and based on at least one of the request and the trigger keyword, a first action data structure. The method may include selecting, by a content selector component, a digital component based on at least one of the request and the trigger keyword. The method may include identifying, by a source allocation component, a plurality of candidate content sources after identifying the digital component. Each of the plurality of content sources may include a respective instance of the digital component. The method may include determining, by the source allocation component, a characteristic for each of the plurality of candidate content sources and the instances of the digital component. The method may include selecting, by the source allocation component, a content source from the plurality of candidate content sources based on the characteristics for each of the plurality of candidate content sources and the instances of the digital components. The method may include requesting, by the source allocation component, an instance of the digital component from the content source.

Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems for multi-modal transmission of packetized data in a voice activated data packet-based computer network environment. The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways.

Systems and methods of the present disclosure relate generally to allocation of content resources in a voice activated packet (or other protocol) based computer network. A request for a digital component, by a client computing device, can be fulfilled by multiple content sources. For example, each of the content sources can include an instance of the digital component and be configured to provide the instance of the digital component to the requesting client computing device. Fulfillment by the different content sources can consume different amounts of computational and network resources. For example, some instances of the digital component may be of higher quality (and have a larger file size) or may be located at geographically remote locations. The selection of the content provider can therefore improve the efficiency and effectiveness of data packet transmission over one or more computer networks by, for example, selecting content providers that can provide the digital component to the client computing device while using fewer computational and network resources. The system can rank possible candidate content sources based on technical or computing parameters such as processor capability or utilization rate, memory capability or availability, battery status, available power, network bandwidth utilization, interface parameters, file characteristics, bandwidth availability, or any combination thereof. By selecting an appropriate candidate content source to provide the digital component for rendering from the client computing device, the data processing system can reduce network bandwidth usage, latency, or processing utilization or power consumption of the client computing device that renders the digital component. This saves processing power and other computing resources such as memory, reduces electrical power consumption by the data processing system and the reduced data transmissions via the computer network reduces bandwidth requirements and usage of the data processing system.

The systems and methods described herein can include a data processing system that receives an input audio query, which can also be referred to herein as an input audio signal. From the input audio query, the data processing system can identify a request and a trigger keyword corresponding to the request. Based on the trigger keyword or the request, the data processing system can generate a first action data structure. For example, the first action data structure can include an organic response to the input audio query received from a client computing device, and the data processing system can provide the first action data structure to the same client computing device for rendering as audio output via the same interface from which the request was received.

The data processing system can also select at least one digital component based on the trigger keyword or the request. The digital components can be content items, such as digital documents, videos, songs, webpages, portions of webpages, or other electronic files. The digital components can be the data associated with a user or a user account. For example, a digital component can be the emails or other data associated with a user's email account. The data processing system can identify or determine a plurality of candidate content sources that can provide the digital component to the client computing device. The data processing system can calculate one or more characteristics for each of the candidate content sources.

The data processing system can provide the digital component or the first action data structure by packet or other protocol based data message transmission via a computer network to a client computing device. The data processing system can initiate a session between the selected candidate content source and the client computing device such that the candidate content source provides the client computing device the digital component rather than relaying the digital component to the client computing device through the data processing system.

The digital component can cause an audio driver component of the client computing device to generate an acoustic wave, e.g., an audio output, which can be output from the client computing device. The audio (or other) output can correspond to the first action data structure or to the digital component. For example the first action data structure can be routed as audio output, and the digital component can be routed as a text based message. By routing the first action data structure and the digital component to different interfaces, the data processing system can conserve resources utilized by each interface, relative to providing both the first action data structure and the digital component to the same interface. This results in fewer data processing operations, less memory usage, or less network bandwidth utilization by the selected interfaces (or their corresponding devices) than would be the case without separation and independent routing of the first action data structure and the digital component.

<FIG> depicts an example system <NUM> for multi-modal transmission of packetized data in a voice activated data packet (or other protocol) based computer network environment. The system <NUM> can include at least one data processing system <NUM>. The data processing system <NUM> can include at least one server having at least one processor. For example, the data processing system <NUM> can include a plurality of servers located in at least one data center or server farm. The data processing system <NUM> can determine, from an input audio signal a request and a trigger keyword associated with the request. Based on the request and trigger keyword the data processing system <NUM> can determine or select at least one action data structure, and can select at least one digital component (and initiate other actions as described herein). The data processing system <NUM> can identify candidate interfaces for rendering of the action data structures or the digital components, and can provide the action data structures or the digital components for rendering by one or more candidate interfaces on one or more client computing devices based on resource utilization values for or of the candidate interfaces, for example as part of a voice activated communication or planning system. The action data structures (or the digital components) can include one or more audio files that when rendered provide an audio output or acoustic wave. The action data structures or the digital components can include other content (e.g., text, video, or image content) in addition to audio content.

The data processing system <NUM> can include multiple, logically-grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm or a machine farm. The servers can be geographically dispersed. A data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous - one or more of the servers or machines can operate according to one or more type of operating system platform. The data processing system <NUM> can include servers in a data center that are stored in one or more high-density rack systems, along with associated storage systems, located for example in an enterprise data center. The data processing system <NUM> with consolidated servers in this way can improve system manageability, data security, the physical security of the system, and system performance by locating servers and high performance storage systems on localized high performance networks. Centralization of all or some of the data processing system <NUM> components, including servers and storage systems, and coupling them with advanced system management tools allows more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage.

The data processing system <NUM> can include at least one natural language processor (NLP) component <NUM>, at least one interface <NUM>, at least one prediction component <NUM>, at least one content selector component <NUM>, at least one audio signal generator component <NUM>, at least one direct action application programming interface (API) <NUM>, at least one source allocation component <NUM>, and at least one data repository <NUM>. The NLP component <NUM>, interface <NUM>, prediction component <NUM>, content selector component <NUM>, audio signal generator component <NUM>, direct action API <NUM>, and source allocation component <NUM> can each include at least one processing unit, server, virtual server, circuit, engine, agent, appliance, or other logic device such as programmable logic arrays configured to communicate with the data repository <NUM> and with other computing devices (e.g., at least one client computing device <NUM>, at least one content provider computing device <NUM>, or at least one service provider computing device <NUM>) via the at least one computer network <NUM>. The network <NUM> can include computer networks such as the internet, local, wide, metro or other area networks, intranets, satellite networks, other computer networks such as voice or data mobile phone communication networks, and combinations thereof.

The network <NUM> can include or constitute a display network, e.g., a subset of information resources available on the internet that are associated with a content placement or search engine results system, or that are eligible to include third party digital components as part of a digital component placement campaign. The network <NUM> can be used by the data processing system <NUM> to access information resources such as web pages, web sites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed by the client computing device <NUM>. For example, via the network <NUM> a user of the client computing device <NUM> can access information or data provided by the data processing system <NUM>, the content provider computing device <NUM> or the service provider computing device <NUM>.

The network <NUM> can include, for example a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, a SDH (Synchronous Digital Hierarchy) network, a wireless network or a wireline network, and combinations thereof. The network <NUM> can include a wireless link, such as an infrared channel or satellite band. The topology of the network <NUM> may include a bus, star, or ring network topology. The network <NUM> can include mobile telephone networks using any protocol or protocols used to communicate among mobile devices, including advanced mobile phone protocol ("AMPS"), time division multiple access ("TDMA"), code-division multiple access ("CDMA"), global system for mobile communication ("GSM"), general packet radio services ("GPRS") or universal mobile telecommunications system ("UMTS"). Different types of data may be transmitted via different protocols, or the same types of data may be transmitted via different protocols.

The client computing device <NUM>, the content provider computing device <NUM>, and the service provider computing device <NUM> can each include at least one logic device such as a computing device having a processor to communicate with each other or with the data processing system <NUM> via the network <NUM>. The client computing device <NUM>, the content provider computing device <NUM>, and the service provider computing device <NUM> can each include at least one server, processor or memory, or a plurality of computation resources or servers located in at least one data center. The client computing device <NUM>, the content provider computing device <NUM>, and the service provider computing device <NUM> can each include at least one computing device such as a desktop computer, laptop, tablet, personal digital assistant, smartphone, portable computer, server, thin client computer, virtual server, or other computing device.

The client computing device <NUM> can include at least one sensor <NUM>, at least one transducer <NUM>, at least one audio driver <NUM>, and at least one speaker <NUM>. The sensor <NUM> can include a microphone or audio input sensor. The transducer <NUM> can convert the audio input into an electronic signal, or vice-versa. The audio driver <NUM> can include a script or program executed by one or more processors of the client computing device <NUM> to control the sensor <NUM>, the transducer <NUM> or the audio driver <NUM>, among other components of the client computing device <NUM> to process audio input or provide audio output. The speaker <NUM> can transmit the audio output signal.

The client computing device <NUM> can be associated with an end user that enters voice queries as audio input into the client computing device <NUM> (via the sensor <NUM>) and receives audio output in the form of a computer generated voice that can be provided from the data processing system <NUM> (or the content provider computing device <NUM> or the service provider computing device <NUM>) to the client computing device <NUM>, output from the speaker <NUM>. The audio output can correspond to an action data structure received from the direct action API <NUM>, or a digital component selected by the content selector component <NUM>. The computer generated voice can include recordings from a real person or computer generated language.

The content provider computing device <NUM> (or the data processing system <NUM> or service provider computing device <NUM>) can provide audio based digital components or action data structures for display by the client computing device <NUM> as an audio output. The action data structure of a digital component can include an organic response or offer for a good or service, such as a voice based message that states: "Today it will be sunny and <NUM> degrees at the beach" as an organic response to a voice-input query of "Is today a beach day?". The data processing system <NUM> (or other system <NUM> component such as the content provider computing device <NUM>) can also provide a digital component as a response, such as a voice or text message based digital component offering sunscreen.

The content provider computing device <NUM> or the data repository <NUM> can include memory to store a series of audio action data structures or digital components that can be provided in response to a voice based query. The action data structures and digital components can include packet based data structures for transmission via the network <NUM>. The content provider computing device <NUM> can also provide audio or text based digital components (or other digital components) to the data processing system <NUM> where they can be stored in the data repository <NUM>. The data processing system <NUM> can select the audio action data structures or text based digital components and provide (or instruct the content provider computing device <NUM> to provide) them to the same or different client computing devices <NUM> responsive to a query received from one of those client computing device <NUM>. The audio based action data structures can be exclusively audio or can be combined with text, image, or video data. The digital components can be exclusively text or can be combined with audio, image or video data.

The service provider computing device <NUM> can include at least one service provider natural language processor (NLP) component <NUM> and at least one service provider interface <NUM>. The service provider NLP component <NUM> (or other components such as a direct action API of the service provider computing device <NUM>) can engage with the client computing device <NUM> (via the data processing system <NUM> or bypassing the data processing system <NUM>) to create a back-and-forth real-time voice or audio based conversation (e.g., a session) between the client computing device <NUM> and the service provider computing device <NUM>. For example, the service provider interface <NUM> can receive or provide data messages (e.g., action data structures or digital components) to the direct action API <NUM> of the data processing system <NUM>. The direct action API <NUM> can also generate the action data structures independent from or without input from the service provider computing device <NUM>. The service provider computing device <NUM> and the content provider computing device <NUM> can be associated with the same entity. For example, the content provider computing device <NUM> can create, store, or make available digital components for beach related services, such as sunscreen, beach towels or bathing suits, and the service provider computing device <NUM> can establish a session with the client computing device <NUM> to respond to a voice input query about the weather at the beach, directions for a beach, or a recommendation for an area beach, and can provide these digital components to the end user of the client computing device <NUM> via an interface of the same client computing device <NUM> from which the query was received, a different interface of the same client computing device <NUM>, or an interface of a different client computing device. The data processing system <NUM>, via the direct action API <NUM>, the NLP component <NUM> or other components can also establish the session with the client computing device, including or bypassing the service provider computing device <NUM>, for example to provide an organic response to a query related to the beach.

The data repository <NUM> can include one or more local or distributed databases, and can include a database management system. The data repository <NUM> can include computer data storage or memory and can store one or more parameters <NUM>, one or more policies <NUM>, content data <NUM>, or templates <NUM> among other data. The parameters <NUM>, policies <NUM>, and templates <NUM> can include information such as rules about a voice based session between the client computing device <NUM> and the data processing system <NUM> (or the service provider computing device <NUM>). The content data <NUM> can include digital components for audio output or associated metadata, as well as input audio messages that can be part of one or more communication sessions with the client computing device <NUM>.

The system <NUM> can optimize processing of action data structures and digital components in a voice activated data packet (or other protocol) environment. For example, the data processing system <NUM> can include or be part of a voice activated assistant service, voice command device, intelligent personal assistant, knowledge navigator, event planning, or other assistant program. The data processing system <NUM> can provide one or more instances of action data structures as audio output for display from the client computing device <NUM> to accomplish tasks related to an input audio signal. For example, the data processing system can communicate with the service provider computing device <NUM> or other third party computing devices to generate action data structures with information about a beach, among other things. For example, an end user can enter an input audio signal into the client computing device <NUM> of: "OK, I would like to go to the beach this weekend" and an action data structure can indicate the weekend weather forecast for area beaches, such as "it will be sunny and <NUM> degrees at the beach on Saturday, with high tide at 3pm.

The action data structures can include a number of organic or non-sponsored responses to the input audio signal. For example, the action data structures can include a beach weather forecast or directions to a beach. The action data structures in this example include organic, or non-sponsored content that is directly responsive to the input audio signal. The digital components responsive to the input audio signal can include sponsored or non-organic content, such as an offer to buy sunscreen from a convenience store located near the beach. In this example, the organic action data structure (beach forecast) is responsive to the input audio signal (a query related to the beach), and the digital component (a reminder or offer for sunscreen) is also responsive to the same input audio signal. The data processing system <NUM> can evaluate system <NUM> parameters (e.g., power usage, available displays, formats of displays, memory requirements, bandwidth usage, power capacity or time of input power (e.g., internal battery or external power source such as a power source from a wall output)) to provide the action data structure and the digital component to different candidate interfaces on the same client computing device <NUM>, or to different candidate interfaces on different client computing devices <NUM>.

The data processing system <NUM> can include an application, script or program installed at the client computing device <NUM>, such as an app to communicate input audio signals (e.g., as data packets via a packetized or other protocol based transmission) to at least one interface <NUM> of the data processing system <NUM> and to drive components of the client computing device <NUM> to render output audio signals (e.g., for action data structures) or other output signals (e.g., digital components). The data processing system <NUM> can receive data packets or other signal that includes or identifies an input audio signal. For example, the data processing system <NUM> can execute or run the NLP component <NUM> to receive the input audio signal.

The NLP component <NUM> can convert the input audio signal into recognized text by comparing the input signal against a stored, representative set of audio waveforms (e.g., in the data repository <NUM>) and choosing the closest matches. The representative waveforms are generated across a large set of users, and can be augmented with speech samples. After the audio signal is converted into recognized text, the NLP component <NUM> can match the text to words that are associated, for example via training across users or through manual specification, with actions that the data processing system <NUM> can serve.

The input audio signal can be detected by the sensor <NUM> (e.g., a microphone) of the client computing device. Via the transducer <NUM>, the audio driver <NUM>, or other components the client computing device <NUM> can provide the input audio signal to the data processing system <NUM> (e.g., via the network <NUM>) where it can be received (e.g., by the interface <NUM>) and provided to the NLP component <NUM> or stored in the data repository <NUM> as content data <NUM>.

The NLP component <NUM> can receive or otherwise obtain the input audio signal. From the input audio signal, the NLP component <NUM> can identify at least one request or at least one trigger keyword corresponding to the request. The request can indicate intent or subject matter of the input audio signal. The trigger keyword can indicate a type of action likely to be taken. For example, the NLP component <NUM> can parse the input audio signal to identify at least one request to go to the beach for the weekend. The trigger keyword can include at least one word, phrase, root or partial word, or derivative indicating an action to be taken. For example, the trigger keyword "go" or "to go to" from the input audio signal can indicate a need for transport or a trip away from home. In this example, the input audio signal (or the identified request) does not directly express an intent for transport, however the trigger keyword indicates that transport is an ancillary action to at least one other action that is indicated by the request.

The prediction component <NUM> (or other mechanism of the data processing system <NUM>) can generate, based on the request or the trigger keyword, at least one action data structure associated with the input audio signal. The action data structure can indicate information related to subject matter of the input audio signal. The action data structure can include one or more than one action, such as organic responses to the input audio signal. For example, the input audio signal "OK, I would like to go to the beach this weekend" can include at least one request indicating an interest for a beach weather forecast, surf report, or water temperature information, and at least one trigger keyword, e.g., "go" indicating travel to the beach, such as a need for items one may want to bring to the beach, or a need for transportation to the beach. The prediction component <NUM> can generate or identify subject matter for at least one action data structure, an indication of a request for a beach weather forecast, as well as subject matter for a digital component, such as an indication of a query for sponsored content related to spending a day at a beach. From the request or the trigger keyword the prediction component <NUM> (or other system <NUM> component such as the NLP component <NUM> or the direct action API <NUM>) predicts, estimates, or otherwise determines subject matter for action data structures or for digital components. From this subject matter, the direct action API <NUM> can generate at least one action data structure and can communicate with at least one content provider computing device <NUM> to obtain at least one digital component. The prediction component <NUM> can access the parameters <NUM> or policies <NUM> in the data repository <NUM> to determine or otherwise estimate requests for action data structures or digital components. For example, the parameters <NUM> or policies <NUM> could indicate requests for a beach weekend weather forecast action or for digital components related to beach visits, such as a digital component for sunscreen.

The content selector component <NUM> can obtain indications of any of the interest in or request for the action data structure or for the digital component. For example, the prediction component <NUM> can directly or indirectly (e.g., via the data repository <NUM>) provide an indication of the action data structure or digital component to the content selector component <NUM>. The content selector component <NUM> can obtain this information from the data repository <NUM>, where it can be stored as part of the content data <NUM>. The indication of the action data structure can inform the content selector component <NUM> of a need for area beach information, such as a weather forecast or products or services the end user may need for a trip to the beach.

From the information received by the content selector component <NUM>, e.g., an indication of a forthcoming trip to the beach, the content selector component <NUM> can identify at least one digital component. The digital component can be responsive or related to the subject matter of the input audio query. For example, the digital component can include data message identifying a store near the beach that has sunscreen, or offering a taxi ride to the beach. The content selector component <NUM> can query the data repository <NUM> to select or otherwise identify the digital component, e.g., from the content data <NUM>. The content selector component <NUM> can also select the digital component from the content provider computing device <NUM>. For example responsive to a query received from the data processing system <NUM>, the content provider computing device <NUM> can provide a digital component to the data processing system <NUM> (or component thereof) for eventual output by the client computing device <NUM> that originated the input audio signal, or for output to the same end user by a different client computing device <NUM>.

The audio signal generator component <NUM> can generate or otherwise obtain an output signal that includes the digital component (as well as the action data structure) responsive to the input audio signal. For example, the data processing system <NUM> can execute the audio signal generator component <NUM> to generate or create an output signal corresponding to the action data structure or to the digital component. The interface <NUM> of the data processing system <NUM> can provide or transmit one or more data packets that include the output signal via the computer network <NUM> to any client computing device <NUM>. The interface <NUM> can be designed, configured, constructed, or operational to receive and transmit information using, for example, data packets. The interface <NUM> can receive and transmit information using one or more protocols, such as a network protocol. The interface <NUM> can include a hardware interface, software interface, wired interface, or wireless interface. The interface <NUM> can facilitate translating or formatting data from one format to another format. For example, the interface <NUM> can include an application programming interface that includes definitions for communicating between various components, such as software components of the system <NUM>.

The data processing system <NUM> can provide the output signal including the action data structure from the data repository <NUM> or from the audio signal generator component <NUM> to the client computing device <NUM>. The data processing system <NUM> can provide the output signal including the digital component from the data repository <NUM> or from the audio signal generator component <NUM> to the same or to a different client computing device <NUM>.

The data processing system <NUM> can also instruct, via data packet transmissions, the content provider computing device <NUM> or the service provider computing device <NUM> to provide the output signal (e.g., corresponding to the action data structure or to the digital component) to the client computing device <NUM>. The output signal can be obtained, generated, transformed to or transmitted as one or more data packets (or other communications protocol) from the data processing system <NUM> (or other computing device) to the client computing device <NUM>.

The content selector component <NUM> can select the digital component or the action data structure as part of a real-time content selection process. For example, the action data structure can be provided to the client computing device <NUM> for transmission as audio output by an interface of the client computing device <NUM> in a conversational manner in direct response to the input audio signal. The real-time content selection process to identify the action data structure and provide the digital component to the client computing device <NUM> can occur within one minute or less from the time of the input audio signal and be considered real-time. The data processing system <NUM> can also identify and provide the digital component to at least one interface of the client computing device <NUM> that originated the input audio signal, or to a different client computing device <NUM>.

The action data structure (or the digital component), for example obtained or generated by the audio signal generator component <NUM> transmitted via the interface <NUM> and the computer network <NUM> to the client computing device <NUM>, can cause the client computing device <NUM> to execute the audio driver <NUM> to drive the speaker <NUM> to generate an acoustic wave corresponding to the action data structure or to the digital component. The acoustic wave can include words of or corresponding to the action data structure or digital component.

The acoustic wave representing the action data structure can be output from the client computing device <NUM> separately from the digital component. For example, the acoustic wave can include the audio output of "Today it will be sunny and <NUM> degrees at the beach. " In this example, the data processing system <NUM> obtains the input audio signal of, for example, "OK, I would like to go to the beach this weekend. " From this information the NLP component <NUM> identifies at least one request or at least one trigger keyword, and the prediction component <NUM> uses the requests or trigger keywords to identify a request for an action data structure or for a digital component. The content selector component <NUM> (or other component) can identify, select, or generate a digital component for, e.g., sunscreen available near the beach. The direct action API <NUM> (or other component) can identify, select, or generate an action data structure for, e.g., the weekend beach forecast. The data processing system <NUM> or component thereof such as the audio signal generator component <NUM> can provide the action data structure for output by an interface of the client computing device <NUM>. For example, the acoustic wave corresponding to the action data structure can be output from the client computing device <NUM>. The data processing system <NUM> can provide the digital component for output by a different interface of the same client computing device <NUM> or by an interface of a different client computing device <NUM>.

The packet based data transmission of the action data structure by data processing system <NUM> to the client computing device <NUM> can include a direct or real-time response to the input audio signal of "OK, I would like to go to the beach this weekend" so that the packet based data transmissions via the computer network <NUM> that are part of a communication session between the data processing system <NUM> and the client computing device <NUM> with the flow and feel of a real-time person to person conversation. This packet based data transmission communication session can also include the content provider computing device <NUM> or the service provider computing device <NUM>.

The content selector component <NUM> can select the digital component or action data structure based on at least one request or at least one trigger keyword of the input audio signal. For example, the requests of the input audio signal "OK, I would like to go to the beach this weekend" can indicate subject matter of the beach, travel to the beach, or items to facilitate a trip to the beach. The NLP component <NUM> or the prediction component <NUM> (or other data processing system <NUM> components executing as part of the direct action API <NUM>) can identify the trigger keyword "go" "go to" or "to go to" and can determine a transportation request to the beach based at least in part on the trigger keyword. The NLP component <NUM> (or other system <NUM> component) can also determine a solicitation for digital components related to beach activity, such as for sunscreen or beach umbrellas. Thus, the data processing system <NUM> can infer actions from the input audio signal that are secondary requests (e.g., a request for sunscreen) that are not the primary request or subject of the input audio signal (information about the beach this weekend).

The action data structures and digital components can correspond to subject matter of the input audio signal. The direct action API <NUM> can execute programs or scripts, for example from the NLP component <NUM>, the prediction component <NUM>, or the content selector component <NUM> to identify action data structures or digital components for one or more of these actions. The direct action API <NUM> can execute a specified action to satisfy the end user's intention, as determined by the data processing system <NUM>. Depending on the action specified in its inputs, the direct action API <NUM> can execute code or a dialog script that identifies the parameters required to fulfill a user request. Such code can lookup additional information, e.g., in the data repository <NUM>, such as the name of a home automation service, or it can provide audio output for rendering at the client computing device <NUM> to ask the end user questions such as the intended destination of a requested taxi. The direct action API <NUM> can determine necessary parameters and can package the information into an action data structure, which can then be sent to another component such as the content selector component <NUM> or to the service provider computing device <NUM> to be fulfilled.

The direct action API <NUM> of the data processing system <NUM> can generate, based on the request or the trigger keyword, the action data structures. The action data structures can be generated responsive to the subject matter of the input audio signal. The action data structures can be included in the messages that are transmitted to or received by the service provider computing device <NUM>. Based on the input audio signal parsed by the NLP component <NUM>, the direct action API <NUM> can determine to which, if any, of a plurality of service provider computing devices <NUM> the message should be sent. For example, if an input audio signal includes "OK, I would like to go to the beach this weekend," the NLP component <NUM> can parse the input audio signal to identify requests or trigger keywords such as the trigger keyword word "to go to" as an indication of a need for a taxi. The direct action API <NUM> can package the request into an action data structure for transmission as a message to a service provider computing device <NUM> of a taxi service. The message can also be passed to the content selector component <NUM>. The action data structure can include information for completing the request. In this example, the information can include a pick up location (e.g., home) and a destination location (e.g., a beach). The direct action API <NUM> can retrieve a template <NUM> from the data repository <NUM> to determine which fields to include in the action data structure. The direct action API <NUM> can retrieve content from the data repository <NUM> to obtain information for the fields of the data structure. The direct action API <NUM> can populate the fields from the template with that information to generate the data structure. The direct action API <NUM> can also populate the fields with data from the input audio signal. The templates <NUM> can be standardized for categories of service providers or can be standardized for specific service providers. For example, ride sharing service providers can use the following standardized template <NUM> to create the data structure: {client_device_identifier; authentication_credentials; pick_up_location; destination_location; no_passengers; service_level}.

The content selector component <NUM> can identify, select, or obtain multiple digital components resulting from a multiple content selection processes. The content selection processes can be real-time, e.g., part of the same conversation, communication session, or series of communications sessions between the data processing system <NUM> and the client computing device <NUM> that involve common subject matter. The conversation can include asynchronous communications separated from one another by a period of hours or days, for example. The conversation or communication session can last for a time period from receipt of the first input audio signal until an estimated or known conclusion of a final action related to the first input audio signal, or receipt by the data processing system <NUM> of an indication of a termination or expiration of the conversation. For example, the data processing system <NUM> can determine that a conversation related to a weekend beach trip begins at the time or receipt of the input audio signal and expires or terminates at the end of the weekend, e.g., Sunday night or Monday morning. The data processing system <NUM> that provides action data structures or digital components for rendering by one or more interfaces of the client computing device <NUM> or of another client computing device <NUM> during the active time period of the conversation (e.g., from receipt of the input audio signal until a determined expiration time) can be considered to be operating in real-time. In this example the content selection processes and rendering of the digital components and action data structures occurs in real time.

The data processing system <NUM> can include the interface management component to poll, determine, identify, or select interfaces for rendering of the action data structures and of the digital components related to the input audio signal. For example, the interface management component can identify one or more candidate interfaces of client computing devices <NUM> associated with an end user that entered the input audio signal (e.g., "What is the weather at the beach today?") into one of the client computing devices <NUM> via an audio interface. The interfaces can include hardware such as sensor <NUM> (e.g., a microphone), speaker <NUM>, or a screen size of a computing device, alone or combined with scripts or programs (e.g., the audio driver <NUM>) as well as apps, computer programs, online documents (e.g., webpage) interfaces and combinations thereof.

The interfaces can include social media accounts, text message applications, or email accounts associated with an end user of the client computing device <NUM> that originated the input audio signal. Interfaces can include the audio output of a smartphone, or an app based messaging device installed on the smartphone, or on a wearable computing device, among other client computing devices <NUM>. The interfaces can also include display screen parameters (e.g., size, resolution), audio parameters, mobile device parameters, (e.g., processing power, battery life, existence of installed apps or programs, or sensor <NUM> or speaker <NUM> capabilities), content slots on online documents for text, image, or video renderings of digital components, chat applications, laptops parameters, smartwatch or other wearable device parameters (e.g., indications of their display or processing capabilities), or virtual reality headset parameters.

The interface management component can poll a plurality of interfaces to identify candidate interfaces. Candidate interfaces include interfaces having the capability to render a response to the input audio signal, (e.g., the action data structure as an audio output, or the digital component that can be output in various formats including non-audio formats). The interface management component can determine parameters or other capabilities of interfaces to determine that they are (or are not) candidate interfaces. For example, the interface management component can determine, based on parameters <NUM> of the digital component or of a first client computing device <NUM> (e.g., a smartwatch wearable device), that the smartwatch includes an available visual interface of sufficient size or resolution to render the digital component. The interface management component can also determine that the client computing device <NUM> that originated the input audio signal has a speaker <NUM> hardware and installed program e.g., an audio driver or other script to render the action data structure.

The interface management component (or other data processing system <NUM> component) can convert the digital component for delivery in a modality compatible with the candidate interface. For example, if the candidate interface is a display of a smartwatch, smartphone, or tablet computing device, the interface management component can size the digital component for appropriate visual display given the dimensions of the display screen associated with the interface. The interface management component can also convert the digital component to a packet or other protocol based format, including proprietary or industry standard format for transmission to the client computing device <NUM> associated with the selected interface. The interface selected by the interface management component for the digital component can include an interface accessible from multiple client computing devices <NUM> by the end user. For example, the interface can be or include a social media account that the end user can access via the client computing device <NUM> that originated the input audio signal (e.g., a smartphone) as well as other client computing devices such as tablet or desktop computers or other mobile computing devices.

The interface management component can also select at least one candidate interface for the action data structure. This interface can be the same interface from which the input audio signal was obtained, e.g., a voice activated assistant service executed at a client computing device <NUM>. This can be the same interface or a different interface than the interface management component selects for the digital component. The interface management component (or other data processing system <NUM> components) can provide the action data structure to the same client computing device <NUM> that originated the input audio signal for rendering as audio output as part of the assistant service. The interface management component can also transmit or otherwise provide the digital component to the selected interface for the digital component, in any converted modality appropriate for rendering by the selected interface.

Thus, the interface management component can provide the action data structure as audio output for rendering by an interface of the client computing device <NUM> responsive to the input audio signal received by the same client computing device <NUM>. The interface management component can also provide the digital component for rendering by a different interface of the same client computing device <NUM> or of a different client computing device <NUM> associated with the same end user. For example, the action data structure, e.g., "it will be sunny and <NUM> degrees at the beach on Saturday" can be provided for audio rendering by the client computing device as part of an assistant program interface executing in part at the client computing device <NUM>, and the digital component e.g., a text, audio, or combination digital component indicating that "sunscreen is available from the convenience store near the beach" can be provided for rendering by an interface of the same or a different computing device <NUM>, such as an email or text message accessible by the same or a different client computing device <NUM> associated with the end user.

Separating the digital component from the action data structure and sending the digital component as, for example, a text message rather than an audio message can result in reduced processing power for the client computing device <NUM> that accesses the digital component since, for example, text message data transmissions are less computationally intensive than audio message data transmissions. This separation can also reduce power usage, memory storage, or transmission bandwidth used to render the digital component. This results in increased processing, power, and bandwidth efficiencies of the system <NUM> and devices such as the client computing devices <NUM> and the data processing system <NUM>. This increases the efficiency of the computing devices that process these transactions, and increases the speed with which the digital components can be rendered. The data processing system <NUM> can process thousands, tens of thousands or more input audio signals simultaneously so the bandwidth, power, and processing savings can be significant and not merely incremental or incidental.

The interface management component can provide or deliver the digital component to the same client computing device <NUM> (or a different device) as the action data structure subsequent to delivery of the action data structure to the client computing device <NUM>. For example, the digital component can be provided for rendering via the selected interface upon conclusion of audio output rendering of the action data structure. The interface management component can also provide the digital component to the selected interface concurrent with the provision of the action data structure to the client computing device <NUM>. The interface management component can provide the digital component for delivery via the selected interface within a pre-determined time period from receipt of the input audio signal by the NLP component <NUM>. The time period, for example, can be any time during an active length of the conversation of session. For example, if the input audio signal is "I would like to go to the beach this weekend" the pre-determined time period can be any time from receipt of the input audio signal through the end of the weekend, e.g., the active period of the conversation. The pre-determined time period can also be a time triggered from rendering of the action data structure as audio output by the client computing device <NUM>, such as within <NUM> minutes, one hour or one day of this rendering.

The interface management component can provide the action data structure to the client computing device <NUM> with an indication of the existence of the digital component. For example, the data processing system <NUM> can provide the action data structure that renders at the client computing device <NUM> to provide the audio output "it will be sunny and <NUM> degrees at the beach on Saturday, check your email for more information. " The phrase "check your email for more information" can indicate the existence of a digital component, e.g., for sunscreen, provided by the data processing system <NUM> to an interface (e.g., email). In this example, sponsored content can be provided as digital components to the email (or other) interface and organic content such as the weather can be provided as the action data structure for audio output.

The data processing system <NUM> can include the source allocation component <NUM>. The source allocation component <NUM> can determine which elements of the system <NUM> include instances (or associated instances) of a digital component. For example, once the content selector component <NUM> selects a digital component, the source allocation component <NUM> can determine which of the content provider device <NUM>, the service provider device <NUM>, and the client computing device <NUM> have an instance of the digital component. The client computing device <NUM>, the service provider devices <NUM>, and the content provider devices <NUM> that have an instance of the digital component can be referred to as candidate content sources. The interfaces identified by the interface management component can be the interfaces of the candidate content sources. In some implementations, the source allocation component <NUM> can store, in the data repository <NUM>, an index of the digital components stored or offered by each of the candidate content sources. In some implementations, the source allocation component <NUM> can poll each of (or a sub-population of) the content provider devices <NUM> and service provider devices <NUM> to determine if the content provider devices <NUM> or the service provider devices <NUM> include an instance of the digital component. For example, the source allocation component <NUM> can send a request to a content provider device <NUM> for the digital component. If the content provider device <NUM> responds affirmatively, the source allocation component <NUM> can flag the content provider device <NUM> as a candidate content source. In some implementations, the content provider device's (or other potential candidate content source's) response can include data and other information about the potential candidate content source and its instance of the digital component that the source allocation component <NUM> can use to generate a characteristic for the potential candidate content source and for the instance of the digital component stored on each of the potential candidate content sources.

The source allocation component <NUM> can determine or calculate characteristics for each of the instances of the digital component. The characteristics can indicate the quality (or a quality metric) of the instance of the digital component. The source allocation component <NUM> can also calculate a characteristic (or quality metrics) for each of the candidate content sources. In some implementations, the source allocation component <NUM> can determine a characteristic for each of the instances of the digital component identified in the system <NUM>. For example, a candidate content source can include multiple instances of the digital component (e.g., a high-resolution and a low-resolution of the digital component). The source allocation component <NUM> can determine a separate characteristic for each of the instances of the candidate content source's digital component.

The source allocation component <NUM> can use the characteristics to indicate or determine if the candidate content source can provide to or the client computing device <NUM> can render the digital component. The characteristics for the digital component and candidate content sources can include parameters <NUM> obtained from the data repository <NUM> or other parameters obtained from the client computing device <NUM>, such as bandwidth or processing utilization or requirements, processing power, power requirements, battery status, memory utilization or capabilities, or other interface parameters that indicate the available of an interface to render action data structures or digital components. The battery status can indicate a type of power source (e.g., internal battery or external power source such as via an output), a charging status (e.g., currently charging or not), or an amount of remaining battery power.

The source allocation component <NUM> can use characteristics for the candidate content sources, qualities of their respective connection to the data processing system <NUM> or the client computing device <NUM>, and the characteristics of the respective instances of the digital component. For example, the characteristics can be based on the bandwidth of the connections, the utilization of the connections, the utilization of the candidate content sources, whether the instance is a free or paid-for instance of the digital component, quality of the digital component, file size of the digital component, encoding scheme of the digital component, file format of the digital component, distances between the client computing device <NUM> and the storage location of the instance of the digital component, location of the client computing device <NUM>, location of the digital component, or any combination thereof. For example, the source allocation component <NUM> can determine that the digital component is stored or available on a candidate content source that is a personal computer local (e.g., on the same Wi-Fi network) to the client computing device <NUM>, which the client computing device <NUM> has access to. In this example, the location characteristic can indicate that the candidate content source is available to provide the digital component. In another example, the client computing device <NUM> may not be on the same Wi-Fi network as the candidate content source that is the personal computer. In this example, the location characteristic can indicate that the candidate content source is not available to provide the digital component.

The source allocation component <NUM> can order or rank the candidate content sources (or their respective instances of the digital component) in a hierarchy or ranking based on the characteristics. For example, different parameters (e.g., processing requirements, display screen size, accessibility to the end user, and file quality) can be given different weights. The source allocation component <NUM> can rank one or more of the characteristics of the candidate interfaces based on their weights to determine an optimal corresponding candidate content source for providing the digital component. For example, based on this hierarchy, the source allocation component <NUM> can select the highest ranked candidate content source for providing the digital component to the client computing device <NUM>.

In some implementations, the candidate content source with the highest ranked characteristic can indicate that providing the digital component from that candidate content source would consume the least amount of computational or network resources. For example, the use of a local instance of the digital component, already stored on the client computing device <NUM>, can consume fewer resources when compared to a remotely stored instance of the digital component. In some implementations, the instance of the digital component with the highest characteristic can indicate that the delivery of the digital component from that candidate content source will provide a relatively higher user experience. For example, the client computing device <NUM> may include a low-quality version of the digital component that was downloaded to the client computing device <NUM> when the client computing device <NUM> was connected to the network <NUM> via a cellular connection. The source allocation component <NUM> may assign a relatively higher characteristic to an instance of the digital component that is remote to the client computing device <NUM> but is a high-quality version of the digital component. In some implementations, a relatively high characteristic for an instance of the digital component can over-ride a relatively low characteristic for the candidate content source. In other implementations, a relatively high characteristic for a candidate content source can over-ride a relatively low characteristic for the instance of the digital component. For example, the data processing system <NUM> can determine to send a low-resolution video to a client computing device <NUM> when the client computing device <NUM> has a low bandwidth connection to the data processing system <NUM>.

The data processing system <NUM> can also provide the action data structure with a prompt that queries the user to determine user interest in obtaining the digital component. For example, the action data structure can indicate "it will be sunny and <NUM> degrees at the beach on Saturday, would you like to hear about some services to assist with your trip?" The data processing system <NUM> can receive another input audio signal from the client computing device <NUM> in response to the prompt "would you like to hear about some services to assist with your trip?" such as "sure". The NLP component <NUM> can parse this response, e.g., "sure" and interpret it as authorization for audio rendering of the digital component by the client computing device <NUM>. In response, the data processing system <NUM> can provide the digital component for audio rendering by the same client computing device <NUM> from which the response "sure" originated.

The data processing system <NUM> can delay transmission of the digital component associated with the action data structure to optimize processing utilization. For example, the data processing system <NUM> provide the action data structure for rendering as audio output by the client computing device in real-time responsive to receipt of the input audio signal, e.g., in a conversational manner, and can delay digital component transmission until an off-peak or non-peak period of data center usage, which results in more efficient utilization of the data center by reducing peak bandwidth usage, heat output or cooling requirements. The data processing system <NUM> can also initiate a conversion or other activity associated with the digital component, such as ordering a car service responsive to a response to the action data structure or to the digital component, based on data center utilization rates or bandwidth metrics or requirements of the network <NUM> or of a data center that includes the data processing system <NUM>.

Based on a response to a digital component or to the action data structure for a subsequent action, such as a click on the digital component rendered via the selected interface, the data processing system <NUM> can identify a conversion, or initiate a conversion or action. Processors of the data processing system <NUM> can invoke the direct action API <NUM> to execute scripts that facilitate the conversion action, such as to order a car from a car share service to take the end user to or from the beach. The direct action API <NUM> can obtain content data <NUM> (or parameters <NUM> or policies <NUM>) from the data repository <NUM>, as well as data received with end user consent from the client computing device <NUM> to determine location, time, user accounts, logistical or other information in order to reserve a car from the car share service. Using the direct action API <NUM>, the data processing system <NUM> can also communicate with the service provider computing device <NUM> to complete the conversion by in this example making the car share pick up reservation.

<FIG> illustrates a block diagram of the selection of a digital component in the system illustrated in <FIG>. The system <NUM> includes a content provider device <NUM> and a service provider device <NUM>. The client computing device <NUM>, the content provider device <NUM>, and the service provider device <NUM> can each include one or more instances of the digital component <NUM>. Each of the components of the system <NUM> identified by the data processing system <NUM> as storing an instance of the digital component <NUM> can be referred to as candidate content sources. The data processing system <NUM>, the client computing device <NUM>, the content provider device <NUM>, and the service provider device <NUM> are each connected to via connections <NUM>. The connections <NUM> can be made through the network <NUM>. The client computing device <NUM> can have a connection <NUM> to the data processing system <NUM>, content provider device <NUM>, the service provider device <NUM>, or any combination thereof. In some implementations, the client computing device <NUM> can only have a connection with the data processing system <NUM>, via, for example, the network <NUM>.

As described further in relation to <FIG>, among others, the client computing device <NUM> can transmit an input audio signal to the data processing system <NUM>. The data processing system <NUM> can execute the NLP component <NUM>, which can identify a request and one or more trigger keywords in the input audio signal. The data processing system <NUM> can select a digital component <NUM> based on the request and one or more trigger keywords. For example, the request can be to play a specific song on the client computing device <NUM>, which may be a speaker-based assistant device. Instances of the digital component <NUM>, identified by the data processing system <NUM>, can be stored on the client computing device <NUM>, the content provider device <NUM>, the service provider device <NUM>, or any combination thereof. The data processing system <NUM> can determine which instance of the digital component <NUM> should be provided to the client computing device <NUM> by calculating a characteristic of each of the candidate content sources, the connections <NUM> thereto, and the instances of the digital component. The characteristic for a given candidate content source or instance of the digital component can be based on characteristics of the candidate content source itself and the digital component <NUM> stored therein. For example, the characteristics can include the quality of the instance of the digital component <NUM>, the availability of the digital component <NUM>, or the utilization of the content provider device <NUM>. The characteristic can also be based on the utilization, status, or other characteristics of the client computing device <NUM>. The utilization or characteristics can include the type of connection to the network <NUM>, the battery status, power status, processor capabilities, display capabilities, or memory status of the client computing device <NUM>.

As illustrated for example by the content provider device <NUM> in <FIG>, a candidate content source can include multiple instances of the digital component <NUM>. When a candidate content source contains more than one instance of the digital component <NUM>, the source allocation component <NUM> can calculate or otherwise determine a characteristic to be associated with each of the instances of the digital component <NUM> on the candidate content source and each of the respective candidate content sources. The different instances of the digital component <NUM> can have different characteristics or can be duplicates of the same digital component <NUM>. The characteristics can include file quality, file formats, encoding types, file size, or permission requirements. For example, if the digital component <NUM> is a video file, a first instance of the digital component <NUM> on a candidate content source can be a high-definition version of the video file and a second instance of the digital component <NUM> can be a standard-definition version of the video file.

Each of the multiple instances of a digital component <NUM> on a given candidate content source can correspond to a different user or account. For example, the content provider device <NUM> can be an email provider. The first instance of the digital component <NUM> can correspond to the email account of a first user and the second instance of the digital component <NUM> can correspond to the email account of a second user. For example, a first user of the client computing device <NUM> can provide the input audio signal, "Ok, read me my last email. " The client computing device <NUM> or data processing system <NUM> can determine which user of the client computing device <NUM> provided the input audio signal. The data processing system <NUM> can then assign the digital component <NUM> (e.g., email account) on the candidate content source associated a second user of the client computing device <NUM> with a low or null characteristic such that that email account is not selected.

The client computing device <NUM> can already include an instance of the digital component <NUM>, as illustrated in <FIG>. The data processing system <NUM> can determine the client computing device <NUM> includes an instance of the digital component <NUM> and the data processing system <NUM> can label the client computing device <NUM> as a candidate content source. The data processing system <NUM> can still determine that the content provider device <NUM> and the service provider device <NUM> are candidate content sources and can assign at least one characteristic to the client computing device <NUM>, the content provider device <NUM>, and the service provider device <NUM>. In some implementations, the data processing system <NUM> can determine the client computing device <NUM> has a relatively high characteristic because substantially no network resources would be used in rendering or providing the digital component <NUM> on the client computing device <NUM>. However, one of the candidate content sources could have a high-quality version of the digital component <NUM> (or other characteristic) that causes the data processing system <NUM> to assign the remote candidate content source a relatively higher characteristic when compared to the client computing device <NUM>. In this example, the data processing system <NUM> could transmit (or have the candidate content source transmit) the digital component <NUM> to the client computing device <NUM> even through the client computing device <NUM> already possessed an instance of the digital component <NUM>.

As the source allocation component <NUM> selects content sources from the candidate content sources, the source allocation component <NUM> can generate a dynamic index of the content stored on each of the candidate content sources or a dynamic index of which candidate content source is selected. The index can be valid for a predetermined amount of time. If the source allocation component <NUM> determines the index is still valid when a subsequent request for a digital component is received, the source allocation component <NUM> can use the index to select a content source without first finding a plurality of candidate content sources and calculating characteristics for each of the candidate content sources.

<FIG> illustrates a block diagram of an example method <NUM> to allocate content sources in a voice activated packet-based computer network environment. The method <NUM> can include receiving data packets (ACT <NUM>). The method <NUM> can include identifying requests and trigger keywords (ACT <NUM>). The method <NUM> can include selecting a digital component (ACT <NUM>). The method <NUM> can include identifying candidate content sources (ACT <NUM>). The method <NUM> can include determining characteristics (ACT <NUM>). The method <NUM> can include selecting a content source (ACT <NUM>).

As set forth above, the method <NUM> can include receiving data packets (ACT <NUM>). The data packets can include an input audio signal. The input audio signal can be detected by a sensor of a client computing device and transmitted by the client computing device to the data processing system <NUM>. The NLP component <NUM>, executed by the data processing system <NUM>, can receive from the client computing device <NUM> the data packets. The data packets can be received via the network <NUM> as packet or other protocol based data transmissions.

The method <NUM> can include identifying requests and trigger keywords (ACT <NUM>). The NLP component <NUM> can identify the request and the trigger keywords in the input audio signal that the data processing system <NUM> received as data packets. For example, the NLP component <NUM> can parse the input audio signal to identify requests that relate to subject matter of the input audio signal. The NLP component <NUM> can parse the input audio signal to identify trigger keywords that can indicate, for example, actions associated with the requests.

The method <NUM> can include selecting at least one digital component (ACT <NUM>). The content selector component <NUM> can receive the requests or the trigger keywords and based on this information can select one or more digital components. For example, the input audio signal can be "Ok, play Neil Young" and the NLP component <NUM> can determine the request is for a music song performed by Neil Young. In this example, the content selector component <NUM> can select a digital component that includes a song performed by Neil Young, a Neil Young streaming radio station, or a song in a style similar to that of Neil Young. The data processing system <NUM> can also generate at least one action data structure.

The method <NUM> can include identifying content sources (ACT <NUM>). The source allocation component <NUM> can identify a plurality of candidate content sources subsequent to identifying the digital component. Each of the plurality of candidate content sources can include (e.g., store an instance of) the digital component or a related digital component. Continuing the above example, each of the candidate content sources can be a different online music streaming service that include the Neil Young song selected during ACT <NUM>.

The method <NUM> can include determining one or more characteristics (ACT <NUM>). The source allocation component <NUM> can determine a characteristic for each of the plurality of candidate content sources and a characteristic for each instance of the digital component they contain. The characteristic can be based on a connection between the respective candidate content source and the client computing device. For example, the source allocation component <NUM> can consider the bandwidth of the connection and the quality of the connection. The source allocation component <NUM> can also incorporate the type and quality of the connection the client computing device has with the network. For example, whether the client computing device is connected to the network via WiFi or via a cellular connection. In some implementations, the characteristic for each of the respective candidate content source can be based on the file quality of the instance of the digital component the respective candidate content source has. For example, the quality of the instance can include the bit rate, file size, encoding codec, and file format.

The characteristic can be based on a purchase status or availability of the digital component. For example, the user may have purchased an instance of the digital component from a first candidate content source. The first candidate content source can be given a higher characteristic as compared to a second candidate content source from which the user has not purchased the digital component. The characteristic can be based on the location of the candidate content source and its respective instance of the digital component. The location can be the physical location of the digital component. For example, an instance of the digital component stored local to the client computing device can be given a higher quality score compared to a remote instance of the digital component. Similarly, a candidate content source storing an instance of the digital component within the same geographical region as the client computing device (e.g., the same state or country) can be given a higher characteristic as compared to a candidate content source storing its instance of the digital component in a different geographical region.

The characteristic for each of the respective candidate content sources can be based on the utilization (or status) of the client computing devices. The utilization of the client computing device can be based on the processor capability, memory capability or availability, battery status, available power, network bandwidth utilization, or interface parameters. By basing the characteristic on the client computing device's utilization, the data processing system <NUM> can reduce network bandwidth usage, latency, or processing utilization or power consumption of the client computing device that renders the digital component. This saves processing power and other computing resources such as memory, reduces electrical power consumption by the data processing system and the reduced data transmissions via the computer network reduces bandwidth requirements and usage of the data processing system. For example, the candidate content sources containing instances of the digital component that the client computing device cannot properly render can be given a relatively lower characteristic. In some implementations, the characteristic is based only on the utilization or status of the client computing device.

The source allocation component <NUM> can select a first plurality of candidate content sources. The source allocation component <NUM> can determine a characteristic for each of the first plurality of candidate content sources. The source allocation component <NUM> can compare the characteristics to a threshold. If the characteristics of the first plurality of candidate content sources do not cross the threshold, the source allocation component <NUM> can select a second plurality of candidate content sources. The source allocation component <NUM> can calculate and compare the characteristics of the second plurality of candidate content sources to the threshold. The source allocation component <NUM> can repeat this process until one or more candidate content sources are found to have a characteristic above the threshold. Each of the plurality of candidate content sources can include a single candidate content source. For example, the source allocation component <NUM> can start with the candidate content source that would consume the least amount of computational and network resources in delivering the digital component to the client computing device (but may have a relatively lower characteristic). If the first candidate content source is determined to have a characteristic above the quality threshold, the source allocation component <NUM> can select the first candidate content source as the content source to provide the digital component to the client computing device. If the first candidate content source's characteristic is below the quality threshold, the source allocation component <NUM> can continue to select and test candidate content sources until the source allocation component <NUM> locates a candidate content source with a characteristic above the quality threshold.

The method <NUM> can include selecting a content source from the plurality of candidate content sources (ACT <NUM>). The source allocation component <NUM> can select a content source from the plurality of candidate content sources based on a ranking of the characteristics for each of the plurality of candidate content sources. The ranking can identify which of the candidate content source has the highest quality instance of the digital component, the best connection to the data processing system <NUM> or the client computing device, or would consume the least amount of network or computational resources in transmitting the digital component to the client computing device.

The data processing system <NUM> can request a copy of the digital component from the selected candidate content source and then provide the digital component to the client computing device. In other implementations, the data processing system <NUM> can initiate a session between the selected candidate content source and the client computing device. The selected candidate content source can stream, transmit, or otherwise provide the digital component to the client computing device without first providing the digital component to the data processing system <NUM>. In some implementations, the data processing system <NUM> can store a reference to the selected candidate content source in association with a reference to the client computing device for a predetermined length of time. If the client computing device requests a subsequent digital component within the predetermined length of time, the data processing system <NUM> can request the digital component from the previously selected candidate content source without first determining and ranking a plurality of candidate content sources. In other implementations, the data processing system <NUM> can perform the method <NUM> each time that the client computing device requests a digital component.

Each aspect of the disclosure may include one or more of the following features. The source allocation component can: receive the digital component from the content source; and transmit the digital component to the client computing device. The source allocation component can: detect a second instance of the digital component on one of the plurality of candidate content sources; and determine a second characteristic for the second instance of the digital component on one of the plurality of content sources. The source allocation component may: determine a utilization rate of the client computing device; and select the content source based on the utilization rate of the client computing device. The utilization rate may be based on at least one of a battery level of the client computing device, a storage availability of the client computing device, or a processor availability of the client computing device. The content selector component may: select the content source from the plurality of candidate content sources based on the characteristic of the content source being higher than the characteristic of the respective instances of the digital component on the content source. The content selector component may: select the content source from the plurality of candidate content sources based on the characteristic of the content source being lower than the characteristic of the respective instances of the digital component on the content source. The request may be a request to initiate a session between the client computing device and the content source. The content selector component may select a second digital component. The source allocation component may: identify a second plurality of candidate content sources, each of the second plurality of candidate content sources comprising a respective instance of the second digital component; determine a characteristic for each of the second plurality of candidate content sources; determine the characteristic for each of the second plurality of candidate content sources is below a predetermined threshold; and identify a third plurality of candidate content sources, each of the third plurality of candidate content sources comprising a respective instance of the second digital component. The content selector component may select a second digital component; and request the second digital component from the content source based on determining the content source is valid. The natural language processor component may determine an account associated with the input audio signal; the source allocation component to determine the characteristic for each of the respective instances of the digital component. The characteristic for a respective one of the plurality of candidate content sources may be based on at least one of a connection quality or a bandwidth availability between the data processing system and the respective one of the plurality of candidate content sources. The characteristic for a respective one of the plurality of candidate content sources may be based on a connection quality or a bandwidth availability between the client computing device and the respective one of the plurality of candidate content sources. The characteristic for a respective one of the plurality of candidate content sources may be based on a quality characteristic or a storage location of the respective instance of the digital component stored at the respective one of the plurality of candidate content sources. Transmitting, by the source allocation component, the instance of the digital component to the client computing device. Detecting, by the source allocation component, a second instance of the digital component on one of the plurality of candidate content sources; and determining, by the source allocation component, a second characteristic for the second instance of the digital component on one of the plurality of content sources. Selecting, by the content selector component, a second digital component; identifying, by the source allocation component, a second plurality of candidate content sources, each of the second plurality of candidate content sources comprising a respective instance of the second digital component; determining, by the source allocation component, a characteristic for each of the second plurality of candidate content sources; determining, by the source allocation component, the characteristic for each of the second plurality of candidate content sources is below a predetermined threshold; and identifying, by the source allocation component, a third plurality of candidate content sources, each of the third plurality of candidate content sources comprising a respective instance of the second digital component. Determining, by the natural language processor component, an account associated with the input audio signal; determining, by the source allocation component, the characteristic for each of the respective instances of the digital component.

The computer system or computing system <NUM> can include or be used to implement the system <NUM>, or its components such as the data processing system <NUM>. The computing system <NUM> includes a bus <NUM> or other communication component for communicating information and a processor <NUM> or processing circuit coupled to the bus <NUM> for processing information. The computing system <NUM> can also include one or more processors <NUM> or processing circuits coupled to the bus for processing information. The computing system <NUM> also includes main memory <NUM>, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus <NUM> for storing information, and instructions to be executed by the processor <NUM>. The main memory <NUM> can be or include the data repository <NUM>. The main memory <NUM> can also be used for storing position information, temporary variables, or other intermediate information during execution of instructions by the processor <NUM>. The computing system <NUM> may further include a read only memory (ROM) <NUM> or other static storage device coupled to the bus <NUM> for storing static information and instructions for the processor <NUM>. A storage device <NUM>, such as a solid state device, magnetic disk or optical disk, can be coupled to the bus <NUM> to persistently store information and instructions. The storage device <NUM> can include or be part of the data repository <NUM>.

The computing system <NUM> may be coupled via the bus <NUM> to a display <NUM>, such as a liquid crystal display, or active matrix display, for displaying information to a user. An input device <NUM>, such as a keyboard including alphanumeric and other keys, may be coupled to the bus <NUM> for communicating information and command selections to the processor <NUM>. The input device <NUM> can include a touch screen display <NUM>. The input device <NUM> can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor <NUM> and for controlling cursor movement on the display <NUM>. The display <NUM> can be part of the data processing system <NUM>, the client computing device <NUM> or other component of <FIG>, for example.

For situations in which the systems discussed herein collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features that may collect personal information (e.g., information about a user's social network, social actions or activities, a user's preferences, or a user's location), or to control whether or how to receive content from a content server or other data processing system that may be more relevant to the user. In addition, certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed when generating parameters. For example, a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, postal code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about him or her and used by the content server.

The subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatuses. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. While a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices).

The terms "data processing system" "computing device" "component" or "data processing apparatus" encompass various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The interface management component, direct action API <NUM>, content selector component <NUM>, prediction component <NUM> or NLP component <NUM> and other data processing system <NUM> components can include or share one or more data processing apparatuses, systems, computing devices, or processors.

A computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs (e.g., components of the data processing system <NUM>) to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

The subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back-end, middleware, or front-end components.

The computing system such as system <NUM> or system <NUM> can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network (e.g., the network <NUM>). In some implementations, a server transmits data (e.g., data packets representing action data structures or digital components) to a client computing device (e.g., to the client computing device <NUM> for purposes of displaying data to and receiving user input from a user interacting with the client computing device, or to the service provider computing device <NUM> or the content provider computing device <NUM>). Data generated at the client computing device (e.g., a result of the user interaction) can be received from the client computing device at the server (e.g., received by the data processing system <NUM> from the computing device <NUM> or the content provider computing device <NUM> or the service provider computing device <NUM>).

The separation of various system components does not require separation in all implementations, and the described program components can be included in a single hardware or software product. For example, the NLP component <NUM>, the content selector component <NUM>, the interface management component, or the prediction component <NUM> can be a single component, app, or program, or a logic device having one or more processing circuits, or part of one or more servers of the data processing system <NUM>.

Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.

Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to "an implementation," "some implementations," "one implementation" or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation or embodiment. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

References to "or" may be construed as inclusive so that any terms described using "or" may indicate any of a single, more than one, and all of the described terms. For example, a reference to "at least one of 'A' and 'B'" can include only 'A', only 'B', as well as both 'A' and 'B'. Such references used in conjunction with "comprising" or other open terminology can include additional items.

Claim 1:
A system (<NUM>) to allocate content sources in a voice activated packet-based computer network environment, comprising:
a natural language processor component (<NUM>) executed by a data processing system (<NUM>), to parse an input audio signal to identify a request and a trigger keyword corresponding to the request;
a content selector component (<NUM>) to select a digital component (<NUM>) based on at least one of the request and the trigger keyword;
a source allocation component (<NUM>) to identify, subsequent to the selection of the digital component, a plurality of candidate content sources, each of the plurality of candidate content sources comprising a respective instance of the digital component;
the source allocation component to determine at least one of a characteristic for each of the plurality of candidate content sources or a characteristic of each of the respective instances of the digital component;
the source allocation component to select a content source (<NUM>) from the plurality of candidate content sources based on at least one of the characteristic for each of the plurality of candidate content sources or the characteristic of each of the respective instances of the digital component; and
the source allocation component to request the digital component from the content source.