Patent Description:
Computing devices can provide digital content to networked client devices. Excessive network transmissions may be required to find specific location or content within the provided digital content. Additional, voice-based interfaces may not be capable of moving to specific locations within the digital content, which can result in wasted network resources as the entity of the digital content is transmitted to the network client device.

<CIT> states in its abstract: methods and apparatus related to identifying a video for completing a task and determining a plurality of video segments of the identified video based on one or more attributes of the task. A task and a plurality of how-to videos related to the task may be identified. A how-to video may be selected and a plurality of video segments of the selected how-to video may be determined. One or more video segments may be associated with one or more task attributes that relate to performing the task. The selected video may be provided to a user and segmented, indexed, and/or annotated based on the associated video segments. In some implementations a given object utilized in performing the task may be identified and one or more video segments corresponding to the given object may be identified and/or provided to the user.

<CIT> states in its abstract: a new way of providing search results that include audio/video thumbnails for searches of audio and video content is disclosed. An audio/video thumbnail includes one or more audio/video segments retrieved from within the content of audio/video files selected as relevant to a search or other user input. For an audio/video thumbnail of more than one segment, the audio/video segments from an individual audio/video file responsive to the search are concatenated into a multi-segment audio/video thumbnail. The audio/video segments provide enough information to be indicative of the nature of the audio/video file from which each of the audio/video thumbnails is retrieved, while also fast enough that a user can scan through a series of audio/video thumbnails relatively quickly. A user can then watch or listen to the series of audio/video thumbnails, which provide a powerful indication of the full content of the search results, and make searching for audio/video content easier and more effective, across a broad range of computing devices.

<CIT> states in its abstract: systems and methods for voice searching media content based on metadata or subtitles are provided. Metadata associated with media content can be pre-processed at a media server. Upon receiving a vocal command representative of a search for an aspect of the media content, the media server performs a search for one or more portions of the media content relevant to the aspect of the media content being searched for. The media performs the search by matching the aspect of the media content being searched for with the pre-processed metadata.

<CIT> states in its abstract: disclosed are systems, methods, and computer readable media for retrieving digital images. The method embodiment includes converting a descriptive audio stream of a digital video that is provided for the visually impaired to text and then aligning that text to the appropriate segment of the digital video. The system then indexes the converted text from the descriptive audio stream with the text's relationship to the digital video. The system enables queries using action words describing a desired scene from a digital video.

The invention to which this European patent relates provides a system according to claim <NUM>.

Further preferable aspects of the system are defined by the claims dependent on claim <NUM>.

The invention to which this European patent relates also provides a method according to claim <NUM>.

Further preferable aspects of the method are defined by the claims dependent on claim <NUM>.

The foregoing information and the following detailed description include illustrative examples of various aspects and implementations and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations and are incorporated in and constitute a part of this specification. Aspects and embodiments of the disclosed subject matter may be combined within the scope defined by the claims.

Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems of transferring data in a secure processing environment. The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways.

The present disclosure is generally directed towards controlling digital components in a voice-activated system. Interfaces for presenting video can include inputs that mimic electro-mechanical playback devices (e.g., VCRs), and include buttons such as stop, play, fast forward, and rewind. Interacting with these limited inputs with a voice-activated system can be difficult. Additionally, the difficulty in interacting with the video content make it difficult for users to select and watch only portions of video content. Difficulty in finding and watching only the desired portions of video content can result in computational and network waste as the end user may often watch or download the whole video content rather than only the needed portions.

Systems and methods of the present technical solution enable a multi-modal interface for voice-based devices, such as digital assistants. The solution can enable a user to interact with video and other content through a touch interface and through voice commands. In addition to inputs such as stop and play, the present solution can also automatically generate annotations for displayed video files. From the annotations, the solution can identify one or more break points that are associated with different scenes, video portions, or how-to steps in the video. The digital assistant can receive input audio signal and parse the input audio signal to identify semantic entities within the input audio signal. The digital assistant can map the identified semantic entities to the annotations to select a portion of the video that corresponds to the users request in the input audio signal. The digital assistant can then jump to the selected portion of the video. Enabling the user of a voice-based digital assistant to search for specific content within a video can reduce computational and network resources by enabling users to skip to the requested portion of the video, enabling only portions of the video to be transmitted over the network rather than the whole video. The present solution also provides the user with a new and improved user interface for interacting with video on voice-based devices, improving the human-machine interaction process.

<FIG> illustrates an example system <NUM> to control digital components in a voice-activated system. The system <NUM> includes a digital component selection infrastructure. The system <NUM> can include a data processing system <NUM>. The data processing system <NUM> can communicate with one or more of a digital component provider device <NUM> (e.g., content provider device) or client computing devices <NUM> via a network <NUM>. The network <NUM> can include computer networks such as the Internet, local, wide, metro, or other area networks, intranets, satellite networks, and other communication networks such as voice or data mobile telephone networks. The network <NUM> can be used to access information resources such as web pages, web sites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed on at least one computing device <NUM>, such as a laptop, desktop, tablet, digital assistant, personal digital assistant, smartwatch, wearable device, smart phone, portable computers, or speaker. For example, via the network <NUM> a user of the client computing device <NUM> can access information or data provided by a digital component provider device <NUM>. The client computing device <NUM> may or may not include a display. For example, the client computing device <NUM> may include limited types of user interfaces, such as a microphone and speaker (e.g., the client computing device <NUM> can include a voice-drive or audio-based interface). The primary user interface of the computing device <NUM> may be a microphone and speaker.

The network <NUM> can include or constitute a display network, e.g., a subset of information resources available on the internet that are associated with a content placement or search engine results system, or that are eligible to include third party digital components. The network <NUM> can be used by the data processing system <NUM> to access information resources such as web pages, web sites, domain names, or uniform resource locators that can be presented, output, rendered, or displayed by the client computing device <NUM>. For example, via the network <NUM> a user of the client computing device <NUM> can access information or data provided by the digital component provider device <NUM>.

The network <NUM> may be any type or form of network and may include any of the following: a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, a SDH (Synchronous Digital Hierarchy) network, a wireless network and a wireline network. The network <NUM> may include a wireless link, such as an infrared channel or satellite band. The topology of the network <NUM> may include a bus, star, or ring network topology. The network may include mobile telephone networks using any protocol or protocols used to communicate among mobile devices, including advanced mobile phone protocol ("AMPS"), time division multiple access ("TDMA"), code-division multiple access ("CDMA"), global system for mobile communication ("GSM"), general packet radio services ("GPRS"), or universal mobile telecommunications system ("UMTS"). Different types of data may be transmitted via different protocols, or the same types of data may be transmitted via different protocols.

The system <NUM> can include at least one data processing system <NUM>. The data processing system <NUM> can include at least one logic device such as a computing device having a processor to communicate via the network <NUM>, for example, with the computing device <NUM> or the digital component provider device <NUM>. The data processing system <NUM> includes at least one computation resource, server, processor or memory. For example, the data processing system <NUM> can include a plurality of computation resources or servers located in at least one data center. The data processing system <NUM> can include multiple, logically-grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm or a machine farm. The servers can also be geographically dispersed. A data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous-one or more of the servers or machines can operate according to one or more type of operating system platform.

Servers in the machine farm can be stored in high-density rack systems, along with associated storage systems, and located in an enterprise data center. For example, consolidating the servers in this way may improve system manageability, data security, the physical security of the system, and system performance by locating servers and high performance storage systems on localized high performance networks. Centralization of all or some of the data processing system <NUM> components, including servers and storage systems, and coupling them with advanced system management tools allows more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage.

The client computing device <NUM> can include, execute, interface, or otherwise communicate with one or more of at least one local digital assistant <NUM>, at least one sensor <NUM>, at least one transducer <NUM>, at least one audio driver <NUM>, or at least one display <NUM>. The sensor <NUM> can include, for example, a camera, an ambient light sensor, proximity sensor, temperature sensor, accelerometer, gyroscope, motion detector, GPS sensor, location sensor, microphone, video, image detection, or touch sensor. The transducer <NUM> can include or be part of a speaker or a microphone. The audio driver <NUM> can provide a software interface to the hardware transducer <NUM>. The audio driver <NUM> can execute the audio file or other instructions provided by the data processing system <NUM> to control the transducer <NUM> to generate a corresponding acoustic wave or sound wave. The display <NUM> can include one or more hardware or software component configured to provide a visual indication or optical output, such as a light emitting diode, organic light emitting diode, liquid crystal display, laser, or display.

The local digital assistant <NUM> can include or be executed by one or more processors, logic array, or memory. The local digital assistant <NUM> can detect a keyword and perform an action based on the keyword. The local digital assistance <NUM> can be an instance of the remote digital assistance component <NUM> executed at the data processing system <NUM> or can perform any of the functions of the remote digital assistance component <NUM>. The local digital assistant <NUM> can filter out one or more terms or modify the terms prior to transmitting the terms as data to the data processing system <NUM> (e.g., remote digital assistant component <NUM>) for further processing. The local digital assistant <NUM> can convert the analog audio signals detected by the transducer <NUM> into a digital audio signal and transmit one or more data packets carrying the digital audio signal to the data processing system <NUM> via the network <NUM>. The local digital assistant <NUM> can transmit data packets carrying some or all of the input audio signal responsive to detecting an instruction to perform such transmission. The instruction can include, for example, a trigger keyword or other keyword or approval to transmit data packets comprising the input audio signal to the data processing system <NUM>.

The local digital assistant <NUM> can perform a pre-filtering or pre-processing on the input audio signal to remove certain frequencies of audio. The pre-filtering can include filters such as a low-pass filter, high-pass filter or a bandpass filter. The filters can be applied in the frequency domain. The filters can be applied using digital signal processing techniques. The filter can be configured to keep frequencies that correspond to a human voice or human speech while eliminating frequencies that fall outside the typical frequencies of human speech. For example, a bandpass filter can be configured to remove frequencies below a first threshold (e.g., <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>) and above a second threshold (e.g., <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>). Applying a bandpass filter can reduce computing resource utilization in downstream processing. The local digital assistant <NUM> on the computing device <NUM> can apply the bandpass filter prior to transmitting the input audio signal to the data processing system <NUM>, thereby reducing network bandwidth utilization. However, based on the computing resources available to the computing device <NUM> and the available network bandwidth, it may be more efficient to provide the input audio signal to the data processing system <NUM> to allow the data processing system <NUM> to perform the filtering.

The local digital assistant <NUM> can apply additional pre-processing or pre-filtering techniques such as noise reduction techniques to reduce ambient noise levels that can interfere with natural language processor. Noise reduction techniques can improve accuracy and speed of the natural language processor, thereby improving the performance of the data processing system <NUM> and manage rendering of a graphical user interface provided via the display <NUM>.

The client computing device <NUM> can be associated with an end user that enters voice queries as audio input into the client computing device <NUM> (via the sensor <NUM> or transducer <NUM>) and receives audio (or other) output from the data processing system <NUM> or digital component provider device <NUM> to present, display, or render to the end user of the client computing device <NUM>. The digital component can include a computer-generated voice that can be provided from the data processing system <NUM> or digital component provider device <NUM> to the client computing device <NUM>. The client computing device <NUM> can render the computer-generated voice to the end user via the transducer <NUM> (e.g., a speaker). The computer-generated voice can include recordings from a real person or computer-generated language. The client computing device <NUM> can provide visual output via a display device <NUM> communicatively coupled to the computing device <NUM>.

The end user that enters the voice queries to the client computing device <NUM> can be associated with multiple client computing devices <NUM>. For example, the end user can be associated with a first client computing device <NUM> that can be a speaker-based digital assistant device, a second client computing device <NUM> that can be a mobile device (e.g., a smartphone), and a third client computing device <NUM> that can be a desktop computer. The data processing system <NUM> can associate each of the client computing devices <NUM> through a common login, location, network, or other linking data. For example, the end user may log into each of the client computing devices <NUM> with the same account user name and password.

The client computing device <NUM> receives an input audio signal detected by a sensor <NUM> (e.g., microphone) of the computing device <NUM>. The input audio signal can include, for example, a query, question, command, instructions, or other statement provided in a language. The input audio signal can include an identifier or name of a third-party (e.g., a digital component provider device <NUM>) to which the question or request is directed.

The client computing device <NUM> can include, execute, or be referred to as a digital assistant device. The digital assistant device can include one or more components of the computing device <NUM>. The digital assistant device can include a graphics driver that can receive display output from the data processing system <NUM> and render the display output on display <NUM>. The graphics driver can include hardware or software components that control or enhance how graphics or visual output is displayed on the display <NUM>. The graphics driver can include, for example, a program that controls how the graphic components work with the rest of the computing device <NUM> (or digital assistant). The local digital assistant <NUM> can filter the input audio signal to create a filtered input audio signal, convert the filtered input audio signal to data packets, and transmit the data packets to a data processing system comprising one or more processors and memory.

The digital assistant device can include an audio driver <NUM> and a speaker component (e.g., transducer <NUM>). The pre-processor component <NUM> receives an indication of the display output and instructs the audio driver <NUM> to generate an output audio signal to cause the speaker component (e.g., transducer <NUM>) to transmit an audio output corresponding to the indication of the display output.

The system <NUM> can include, access, or otherwise interact with at least digital component provider device <NUM>. The digital component provider device <NUM> can include one or more servers that can provide digital components to the client computing device <NUM> or data processing system <NUM>. The digital component provider device <NUM> or components thereof can be integrated with the data processing system <NUM> or executed at least partially by the data processing system <NUM>. The digital component provider device <NUM> can include at least one logic device such as a computing device having a processor to communicate via the network <NUM>, for example with the computing device <NUM>, the data processing system <NUM>, or the digital component provider device <NUM>. The digital component provider device <NUM> can include at least one computation resource, server, processor, or memory. For example, the digital component provider device <NUM> can include a plurality of computation resources or servers located in at least one data center.

A digital component provider device <NUM> can provide audio, visual, or multimediabased digital components for presentation by the client computing device <NUM> as an audio output digital component, visual output digital components, or a mix thereof. The digital component can be or include a digital content. The digital component can be or include a digital object. The digital component can include subscription-based content or pay-for content. A digital component can include a plurality of digital content items. For example, a digital component can be a data stream from a streaming music service (e.g., the digital component provider device <NUM>). The digital components can include or can be digital movies, websites, songs, applications (e.g., smartphone or other client device applications), or other text-based, audio-based, image-based, or video-based content. For example, the digital components can be how-to videos, movies, or other video provided by the digital content provider device <NUM> to the client computing device <NUM>. The digital content provider device <NUM> can provide digital components generated by the digital content provider device <NUM>, uploaded by users, or sources from other digital content provider devices <NUM>.

The digital component provider device <NUM> can provide the digital components to the client computing device <NUM> via the network <NUM> and bypass the data processing system <NUM>. The digital component provider device <NUM> can provide the digital component to the client computing device <NUM> via the network <NUM> and data processing system <NUM>. For example, the digital component provider device <NUM> can provide the digital components to the data processing system <NUM>, which can store the digital components and provide the digital components to the client computing device <NUM> when requested by the client computing device <NUM>.

The data processing system <NUM> can include at least one computation resource or server. The data processing system <NUM> can include, interface, or otherwise communicate with at least one interface <NUM>. The data processing system <NUM> can include, interface, or otherwise communicate with at least one remote digital assistant component <NUM>. The remote digital assistant component <NUM> can include, interface, or otherwise communicate with at least one natural language processor component <NUM>. The data processing system <NUM> can include, interface, or otherwise communicate with at least one digital component selector <NUM>. The data processing system <NUM> can include, interface, or otherwise communicate with at least one annotation component <NUM>. The data processing system <NUM> can include, interface, or otherwise communicate with at least one parsing component <NUM>. The data processing system <NUM> can include, interface, or otherwise communicate with at least one data repository <NUM>. The at least one data repository <NUM> can include or store, in one or more data structures or databases, sets of annotations <NUM>, break points <NUM>, caption data <NUM>, and content data <NUM>. The data repository <NUM> can include one or more local or distributed databases, and can include a database management.

The interface <NUM>, remote digital assistant component <NUM>, the natural language processor component <NUM>, the digital component selector <NUM>, the annotation component <NUM>, and the parsing component <NUM> can each include at least one processing unit or other logic device such as programmable logic array engine, or module configured to communicate with the database repository or database <NUM>. The interface <NUM>, the remote digital assistant component <NUM>, the natural language processor component <NUM>, the digital component selector <NUM>, the annotation component <NUM>, the parsing component <NUM>, and the data repository <NUM> can be separate components, a single component, or part of multiple data processing systems <NUM>. The system <NUM> and its components, such as a data processing system <NUM>, can include hardware elements, such as one or more processors, logic devices, or circuits.

The data processing system <NUM> can include an interface <NUM>. The interface <NUM> can be configured, constructed, or operational to receive and transmit information using, for example, data packets. The interface <NUM> can receive and transmit information using one or more protocols, such as a network protocol. The interface <NUM> can include a hardware interface, software interface, wired interface, or wireless interface. The interface <NUM> can facilitate translating or formatting data from one format to another format. For example, the interface <NUM> can include an application programming interface that includes definitions for communicating between various components, such as software components.

The data processing system <NUM> can include an application, script, or program installed at the client computing device <NUM>, such as a local digital assistant <NUM> to communicate input audio signals to the interface <NUM> of the data processing system <NUM> and to drive components of the client computing device to render output audio signals or visual output. The data processing system <NUM> can receive data packets, a digital file, or other signals that include or identify an input audio signal (or input audio signals). The computing device <NUM> can detect the audio signal via the transducer <NUM> and convert the analog audio signal to a digital file via an analog-to-digital converter. For example, the audio driver <NUM> can include an analog-to-digital converter component. The pre-processor component <NUM> can convert the audio signals to a digital file that can be transmitted via data packets over network <NUM>.

The remote digital assistant component <NUM> of the data processing system <NUM> executes or runs an NLP component <NUM> to receive or obtain the data packets including the input audio signal detected by the sensor <NUM> of the computing device <NUM>. The client computing device <NUM> can also execute an instance of NLP component <NUM> to process language and text at the client computing device <NUM>. The data packets can provide a digital file. The NLP component <NUM> can receive or obtain the digital file or data packets comprising the audio signal and parse the audio signal. For example, the NLP component <NUM> can provide for interactions between a human and a computer. The NLP component <NUM> can be configured with techniques for understanding natural language and enabling the data processing system <NUM> to derive meaning from human or natural language input. The NLP component <NUM> can include or be configured with techniques based on machine learning, such as statistical machine learning. The NLP component <NUM> can utilize decision trees, statistical models, or probabilistic models to parse the input audio signal.

The NLP component <NUM> can perform, for example, functions such as named entity recognition (e.g., given a stream of text, determine which items in the text map to proper names, such as people or places, and what the type of each such name is, such as person, location, or organization), natural language generation (e.g., convert information from computer databases or semantic intents into understandable human language), natural language understanding (e.g., convert text into more formal representations such as first-order logic structures that a computer module can manipulate), machine translation (e.g., automatically translate text from one human language to another), morphological segmentation (e.g., separating words into individual morphemes and identify the class of the morphemes, which can be challenging based on the complexity of the morphology or structure of the words of the language being considered), question answering (e.g., determining an answer to a human-language question, which can be specific or open-ended), and semantic processing (e.g., processing that can occur after identifying a word and encoding its meaning in order to relate the identified word to other words with similar meanings). The NLP component <NUM> can identify semantic representations of the identified words. By identifying semantic representations, the data processing system can match words or phrases based on their similar semantic meanings rather than specific word matches. For example, a search of an input text based on semantic representations can return the synonyms to a searched word rather just the occurrences of only the searched word.

The NLP component <NUM> can convert the input audio signal into recognized text by comparing the input signal against a stored, representative set of audio waveforms (e.g., in the data repository <NUM>) and choosing the closest matches. The set of audio waveforms can be stored in data repository <NUM> or other database accessible to the data processing system <NUM>. The representative waveforms are generated across a large set of users, and then may be augmented with speech samples from the user. After the audio signal is converted into recognized text, the NLP component <NUM> matches the text to words that are associated, for example via training across users or through manual specification, with actions that the data processing system <NUM> can serve. The NLP component <NUM> can convert image or video input to text or digital files. For example, the NLP component <NUM> can detect the speech in a video file, convert the speech into text, and then process the text. The NLP component <NUM> can identify or receive closed caption data in the video files and process the closed caption data to recognize the text or perform semantic analysis on the closed caption data. The NLP component <NUM> can store the closed caption data for each of the digital components as caption data <NUM> in the data repository <NUM>. The NLP component <NUM> can convert the NLP component <NUM> can process, analyze, or interpret image or video input to perform actions, generate requests, or select or identify data structures.

The data processing system <NUM> can receive image or video input signals, in addition to, or instead of, input audio signals. The data processing system <NUM> can process the image or video input signals using, for example, image interpretation techniques, computer vision, a machine learning engine, or other techniques to recognize or interpret the image or video to convert the image or video to a digital file. The one or more image interpretation techniques, computer vision techniques, or machine learning techniques can be collectively referred to as imaging techniques. The data processing system <NUM> (e.g., the NLP component <NUM>) can be configured with the imaging techniques, in addition to, or instead of, audio processing techniques.

The NLP component <NUM> can obtain the input audio signal. From the input audio signal, the NLP component <NUM> can identify at least one request or at least one trigger keyword corresponding to the request. The request can indicate intent, digital components, or subject matter of the input audio signal. The trigger keyword can indicate a type of action likely to be taken. For example, the NLP component <NUM> can parse the input audio signal to identify at least one request to skip to a specific part of a video file. The trigger keyword can include at least one word, phrase, root or partial word, or derivative indicating an action to be taken. For example, the trigger keyword "go," "go to," or "skip" to indicate the end user wants to view a specific portion of the video file.

The NLP component <NUM> parses the input audio signal to identify, determine, retrieve, or otherwise obtain a request for digital components. The digital components can be video-based files, such as streaming movies, shows, or other video files. For instance, the NLP component <NUM> can apply a semantic processing technique to the input audio signal to identify the requested digital component. The NLP component <NUM> can apply the semantic processing technique to the input audio signal to identify a trigger phrase that includes one or more trigger keywords, such as a first trigger keyword and a second trigger keyword. For example, the input audio signal can include the sentence "Play a video of fixing a bike. " The NLP component <NUM> can determine that the input audio signal includes a trigger keyword "play. " The NLP component <NUM> can determine that the request is for a digital component (e.g., a video) of a bike being fixed.

The remote digital assistant component <NUM> of the data processing system <NUM> executes or runs an instance of the annotation component <NUM> to generate sets of annotations for digital components. The annotation component <NUM> generates sets of annotations for the digital components that are transmitted to the client computing device <NUM> for presentations. An annotation set can include one or more annotations for the whole digital component (e.g., video file) or one or more annotations for each of the scenes or steps identified in the digital component. The parsing component <NUM> can use the annotation sets to determine the meaning, semantic meaning, or content contained within the digital component or scene of the digital component. The parsing component <NUM> can use the annotation sets to match requests in input audio signals to the scenes or steps identified in a digital component. The annotation component <NUM> can store the annotations as annotation sets <NUM> in the data repository. The annotation sets <NUM> can be stored in a data structure or database that identifies the digital component, break point, scene, video portion, or any combination thereof with which the annotation set <NUM> is associated.

The annotation component <NUM> generates a set of annotations based on speech recognized or text recognized in the digital component. For example, the digital component can be a video and the NLP component <NUM> can extract and process the speech from the video. Based on the speech content of the video, the annotation component can determine the content of the video and flag keywords. For example, in a how-to video the annotation component can flag the names of tools or specific materials. In this example, if the end user provides the input audio signal "what tools do I need to perform this task" when watching the how-to video, using the annotations, the data processing system can present the portion of the how-to video where the video discusses tools to the end user. The annotation component <NUM> can generate a set of annotations based on the closed caption data associated with the digital component. The set of annotation can include a list of the words spoken during the digital component or a portion thereof. The parsing component <NUM> can perform keyword searches to match keywords identified in an input audio signal with the terms spoken during the digital component or portion thereof. The set of annotations can include a semantic meaning or representation of the terms or phrases in the digital component. The set of annotations can indicate a semantic meaning for each scene or portion of the digital component.

The annotation component <NUM> can generate a set of annotations based on images in the digital component. The annotation component <NUM> can extract frames or image from an image-based or video-based digital component. The annotation component <NUM> can perform image recognition on the images. The set of annotations based on an image can include a database of objects identified in the digital component and the time point at which the identified object occurs in the digital component. The annotation component <NUM> can also detect transitions in video-based digital components. The transitions can be, for example, changes in scenes or fades to black. The transitions can denote the change from a first scene to a second scene. The set of annotations can indicate what type of transition was identified in the digital component and the time point at which the transition occurred in the digital component.

The annotation component <NUM> can generate a set of annotations based on input from a second client computing device <NUM> or digital content provider device <NUM>. For example, the digital component can be provided by a digital content provider device <NUM> or by an end user of the second client computing device <NUM>. The provider of the digital component can annotate the digital component and transmit the annotations as a set of annotations to the data processing system <NUM>. The set of annotations from the provider can include time points in the digital component that indicate the beginning of scenes or steps in a video, keywords, or tags assigned to different portions of the digital component, or the location of defined break points. For example, the owner or creator of the digital component (e.g., a video) can set the location of break points in the meta data of the digital component to identify each of the steps discussed in the digital component.

The remote digital assistant component <NUM> of the data processing system <NUM> can execute or run an instance of the parsing component <NUM> to parse the digital component into different portions based on the sets of annotations generated by the annotation component <NUM>. Parsing the digital components can include dividing the digital component into separate digital components. For example, the parsing component <NUM> can divide a video into a plurality of smaller videos. Each of the smaller videos may include a single scene or step included in the original video. Parsing the digital components can include determining break points in the digital component. A portion of the digital component can be defined as between two break points (or the beginning of the file and a first break point for the first portion of the digital component and the end of the file and the last break point for the last portion of the digital component). The parsing component <NUM> can set the break points based on the sets of annotations. For example, the parsing component <NUM> can set the break points at the transition between scenes. The parsing component <NUM> can set a plurality of break points within a single scene. For example, a single scene may cover a single topic in a how-to video. The portion of how-to video containing the scene of the single topic can be defined by two break points. The parsing component <NUM> can also include a plurality of break points within the scene that indicate different steps taken during the scene. The parsing component <NUM> can set the break points using machine learning and or natural language processing to identify locations in the digital components that may correspond to different steps in a video, transitions in a video, or useful phrases. For example, the parsing component <NUM> can identify annotations that may be helpful or identifiable in the digital component, such as a listing of ingredients, a listing of tools, or a specific type of scene (e.g., a car chase scene), and set break points at those locations. In one example not covered by the claims of the present application, the parsing component <NUM> can also set the break points based on viewing history of the digital component. For example, if only a subportion of a video is typically viewed by users, the parsing component <NUM> can identify the subportion as an important or relevant portion of the video and can set a break point near the beginning of the subportion. The parsing component <NUM> can determine or identify each of the break points for a digital component. The parsing component <NUM> can save the break points as break points <NUM> in the data repository <NUM>. The break points <NUM> can be a database that stores the time points of each of the break points in association with an indication of the digital component. In one example not covered by the claims of the present application, the break points can be set at set intervals within the digital component. For example, the parsing component <NUM> can set a break point every <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or <NUM> minutes of the digital component.

The digital component selector <NUM> can select a digital component that includes text, strings, characters, video files, image files, or audio files that can be processed by the client computing device <NUM> and presented to the user via the display <NUM> or the transducer <NUM> (e.g., speaker). The digital component selector <NUM> selects a digital component that is responsive to the request identified by the NLP component <NUM> in the input audio signal. For a given request, the digital component selector <NUM> can select supplemental digital components that can also be provided with a primary digital component. The primary digital component can be a digital component directly selected responsive to a request. For example, the primary digital component can be the how-to video requested by the user. The supplemental digital components can be an additional digital component that provide additional information or are related to the primary digital component.

The digital component selector <NUM> can select which digital component provider device <NUM> should or can fulfill the request and can forward the request to the digital component provider device <NUM>. For example, the data processing system <NUM> can initiate a session between the digital component provider device <NUM> and the client computing device <NUM> to enable the digital component provider device <NUM> to transmit the digital component to the client computing device <NUM>. The digital component selector <NUM> can request one or more digital components from the digital component provider device <NUM>. The digital component provider device <NUM> can provide digital components to the data processing system <NUM>, which can store the digital components in the data repository <NUM>. Responsive to a request for a digital component, the digital component selector <NUM> can retrieve the digital component from the data repository <NUM>. In response to a request for a digital component, the digital component selector <NUM> can select a portion or all of a digital component to provide the client computing device <NUM> in response to the request.

The digital component selector <NUM> can select multiple digital components via a real-time content selection process. The digital component selector <NUM> can score and rank the digital components and provide multiple digital components to the output merger component <NUM> to allow the output merger component <NUM> to select the highest ranking digital component. The digital component selector <NUM> can select one or more additional digital components that are transmitted to a client computing device <NUM> based on an input audio signal (or keywords and requests contained therein). In one example, the input audio signal can include a request to start a streaming how-to video. The digital component selector <NUM> can select additional digital components (e.g., ads). The additional digital components can be transmitted to the client computing device <NUM> as the digital component selector <NUM> streams the how-to video to the client computing device <NUM>. The additional digital components can inform an end user of additional or related digital component provider devices <NUM> that could fulfill the request from the first client computing device <NUM>.

The digital component selector <NUM> provides the selected digital component selected in response to the request identified in the input audio signal to the computing device <NUM> or local digital assistant <NUM> or application executing on the computing device <NUM> for presentation. Thus, the digital component selector <NUM> can receive the content request from the client computing device <NUM>, select, responsive to the content request, a digital component, and transmit, to the client computing device <NUM>, the digital component for presentation. The digital component selector <NUM> can transmit, to the local digital assistant <NUM>, the selected digital component for presentation by the local digital assistant <NUM> itself or a third-party application executed by the client computing device <NUM>. For example, the local digital assistant <NUM> can play or output an audio signal corresponding to the selected digital component.

The data repository <NUM> stores content data <NUM> that can include, for example, digital components provided by a digital component provider device <NUM> or obtained or determined by the data processing system <NUM> to facilitate content selection. The content data <NUM> can include, for example, digital components (or digital component object) that can include, for example, a content item, an online document, audio, images, video, multimedia content, or third-party content. The digital component provider device <NUM> can provide full-length digital components to the data processing system <NUM> to store as content data <NUM>. The digital component provider device <NUM> can provide portions of the digital components to the data processing system <NUM>.

<FIG> illustrates a block diagram of an example representation of digital component <NUM> over time. The digital component <NUM> can be a video-based digital component, such as a how-to video. The data processing system can identify a plurality of portions <NUM>(<NUM>)-<NUM>(<NUM>), which can generally be referred to as portions <NUM>. The data processing system identifies a plurality of break points <NUM>(<NUM>)-<NUM>(<NUM>), which can generally be referred to as break points <NUM>. The data processing system can also define a break point <NUM> at the beginning and at the end of the digital component.

Each of the portions <NUM> can be defined between two break points <NUM>. For example, portion <NUM>(<NUM>) is defined as the time between break point <NUM>(<NUM>) and break point <NUM>(<NUM>). The data processing system can select the time at which each of the break points <NUM> located based on the sets of annotations generated by the annotation component. The break point <NUM>(<NUM>) corresponds to the start of an instructional portion of the digital component <NUM>. The break point <NUM>(<NUM>) may correspond to a first how-to step of the digital component <NUM>.

<FIG> illustrates a block diagram of an example method <NUM> to control digital components in a voice-activated system. The method <NUM> includes receiving an input signal (ACT <NUM>). The method <NUM> includes parsing the input signal (ACT <NUM>). The method <NUM> includes generating a set of annotations (ACT <NUM>). The method <NUM> can include identifying break points (ACT <NUM>). The method <NUM> includes receiving an input signal (ACT <NUM>) and parsing the input signal (ACT <NUM>). The method <NUM> includes selecting a break point (ACT <NUM>). The method <NUM> includes transmitting a portion of a digital component corresponding to the selected break point corresponding to a start portion of a digital component. (ACT <NUM>).

As set forth above, the method <NUM> can include includes receiving an input signal (ACT <NUM>). The method <NUM> includes receiving, by a natural language processor component executed by a data processing system, the input signal. The input signal is an input audio signal that is detected by a sensor at a first client device. The sensor can be a microphone of the first client device. For example, a digital assistant component executed at least partially by a data processing system that includes one or more processors and memory can receive the input audio signal. The input audio signal can include a conversation facilitated by a digital assistant. The conversation can include one or more inputs and outputs. The conversation can be audio-based, text-based, or a combination of audio and text. The input audio signal can include text input, or other types of input that can provide conversational information. The data processing system can receive the audio input for a session corresponding to the conversation. The data processing system can receive the audio input in one or more portions or as a bulk or batch upload (e.g., multiple portions of the conversations uploaded in a single transmission to reduce the number of transmissions).

The method <NUM> includes parsing the input signal (ACT <NUM>). The NLP component of the data processing system parses the input signal to identify a digital component request. The NLP component can identify a trigger keyword in the input signal. For example, the input audio signal can include "OK, show me how to fix my bike. " The NLP component can parse the input signal to determine the request is for a how-to video showing how to fix a bike. The NLP component can determine that the trigger keyword is to play, which can indicate that the end user wants to start streaming and playing the video to the client computing device. The client computing device can also provide the digital component request to the data processing system in a text form. For example, the end user can use a physical or digital keyboard associated with the client computing device to type a request for a specific video or other file.

Also referring to <FIG>, among others, <FIG> illustrates a client computing device <NUM> at a first point in time <NUM> and during a second point in time <NUM> during the acts of the method <NUM>. The client computing devices <NUM> illustrated in <FIG> illustrates an example user interface that is presented to an end user on the display <NUM>. The user interface is a conversational, voice-based interface. For example, inputs from the user are displayed as starting toward the right side of the display <NUM> and inputs (or responses) from the data processing system are displayed starting toward the left side of the display <NUM>. New inputs or responses are added to the bottom of the conversation - near the bottom of the display <NUM>. As the new inputs or responses are added, the older inputs and responses are scrolled toward the top of the display <NUM>. For example, at time <NUM> a portion of the inputs visible at time <NUM> have been scrolled off the visible portion of the display <NUM>.

As illustrated in <FIG>, the client computing device <NUM> can detect an input audio signal that includes the phrase "how do I fix my bike?" The input audio signal can be processed by the NLP component <NUM> to extract the text of the input audio signal. The text <NUM> of the input audio signal can be displayed to the user as confirmation that the client computing device <NUM> (and data processing system <NUM>) understood and correctly processed the input audio signal. The data processing system selects a digital component <NUM> in response to the request identified in the input audio signal. In the example illustrated in <FIG>, the digital component <NUM> includes a video component <NUM> and a text component <NUM>.

The method <NUM> includes generating a set of annotations (ACT <NUM>). The annotation component <NUM> can generate one or more sets of annotations for the selected digital component. The NLP component <NUM> can process the speech contained in the digital component or closed caption data in or associated with the digital component to generate one or more of the sets of annotations. The annotation component <NUM> can generate one or more sets of annotations based on objects identified in the digital component. For example, the data processing system <NUM> can perform object recognition on the video frames in the digital component to identify objects in the digital component. The annotation component <NUM> can generate a set of annotations based on transitions (e.g., video fade ins or video fade outs) within the digital component. The NLP component <NUM> can generate semantic representations of the speech or text within the digital component. The annotation component <NUM> can generate a set of annotations based on the semantic representations. The annotation component <NUM> can cluster the semantic representations together to determine which portions of the digital component are related to a specific topic or step. For example, for a digital component that is a how-to video of how to fix a bike, the annotation component <NUM> can use semantic representations to identify portions of the how-to video that illustrates the changing of a tire and portions of the how-to video that illustrates how to change the chain of the bike. The annotation component <NUM> can generate a set of annotations that indicates the time portion of the how-to video where tire changing is discussed and the portion of the how-to video where chain changing is discussed.

The method <NUM> can include identifying break points (ACT <NUM>). The data processing system <NUM> identifies the break points based on one or more of the generated sets of annotations. The break points can identify the time points of key frames within the video. The break points can identify scene transitions, the start of each step in a how-to video, the points of interest in the video, the entrance or exit of objects into the video.

An indication of one or more of the break points can be transmitted to the client computing device <NUM>. Also referring to <FIG>, among others, the indication of the break points can be included in digital component <NUM> and transmitted to the client computing device <NUM>. The client computing device <NUM> can render the digital component <NUM> to display a list or indication of the breakpoints the data processing system <NUM> determined are present in the video component <NUM>. The annotation component <NUM> can generate a label for each of the break points. For example, as illustrated in <FIG>, the data processing system <NUM> transmitted a digital component to the client computing device <NUM> that included an indication of three break points. The break points were labeled "Step <NUM>," "Step <NUM>," and "Step <NUM>," and can correspond to a starting time point of step <NUM>, step <NUM>, and step <NUM>, respectively, in the video component <NUM>.

The method <NUM> can also include the data processing system <NUM> selecting one or more supplemental or additional digital components in addition to the primary digital component identified in response to the input audio signal. For example, the supplemental digital components can be related to the same topic, provide additional information related to the primary digital component, or can prompt the end user for more input. Also referring to <FIG>, among others, the data processing system <NUM> selected and transmitted an additional digital component <NUM> to the client computing device <NUM>. The client computing device <NUM> can render and present the additional digital component <NUM> with the primary digital component <NUM>. The additional digital component <NUM> can be an ad for a service provider or content related to the digital component <NUM>.

The method <NUM> can include receiving an input signal (ACT <NUM>). The input signal can be a second input audio signal. The second input audio signal can be detected by the sensor (e.g., microphone) of the client computing device <NUM>. The method <NUM> can include parsing the input signal (ACT <NUM>). The NLP component <NUM> can parse the second input audio signal to identify a keyword, term, or semantic representation within the second input audio signal. For example, and also referring to <FIG>, the second input audio signal can include the phrase "show fixing the chain. " The client computing device <NUM> can display the text <NUM> from the second input audio signal. The NLP component <NUM> can parse the second input audio signal to identify keywords, such as "show" in the above example phase. The NLP component <NUM> can identify that, based on the keyword "show," the use would like to a specific portion of the video component <NUM>. The NLP component <NUM> can identify that the end user wants to see the portion of the video component <NUM> that corresponds to the steps in the video of fixing the bike's chain.

The method <NUM> includes selecting a break point (ACT <NUM>). The data processing system <NUM> selects the break point from the plurality of break points generated based on the one or more sets of annotations of the digital component. The data processing system <NUM> can select the break point based on the keyword, term, or semantic representation identified in the second input audio signal. The data processing system <NUM> can match or associate the keyword, term, or semantic representation with annotations corresponding to each portion of the digital component or each of the plurality of break points.

In the example illustrated in <FIG>, the user provided the second input audio signal that includes the phrase "show fixing the chain. " The second input audio signal can include a request for a specific break point. For example, the client computing device <NUM> rendered the digital component <NUM> that presented to the user a plurality of break points. The second input audio signal can identify or be associated with a break point not previously represented to the end user. For example, as illustrated in <FIG>, the second input audio signal "show fixing the chain" is not a selection of one of the break points: "step <NUM>," "step <NUM>," or "step <NUM>. " When the second input audio signal does not include a specific selection of a break point, the data processing system <NUM> can match or associate the keyword, term, or semantic representation identified in the second input audio signal with annotations corresponding to one of the portions of the digital component or to one of the plurality of break points. For example, the term "chain" can be matched or associated with a portion of the video component <NUM> in which the annotation component <NUM> identified a chain as an object in the video or where the text or semantic meaning of the speech in the portion of the video correspond to a chain.

The method <NUM> includes transmitting a portion of the digital component (ACT <NUM>). The transmitted portion of the digital component corresponds to selected break point. The portion of the digital component can be transmitted to the client computing device <NUM> with instructions that cause the client computing device <NUM> to automatically start playing or rending the transmitted portion of the digital component. For example, and referring to <FIG>, the data processing system <NUM> can select the break point that corresponds to the starting of the portion that illustrates how to change a chain on a bike. The client data processing system <NUM> can transmit the selected portion to the data processing system, and the data processing system <NUM> can begin rendering the portion of the video component that illustrates how to change the chain of the bike. In one example not covered by the claims of the present application, transmitting a portion of the digital component can also include transmitting an indication of the time associated with the selected break point to the client computing device <NUM>. For example, the client computing device <NUM> may have previously received the entity of the digital component. When a break point is selected, the time associated with the break point can be transmitted to the client computing device <NUM> and the client computing device <NUM> can skip to the time in the video associated with the break point.

<FIG> illustrates a block diagram of an example computer system <NUM>. The computer system or computing device <NUM> can include or be used to implement the system <NUM>, or its components such as the data processing system <NUM>. The data processing system <NUM> can include an intelligent personal assistant or voice-based digital assistant. The computing system <NUM> includes a bus <NUM> or other communication component for communicating information and a processor <NUM> or processing circuit coupled to the bus <NUM> for processing information. The computing system <NUM> can also include one or more processors <NUM> or processing circuits coupled to the bus for processing information. The computing system <NUM> also includes main memory <NUM>, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus <NUM> for storing information, and instructions to be executed by the processor <NUM>. The main memory <NUM> can be or include the data repository <NUM>. The main memory <NUM> can also be used for storing position information, temporary variables, or other intermediate information during execution of instructions by the processor <NUM>. The computing system <NUM> may further include a read-only memory (ROM) <NUM> or other static storage device coupled to the bus <NUM> for storing static information and instructions for the processor <NUM>. A storage device <NUM>, such as a solid-state device, magnetic disk or optical disk, can be coupled to the bus <NUM> to persistently store information and instructions. The storage device <NUM> can include or be part of the data repository <NUM>.

The computing system <NUM> may be coupled via the bus <NUM> to a display <NUM>, such as a liquid crystal display, or active matrix display, for displaying information to a user. An input device <NUM>, such as a keyboard including alphanumeric and other keys, may be coupled to the bus <NUM> for communicating information and command selections to the processor <NUM>. The input device <NUM> can include a touch screen display <NUM>. The input device <NUM> can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor <NUM> and for controlling cursor movement on the display <NUM>. The display <NUM> can be part of the data processing system <NUM>, the client computing device <NUM> or other component of <FIG>, for example.

For situations in which the systems discussed herein collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features that may collect personal information (e.g., information about a user's social network, social actions or activities, a user's preferences, or a user's location), or to control whether or how to receive content from a content server or other data processing system that may be more relevant to the user. In addition, certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed when generating parameters. For example, a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, postal code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about him or her and used by the content server.

The subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatuses. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. While a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices).

The terms "data processing system" "computing device" "component" or "data processing apparatus" encompass various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. For example, the interface <NUM>, digital component selector <NUM>, NLP component <NUM>, annotation component <NUM>, parsing component <NUM>, and other data processing system components can include or share one or more data processing apparatuses, systems, computing devices, or processors.

A computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs (e.g., components of the data processing system <NUM>) to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

The subject matter described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back end, middleware, or front end components. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), an inter-network (e.g., the Intemet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system such as system <NUM> or system <NUM> can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network (e.g., the network <NUM>). In some implementations, a server transmits data (e.g., data packets representing a digital component) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server (e.g., received by the data processing system <NUM> from the client computing device <NUM> or the digital component provider device <NUM>).

Actions described herein can be performed in a different order within the scope of the claims.

The separation of various system components does not require separation in all implementations, and the described program components can be included in a single hardware or software product. For example, the NLP component <NUM> or the digital component selector <NUM>, can be a single component, app, or program, or a logic device having one or more processing circuits, or part of one or more servers of the data processing system <NUM>.

Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.

References to "or" may be construed as inclusive so that any terms described using "or" may indicate any of a single, more than one, and all the described terms. For example, a reference to "at least one of 'A' and 'B'" can include only 'A', only 'B', as well as both 'A' and 'B'. Such references used in conjunction with "comprising" or other open terminology can include additional items.

Accordingly, neither the reference signs nor their absence has any limiting effect on the scope of any claim elements.

Claim 1:
A system (<NUM>) to control digital components in a voice-activated system, comprising:
a data processing system (<NUM>) comprising one or more processors and a memory, the one or more processors executing a natural language processor component (<NUM>), an annotation component (<NUM>), a digital component selector (<NUM>) and a parsing component (<NUM>) to:
receive (<NUM>), by the natural language processor component and via interface of the data processing system, a first input audio signal detected by a sensor (<NUM>) at a client computing device (<NUM>);
parse (<NUM>), by the natural language processor component, the first input audio signal to identify a digital component request in the first input audio signal, the digital component request indicating a first digital component, wherein the first digital component comprises audio and/or visual content;
select, by the digital component selector (<NUM>), the first digital component in response to the digital component request identified in the first input audio signal;
generate (<NUM>), by the annotation component, a first set of annotations of the first digital component based at least on speech recognized in the first digital component;
identify (<NUM>), by the parsing component, a plurality of break points (<NUM>) based on at least the first set of annotations to define a plurality of portions (<NUM>) within the audio and/or visual content of the first digital component, wherein each of the portions is defined between two break points (<NUM>(<NUM>), <NUM>(<NUM>)) in the audio and/or visual content of the first digital component;
receive (<NUM>), by the natural language processor component, a second input audio signal detected by the sensor at the client computing device;
parse (<NUM>), by the natural language processor component, the second input audio signal to identify a term in the second input audio signal;
select (<NUM>), by the parsing component, a break point (<NUM>(<NUM>)) from the plurality of break points based on the term in the second input audio signal and the first set of annotations used to identify the break point; and
transmit (<NUM>), by the parsing component to the client computing device, a portion of the first digital component corresponding to the selected break point, wherein the selected break point corresponds to the start of the portion of the first digital component.