Patent Description:
<CIT> discusses receiving a first utterance from a user having a duration below a predetermined threshold, identifying at least one further utterance from the user that provides additional information, generating a concatenated utterance by concatenating the first utterance with the at least one further utterance, transmitting the concatenated utterance to a speech recognition server, receiving a transcription of the concatenated utterance from the speech recognition server that includes a transcription of the first utterance, and extracting the transcription of the first utterance from the transcription of the concatenated utterance.

The object of the invention is solved by the subject-matter of the independent claims.

With the advent of audio streaming, speech transcription has gained a new level of notoriety but at the cost of greater complexity. Large enterprises have quickly learned to replace labor-intensive costs with price-effective automated voice-driven systems that all too well perform common services such as directing existing customers to consummating securities transactions, fulfilling consumer product orders, or directing potential customer inquiries. Whereas networks of not long ago lacked adequate streaming speeds to carry real-time audio files, large gains in recent technology advancements are allowing networks greater and broader opportunities. Enjoying the benefits of fast audio processing, small to large scale commercial enterprises now enjoy providing their customers with flexible cost-based speech-to-text applications.

Take the case of interactive voice recognition (IVR) services. A typical IVR service provides customers various capabilities, for example, servicing a financial security transaction through an E-Trade site, booking with a hotel or making a hospital appointment through respective websites, or servicing a common banking transaction request through a financial or third-party website. On the receiving end, an enterprise system automatically processes audio signals generated from user choice input, a user utterance for example, received through a user voice transmitter device, such as a user smartphone. In such user voice-driven applications, an utterance may be approximately <NUM> seconds in duration. Content discovery voice search engines in media equipment applications accommodating user voice-driven commands receive user utterance input of a similar duration. Through a smart television remote device, for example, a user may speak a short command, "turn to channel <NUM>", to effect a channel change. An audio stream duration for securities transactions, placed through a handheld device, is not much different.

To decipher user utterances, enterprises typically send audio streams to audio speech recognition (ASR) services. ASR services convert audio streams into text files and send back the text files to requesting enterprises for relevant text extraction. In large scale applications, of which there are many, audio stream files are received in droves. Accordingly, existing speech-to-text (STT) services are a significant expense item for enterprises, in large part due to technology complexity, equipment maintenance, and requisite software and hardware enhancements. For example, a cable company may receive hundreds of thousands, if not millions or tens or even hundreds of millions of audio stream files from users located around the world nearly simultaneously. Systems must keep up with reliable and efficient processing of the massive numbers of incoming audio and audio-related signals.

Currently, each audio service recognition (ASR) service offered by companies like Google, Amazon, and Nuance, price an ASR request using a fixed and minimum cost financial model regardless of audio file duration. For example, a <NUM>-second audio stream may incur the same minimum fixed fee for a <NUM>-, <NUM>- or even <NUM>-second audio stream. The wide file size range and fixed-fee financial model is in large part based on a common connection load, the total number of parallel connections, irrespective of audio file duration. Beyond a threshold audio stream duration, however, processing utilization (such as central processing unit (CPU) use) for transcribing an audio file may and often does increase with audio file duration hence a different pricing model structure maybe employed. Stated differently, speech signals with significantly shorter durations are the subject of a different price model than those with longer durations.

In accordance with an existing financial model, employed by the entertainment industry-at-large, for example, audio signals with considerably shorter duration, in the order of <NUM>-<NUM> seconds, are typically priced per number of STT (or ASR) requests or during a predetermined time period. A reduction in the number of STT service requests or a reduction in audio stream length therefore serve as an incentive for reducing ASR service costs.

In accordance with disclosed embodiments and methods, a two-prong audio stream processing technique is employed, one that is transaction-based and another that is time-based. A decision to use one processing technique over the other is largely based on the ASR service financial model but it can be based at least in part on other non-related factors, as discussed below. In a system with a large number of file requests, a transaction-based approach may be better suited whereas a system with fewer, but longer audio files may warrant a time-based approach. In either approach, processing audio stream files prior to transmission to a STT service (or an automatic speech recognition (ASR) service) is optimized by sending a single payload (in a transaction-based approach) or compact files (in a time-based approach). More than one speech signal may form a single compact payload for transcription as a cost-saving and a system efficiency-enhancing measure, particularly at large scales. In the case of time-based approaches, dead time, e.g., silence or non-audible transmission periods, and non-meaningful speech content, are eliminated in compact audio files to shrink audio file sizes and reduce STT service costs. At scale, even greater cost savings may be realized based on a larger number of audio stream files.

In accordance with a pre-STT service transaction-based approach for audio stream processing, N number of audio streams are received during a time window, "N" being an integer value. Each or some of the N audio streams may be received from an independent or unique audio stream source and includes speech content. In non-limiting examples, audio streams may be received from user equipment devices, such as media or entertainment equipment devices, personal computing devices, or user handheld devices. For example, User A may utter a command through a remote control device of a smart entertainment set, User B may utter a command through a laptop microphone while User C may verbalize a command through a smart phone microphone.

In some embodiments, N-<NUM> number of audio stream separators may be generated based on N-number of audio streams. In response to receiving the audio streams, a determination may be made as to whether at least some of the audio streams require transaction-based speech-to-text services and in response to determining that at least one of the audio streams does not require transaction-based speech-to-text services, a determination may be made to perform time-based speech-to-text services on the received audio streams.

In accordance with a time-based service, an audio stream is compacted to generate a compacted audio stream and the compacted audio stream is transmitted to an ASR service for transcription of the speech content to text content. Performing compacting may include removing non-meaningful voice and/or silence from the speech content. In response to transmitting the compacted audio stream for transcription, text content transcription of the audio stream is received from the ASR service for relevant text extraction. Audio stream compacting increases the frequency of transmission of compacted audio streams. At scale, an even greater increase in the frequency of transmission may be realized.

In some embodiments and methods, each audio stream of a second set of processed audio streams, different than an initially-compacted set of audio streams, is trimmed to remove excess speech content from the speech content of each of the second set of the audio streams. Compacting of the second set of processed audio streams may be performed where transcription charges for transcription of audio streams are transaction-based, for example.

In some embodiments, in response to receiving an audio stream and in response to determining the received audio stream does not require time-based speech-to-text services, a transaction-based speech-to-text services approach is implemented. A set of N number of received audio streams (a "set" being a value with a range of <NUM> to N) is concatenated into a buffer to generate a concatenated audio stream and N-<NUM> number of audio stream separators (or one audio stream less than the total number of audio streams in a set) are inserted into the concatenated audio stream. An audio stream separator is inserted between every two adjacent audio streams of the concatenated audio stream to generate a single audio stream payload. Each audio stream separator delineates a beginning of a next audio stream and an end of a preceding audio stream. The single audio stream payload is subsequently transmitted for transcription to an ASR service for converting speech content to text content for all or the set of N audio streams, as the case maybe. In response to transmitting the single audio stream payload for transcription, a text file is received from the ASR service. The text file includes text content delineated with the audio stream separators to designate separations between text content of each audio stream (or "audio file" or "audio stream file").

In some embodiments, the buffer size is based on a time window duration. That is, the speech content of each audio stream may have an associated duration and the buffer size maybe based on a multiple number of a maximum speech content duration among the durations of speech content of the set of audio streams. The maximum duration may correspond to a minimum base transcription service price. That is, transcription charges for transcription of received audio streams may be based on the maximum audio stream duration.

In an embodiment, the buffer may be of a variety of buffer types suitable for processing and storing audio streams. For example, the buffer may be a linear or circular buffer, a last-in-first-out or first-in-first-out buffer. The order of storing the audio streams may be based on or independent of the order in which the audio streams are received from audio sources.

In some disclosed methods, upon receipt of a transcribed text file from a STT (or ASR) service, or during post-STT service processing, each text content of a corresponding audio stream is separated from adjacent text files of corresponding adjacent audio streams in the single payload text file by the audio stream separators. As with pre-STT audio file processing, each delinaeated text file corresponds to an independent audio source.

Prior to a payload transmission for transcription, for each audio stream, a determination may be made as to whether the audio stream can be transmitted in its entirety during the time window and based on the determination, a remaining set of audio streams may be saved for transmission as a part of a subsequent single audio stream payload. For example, in the event one or more audio streams of a current set of audio streams is not received in their entirety during the time window, the one or more audio streams of the current set that are not received in their entirety may be saved in the buffer or other storage locations to be processed and transmitted to the ASR service with a subsequent payload while audio streams received in their entirely and/or previously scheduled for transmission with a current payload may be transmitted with the current payload. The subsequent audio stream payload may be made of another set of audio streams in the N audio streams or a set of audio streams in an N+<NUM> (or N plus a number greater than one) audio streams, of the left-behind audio streams (not previously received in their entirety during a corresponding time window or not previously scheduled for transmission with a current payload) of a previous payload, or a combination. Accordingly, the current single audio stream payload may exclude the remaining set of audio streams.

Transmitting a single audio stream payload for transcription may be performed immediately following processing the last audio stream of a time window but no later than an end of a wait time, the wait time starting from a request for onboarding the buffer with one or more of the N audio streams and ending at a subsequent request to re-onboard the buffer with a next audio stream (of the same or a new set of N number of audio streams). Alternatively, transmitting a single audio stream payload for transcription may be performed immediately following processing the last audio stream of a time window but no later than an end of a wait time, the wait time starting from a request for onboarding the buffer with one or more of the set of N audio streams and ending at a subsequent request to re-onboard the buffer with a next audio stream (of the same or a new set). In some embodiments, transmitting a single audio stream payload for transcription may be performed after processing a predetermined number of audio streams regardless of a time window or it may based on a time window and a predetermined number of audio streams. Transmitting may be performed despite buffer density to prevent noticeable delays in receiving the text file. Where the audio streams have different audio stream sizes, disclosed methods may transmit the payload at the end of the time window regardless of audio stream sizes.

The above and other objects and advantages of the disclosure will be apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings, in which:.

<FIG> illustrates an overview system of transaction-based ASR services, in accordance with some embodiments of the disclosure. In <FIG>, a transaction-based ASR services system is configured as a transaction-based ASR services system <NUM> to include an audio stream processor <NUM> receiving audio stream files from multiple and independent audio sources <NUM>. Audio stream processor <NUM> is further shown to provide a single audio stream payload <NUM> for audio-to-text services to ASR services <NUM>. Audio sources <NUM> receive audio input <NUM>, generally provided by respective users who may be located around the world. Audio sources <NUM> may receive non-user generated audio input, in some embodiments. In a non-limiting example, audio sources <NUM> may receive automated commands from users, bots or machine-learning or artificial intelligence processing sources.

Audio stream processor <NUM> may transmit and receive audio and audio-related data to and from audio sources <NUM> in various manners. For example, communications between audio steam processor <NUM> and one or more audio sources <NUM> may be implemented through a wireless (WiFi) network, wired network, local area network (LAN), or using Bluetooth as further discussed with reference to <FIG>.

Each of the audio sources <NUM> may be a remotely or locally situated device relative to audio stream processor <NUM>. In non-limiting application examples, audio sources <NUM> comprise laptops, entertainment equipment or handheld devices, or any other suitable audio source. In some embodiments, one or more audio sources <NUM> or one or more audio input <NUM> may be generated by the same respective source or same user (or bot), for example.

In the particular embodiment of <FIG>, three audio input <NUM> are received by three respective audio sources <NUM>. Each audio stream generated from one of the three audio input <NUM> includes speech content. For example, still referring to <FIG>, each of the three exemplary audio input <NUM> is shown to include user audio commands "Show me romantic comedy," "Turn to channel <NUM>", and "Record Frozen. " It is understood that while three input <NUM> and three sources <NUM> are shown and discussed herein, any other number of input and sources may be employed. In a typical system <NUM> application, the number of input and sources can exceed hundreds and millions.

As previously indicated, audio stream processor <NUM> may receive audio stream files from one or more locally-situated audio sources <NUM>. For example, one or more of the sources <NUM> may reside in media equipment devices and communicatively coupled to audio stream processor <NUM>. In some embodiments, audio stream processor <NUM> and one or more of the audio sources <NUM> are remotely located. For example, audio stream processor <NUM> may reside in a network cloud, such as a network server, while one or more of the sources <NUM> may be situated in remotely-located user devices, such as laptops or handheld devices communicatively coupled to audio stream processor <NUM> through the network cloud (or "communication network"), as discussed relative to <FIG>.

Audio stream processor <NUM> is shown to include a buffer <NUM>, a buffer processor <NUM>, an audio stream interface <NUM>, an audio signal processor <NUM>, and a storage <NUM>, any, all, or a combination of which may be implemented in hardware, software, or virtually. For example, buffer <NUM> may be made of registers, volatile or non-volatile memory, or database devices. Buffer <NUM> may alternatively or additionally comprise pointers to memory or storage locations or virtual addresses that when mapped to logical and ultimately physical addresses point to physical storage or memory, such as but not limited to storage <NUM>. Buffer <NUM> may therefore comprise any form of suitable storage for audio and audio-related files.

Buffer processor <NUM> generally manages data access from and to buffer <NUM>; audio stream interface <NUM> generally manages data input/output functions to and from audio stream processor <NUM>; audio signal processor <NUM> generally performs audio and audio-related processing functions and arbitration of data input to the audio stream processor <NUM> and data output from audio stream processor <NUM> and further directs other components of the audio stream processor <NUM>, such as buffer processor <NUM>, audio stream interface <NUM>, and storage <NUM>, in performing respective functions; and storage <NUM> generally maintains data and program instructions utilized by audio stream processor <NUM> in carrying out its operations. It is understood that anyone or a combination of the buffer processor <NUM>, audio stream interface <NUM>, audio signal processor <NUM>, and storage <NUM> may be located locally or remotely relative to one another and relative to audio stream processor <NUM>. For example, storage <NUM> may be a part of a device or devices housing buffer <NUM> or it may be remotely-situated with respect to buffer <NUM>. Similarly, audio signal processor <NUM> may be locally or remotely situated relative to buffer <NUM> and/or storage <NUM>. In a non-limiting example, at least a part of buffer <NUM> and buffer processor <NUM> are locally situated relative to one another to effect efficient data access to buffer <NUM>. Similarly, at least a part of storage <NUM> may be locally situated relative to buffer <NUM> to enhance system performance by allowing fast memory and/or storage transactions. For example, at least a part of storage <NUM> may be made of cache memory to facilitate fast instruction execution.

As will be evident relative to the following operational example of <FIG>, in response to transmitting a single payload, such as single audio stream payload <NUM> to ASR services <NUM>, audio stream processor <NUM> may receive from ASR services <NUM> a single text file or transcription of all audio stream files transmitted to ASR services <NUM>.

In an operational example, through audio stream interface <NUM>, audio stream processor <NUM> receives audio stream files from audio sources <NUM> during a time window. Buffer processor <NUM>, with the assistance of audio signal processor <NUM>, saves the received audio stream files into the buffer <NUM>. In some embodiments, buffer processor <NUM>, under the direction of audio signal processor <NUM>, may save received audio stream files in storage <NUM> for audio stream processing and after processing, buffer processor <NUM> may transfer the processed audio streams to buffer <NUM> for transmission to the ASR services. The time window may be programmably set by, for example, the buffer processor or audio signal processor. The time window may be set based on an expected and/or average audio stream duration range, audio stream transmission rates (from audio sources <NUM>), and/or time window determination basis for proper audio stream processor operation.

Audio signal processor <NUM> and buffer processor <NUM> may execute program instructions stored in storage <NUM> to implement the functions performed on and in buffer <NUM> to process the incoming audio files. Buffer processor <NUM> may be directed by audio signal processor <NUM> to perform buffer <NUM>-related functions. It is understood that storage <NUM> may be made of one or more storage devices locally or remotely situated relative to one another. In some embodiments, storage <NUM> may comprise logical or virtual links or pointers that uniquely identify one or more physical address including the data of interest or the address where the data of interest is to be stored.

In a general example, through audio stream interface <NUM>, audio stream processor <NUM> receives "N" number of audio stream files from audio sources <NUM> where "N" is an integer value. Before storing the audio stream files in buffer <NUM>, buffer processor <NUM> concatenates a set of the received audio streams to generate a concatenated audio stream. The set may be one or more, up to and including N, number of audio streams. Before storing the received audio streams in the buffer <NUM>, buffer processor <NUM> further generates N-<NUM> (or one less than the total number of audio streams in the set of audio streams) audio stream separators for the N audio stream files. For example, in system <NUM>, buffer processor <NUM> may generate an audio stream separator for every two adjacent audio streams of the concatenated audio stream to generate a single audio stream payload, such as single audio stream payload <NUM>. The total number of audio stream separators is therefore typically one less than the total number of audio streams of a concatenated audio stream. Each audio stream separator delineates a beginning of a next audio stream and an end of a preceding audio stream. In some embodiments, buffer processor <NUM> generates audio stream separators after storing the received audio streams in the buffer <NUM>.

In the example of <FIG>, three audio stream separators are generated for the three audio stream files received from sources <NUM>. That is, the set of audio stream files equals N received audio stream files. It is understood that for a different number of sources a corresponding different number of audio stream separators may be generated by buffer processor <NUM>.

In some embodiments, buffer processor <NUM> stores the concatenated audio stream in the buffer <NUM> and then adds the audio stream separators as delineations markers between each two adjacent stored audio files of a concatenated audio file. In some embodiments, buffer processor <NUM> first adds the audio stream separators to the concatenated audio stream while the concatenated audio stream is stored in a location other than buffer <NUM>, for example storage <NUM>, and upon completion of processing (generating the single audio stream payload <NUM>), buffer processor <NUM> transfers the generated payload to buffer <NUM> for transmission to ASR services <NUM>.

In the example of <FIG>, buffer processor <NUM> concatenates the three audio stream files with an audio stream separator between every two adjacent audio streams of the concatenated audio stream in buffer <NUM> to generate the single audio stream payload <NUM>, each audio stream separator delineating a beginning of a next audio stream and an end of a preceding audio stream separators. Upon generating and inserting all (N-<NUM>) audio stream separators into the three audio streams, buffer processor <NUM> stores the three audio streams with inserted audio stream separators, collectively forming the single audio stream payload <NUM>, in buffer <NUM>. As previously indicated, each audio stream of a concatenated audio stream is separated from a next or adjacent audio stream by an audio stream separator between every two adjacent audio streams of the concatenated audio stream in buffer <NUM>, forming the single audio stream payload <NUM>. For example, audio stream <NUM> is separated from an immediately previous and adjacent audio stream, shown directly below stream <NUM> with dashed hashed lines, by an audio stream separator, and an immediately preceding audio stream to the previous audio stream, shown by opposite cross hashed lines immediately below the previous audio stream (in buffer <NUM>), is separated from the previous audio stream by another audio stream separator. In this respect, an audio stream separator is inserted between every two adjacent audio streams of the concatenated audio stream in the buffer <NUM> to generate the single audio stream payload <NUM>, each audio stream separator delineating a beginning of a next audio stream and an end of a preceding audio stream of a concatenated audio stream.

In an embodiment, audio stream processor <NUM> receives the N audio streams from sources <NUM> during a set time window. The size of buffer <NUM> may be based on a duration of the time window. In some embodiments, the buffer size may be based on a multiple of the maximum audio stream duration. For example, where three audio streams of <NUM>, <NUM>, and <NUM> seconds in duration are received by audio stream processor <NUM>, the buffer size must be large enough to accommodate a time window of <NUM> seconds.

Under the direction of audio signal processor <NUM>, audio signal processor <NUM> or buffer processor <NUM>, as the case maybe, transmits the single audio stream payload <NUM> for transcription of each of the N audio file speech content- (of each audio stream) to-text content, through audio stream interface <NUM>, to ASR services <NUM>. In response, audio stream processor <NUM>, through audio stream interface <NUM>, receives a transcription of each of the speech content of the N audio streams in the form of a single text content file from ASR services <NUM>. The received text file may include the text content of all N audio files delineated with the audio stream separators. In the case where a set of audio files that includes less than N number of audio files is included in the payload, the received text file includes a number of text files corresponding to the set number of audio files. Audio stream processor <NUM> may then perform extraction of, or solicit an independent device or service, to perform extraction of relevant text information from one or more of the N text content files.

Audio stream processor <NUM> may transmit and receive data to and from ASR services <NUM> in various manners. For example, communications between audio steam processor <NUM> and ASR services <NUM> may be implemented through a wireless or wired network, such as WiFi and local area network (LAN), respectively.

In some embodiments, the size of buffer <NUM> is based on a time window duration. That is, the speech content of each audio stream may have an associated duration and the buffer size maybe based on a multiple number of a maximum speech content duration among the durations of the speech content of the set of audio streams. The maximum duration may correspond to a minimum base transcription service price. That is, transcription charges for transcription of received audio streams may be based on the maximum audio stream duration.

Prior to the transmission of payload <NUM> for transcription, for each audio stream, buffer processor <NUM> may make a determination as to whether the audio stream can be transmitted in its entirety during the time window and based on the determination, a remaining set of audio streams may be saved for transmission with a subsequent single audio stream payload. For example, in the event one or more audio streams of a current set of audio streams are not received in their entirety during the time window, the one or more audio streams of the current set that are not received in their entirety may be saved in the buffer or other storage locations to be processed and transmitted to the ASR service with a subsequent payload while audio streams received in their entirely during the time window and/or previously scheduled for transmission with a current payload may be transmitted with the current payload. The subsequent audio stream payload may be made of another set of audio streams in the N audio streams or a set of audio streams in an N+<NUM> (or N plus a number greater than one) audio streams, left-behind audio streams (not previously received in their entirety during a corresponding time window or previously scheduled for transmission with a current payload) of a previous payload, or a combination. Accordingly, the current single audio stream payload may exclude the remaining set of audio streams.

Transmitting a single audio stream payload, such as payload <NUM>, for transcription may be performed immediately following processing the last audio stream of a time window but no later than an end of a wait time, the wait time starting from a request for onboarding the buffer with one or more of the N audio streams and ending at a subsequent request to re-onboard the buffer with a next audio stream (of the same or a new set of N number of audio streams). Alternatively, transmitting a single audio stream payload for transcription may be performed immediately following processing the last audio stream of a time window but no later than an end of a wait time, the wait time starting from a request for onboarding the buffer with one or more of the set of N audio streams and ending at a subsequent request to re-onboard the buffer with a next audio stream (of the same or a new set). In some embodiments, transmitting a single audio stream payload for transctiption may be performed after processing a predetermined number of audio streams regardless of a time window or it may based on a time window and a predetermined number of audio streams. Transmitting may be performed despite buffer density to prevent noticeable delays in receiving the text file. Where the audio streams have different audio stream sizes, disclosed methods may transmit the payload at the end of the time window regardless of audio stream sizes.

In an embodiment, buffer <NUM> may be configured in a variety of buffer type implementations suitable for processing and storing audio streams. For example, buffer <NUM> may be a linear or circular buffer, a last-in-first-out or first-in-first-out buffer. The order of storing the audio streams may be based on or independent of the order in which the audio streams are received from audio sources.

<FIG> illustrates an overview system of time-based ASR services, in accordance with some embodiments of the disclosure. In <FIG>, a time-based ASR services system is configured as a time-based ASR services system <NUM> to include an audio stream processor <NUM> receiving audio stream files from an audio source <NUM>. Audio stream processor <NUM> is further shown to provide a compacted audio stream for audio-to-text services to ASR services <NUM>. Each of ASR services <NUM> and <NUM> may be an ASR service suitable for processing audio signals. For example, ASR services <NUM> and <NUM> may be performed by any of various ASR service providers including, without limitation, Google LLC of Mountain View, CA and Amazon. of Seattle, WA.

Audio source <NUM> receives audio input <NUM>, generally provided by a respective user who may be located around the world. Audio source <NUM> may receive non-user generated audio input. In a non-limiting example, audio source <NUM> may receive automated commands from one or more users, bots or machine-learning or artificial intelligence processing sources. In this respect, audio source <NUM> is analogous to the sources <NUM> of <FIG>. Indeed, with the exception of functions and embodiments discussed herein or those of obvious variants, the system <NUM> is analogously configured as system <NUM>.

In system <NUM>, while one audio stream is presumed received by audio stream processor <NUM>, it is understood that typically more than one audio stream may be received from the same or other sources. For example, audio stream processor <NUM> may receive three audio streams <NUM>. In some embodiments, audio signal processor <NUM>, audio stream interface <NUM>, and storage <NUM> are configured as audio signal processor <NUM>, audio stream interface <NUM>, and storage <NUM>, respectively, of <FIG>.

In an exemplary operational scenario, audio stream processor <NUM> receives, through audio stream interface <NUM>, an audio stream including speech content from audio source <NUM>. Audio stream compactor <NUM>, operating under the direction of audio signal processor <NUM>, compacts the received audio stream to generate a compacted audio stream. Audio stream processor <NUM> transmits the compacted audio stream for transcription, through audio stream interface <NUM>, to ASR services <NUM>. In response to transmitting the compacted audio stream for transcription, audio stream processor <NUM>, through audio stream interface <NUM>, receives from ASR services <NUM> a text file with text content that is a transcription of the audio stream. Audio stream processor <NUM> may itself or through the solicitation of an independent device or service effect text content search of the returned text file for relevant information.

Audio stream compactor <NUM> may compact the audio stream from audio source <NUM> in a variety of manners. Done in any suitable manner, compacting an audio stream may remove non-meaningful voice and/or silence from the speech content of an audio stream while compacting the audio stream (or "audio file") to increase the frequency of transmission of the audio stream to ASR services <NUM>. Compacting the audio stream may comprise trimming the audio stream to remove excess speech content from the speech content of the audio stream. It is understood that more than one audio stream may be trimmed at any given time. In some embodiments, compacting includes trimming the audio stream of a second set of one or more audio streams to remove excess speech content from the speech content of each of the second set of audio streams. Trimming of the second set of audio streams may be performed where transcription charges for transcription of processed audio streams are transaction-based for improved improve cost-effectiveness, for example.

In some embodiments, audio stream compactor <NUM> performs audio stream compacting by use of lossy or lossless compression algorithms. For example, run-length encoding and decoding may be employed by implementing Lempel-Ziv (LZ) or Lempel-Ziv-Welch (LZW) algorithms. Audio stream compactor <NUM> may be configured in hardware, software or virtually.

In an embodiment, either or both audio streams of systems <NUM> and <NUM> may be encrypted for privacy and reliability reasons. In such cases, decryption may be necessary at a receiving end.

<FIG> show an exemplary single audio stream payload part, in accordance with disclosed methods and embodiments. Analogous to audio stream payload <NUM>, single audio stream payload <NUM> may be generated by audio stream processor <NUM> of <FIG> and stored in buffer <NUM> by buffer processor <NUM>, as previously described relative to <FIG>. <FIG> shows a part of exemplary single audio stream payload <NUM>, built by an audio stream processor of disclosed methods and systems. Payload <NUM> is shown with three audio streams, audio stream "a", audio stream "b", and audio stream "c". Each audio stream "a"-"c" includes two fields, an audio stream content <NUM> and an audio stream header <NUM>. For example, audio stream content <NUM> comprises a series of audio stream contents 302a, 302b, and 302c, each from a unique and independent source or a combination of unique and independent sources, such as sources <NUM>, as earlier described relative to <FIG>. Audio stream header <NUM> is a series of audio stream headers, 304a, 304b, and 304c, each belonging to a corresponding audio stream "a"-"c", respectively. Similarly, audio stream separator <NUM> is a series of audio stream separators, 306a-306c, each separating two adjacent audio streams of audio streams "a"-"c". For example, audio stream separator 306a is shown to delineate audio stream "a" from a preceding audio stream (not shown in <FIG>), audio stream separator 306b is shown to delineate audio stream "a" from audio stream "b" (or audio stream content 302a and audio stream header 304b), and audio stream separator 306c delineates audio stream "b" from audio stream "c" (or audio stream contents 302b and audio stream header 304c).

In an embodiment, each separator 306a-306c is effectively an audio stream delineation indicator and may be implemented in a variety of manners. For example, each audio stream separator may be a flag, bit (or bytes), a particular memory address or a particular value, a pointer, linked list, or any other designator uniquely identifying the location between two adjacently situated independent audio streams.

Audio stream headers 304a-304c each describe aspects of a corresponding audio stream content. In a non-limiting example, an audio stream header may indicate an audio stream content transmission time, the time at which a corresponding audio stream content was transmitted by a corresponding audio source, or an audio stream length, the length of a corresponding audio stream or audio stream content, or whether or the type of encoding/decoding possibly employed prior to transmission and upon receipt of an audio stream, respectively. In this respect, audio stream header 304a includes information relating to audio stream content 302a of audio stream "a"; audio stream header 304b includes information relating to audio stream content 302b of audio stream "b"; and audio stream header 304c includes information relating to audio stream content 302c of audio stream "c". While shown located at the beginning of each audio stream "a", "b", and "c", in some embodiments, an audio stream header may be located at the end of or embedded within a corresponding audio stream.

Audio stream content may and typically does include meaningful voice information and non-meaningful voice or silent periods. Non-meaningful information may be the utterance of "Uh" or "Uhm" and a silent period may be a period where no voice or speech is audible or comprehensible.

In <FIG>, an exploded view of audio stream "b" of <FIG> is shown to include non-meaningful voice and/or silent audio content at <NUM> while meaningful speech content is shown at <NUM>. In accordance with an embodiment, such as the system <NUM>, audio stream compactor <NUM> of <FIG> would remove the non-meaningful voice and/or silent content parts of the audio stream (shown at <NUM> in <FIG>) while leaving the meaningful speech content <NUM> hence reducing the audio stream size of audio stream "b" by approximately <NUM>% and increasing system efficiency by approximately <NUM>%, for instance. In some embodiments, the net result is a reduction in ASR service expenses. In some embodiments, an audio stream may be further compacted by trimming at least some of the audio stream header. For example, while the identity of an audio source may remain relevant, the time of audio stream transmission may be a suitable candidate for removal.

In some embodiments, the system may determine the speech-to-text service based on incoming audio streams, such as those from audio sources <NUM> of <FIG> or <NUM> of <FIG>. <FIG> depicts an illustrative flowchart of a process for determining a STT (or ASR) service approach based on incoming audio streams. In <FIG>, a process <NUM> is shown in flowchart form for determining an appropriate STT service based on an analysis of incoming audio streams. In some embodiments, process <NUM> may be implemented by the audio signal processor <NUM> of <FIG> or the audio signal processor <NUM> of <FIG>. It is understood that other suitable signal processor or computing devices may implement the steps and determinations of process <NUM>.

At step <NUM>, an N number of audio streams are received, for example by audio stream processor <NUM> or <NUM>, of <FIG> and <FIG>, respectively. Next, at <NUM>, a determination is made as to whether the incoming audio streams are to be time-based STT serviced. If at <NUM>, the determination is made to service the audio streams using a time-based approach ("Yes" at <NUM>), process <NUM> continues onto to step <NUM> and a time-based STT service approach, for example, like the STT service approach of <FIG>, is implemented. Otherwise ("No" at <NUM>), process <NUM> proceeds to <NUM> where a second determination is made. At <NUM>, the determination is based on a transaction-based service approach. If at <NUM>, a determination is made to implement a transaction-based service approach ("Yes" at <NUM>), process <NUM> proceeds to step <NUM> where a transaction-based STT service, such as that of <FIG>, is implemented. If the determination at <NUM> yields a negative outcome ("No" at <NUM>), process <NUM> may resume to process a next batch of N audio streams starting from step <NUM>. In some embodiments where other forms of STT services are available, process <NUM> may proceed to test for other services after the determination at <NUM> yields a negative result instead of continuing from step <NUM>. Process <NUM> may end upon testing for all available STT services or process <NUM> may resume from step <NUM> when test for all available STT services is exhausted.

In some embodiments, the order of steps and determinations of process <NUM> may be changed. For example, the determinations at <NUM> and <NUM> may be swapped such that the process tests for a transaction-based approach prior to testing for a time-based approach.

The determination to proceed with a transactional versus a time-based approach may rest, at least in part, on a number of audio stream- or environment-related factors. For example, if the average audio stream size of N number of audio streams exceeds a threshold, process <NUM> may choose a time-based service approach. In a slow network environment, a transactional-based service approach may be better suited. In some embodiments, determining whether at least one of the audio streams requires transaction-based speech-to-text services is simply based, at least in part, on the ASR services <NUM> cost-structure. In some embodiments, determining whether at least one of the audio streams requires transaction-based speech-to-text services is based, at least in part, on the size of the time window for receiving the N number of audio streams.

<FIG> depicts an illustrative flowchart of a process for determining a transaction-based STT service approach. In <FIG>, a process <NUM> is shown in flowchart form for determining a transaction-based STT services approach. The steps and determinations of process <NUM> will be described relative to <FIG> but it is understood that process <NUM> or any derivations thereof may be equally applicable to embodiments other than that of <FIG>.

At step <NUM>, in <FIG>, N number of audio streams, each with speech content, are received during a specified time window. Speech content of each of the N audio streams is analogous to audio stream content 302a, 302b and 302c of <FIG>. Audio stream interface <NUM> may receive <NUM> audio streams from sources <NUM>, as previously described relative to <FIG>. Next, at <NUM>, a determination may be made by buffer processor <NUM> or audio signal processor <NUM> as to whether the ASR service to-be-performed is transaction-based or not and if not ("No" at <NUM>), process <NUM> ends at <NUM>. Otherwise ("Yes" at <NUM>), process <NUM> continues to step <NUM>. In some embodiments, instead of ending at <NUM>, process <NUM> may test for other types of ASR services, such as a time-based service or still other services, as described relative to <FIG>.

At step <NUM>, buffer processor <NUM> generates N-<NUM> audio stream separators and process <NUM> next implements step <NUM>. At step <NUM>, buffer processor <NUM> or audio signal processor <NUM> concatenate the N audio streams into a buffer, such as buffer <NUM>, to generate a single concatenated audio stream. Process <NUM> subsequently performs step <NUM>. At step <NUM>, the audio stream separators generated at step <NUM> are inserted into the concatenated audio stream of step <NUM> to generate a single audio stream payload in buffer <NUM>. It is understood that while buffer <NUM> is shown as a single buffer unit or device, buffer <NUM> may comprise more than one device, therefore, dispersing the single audio stream payload across multiple buffer devices. Next, after completion of step <NUM>, at step <NUM>, audio stream processor <NUM> transmits the single audio stream payload (such as payload <NUM>) of step <NUM> from buffer <NUM> to an ASR service, such as without limitation, ASR service <NUM>, through audio stream interface <NUM> for transcription.

Audio stream processor <NUM> may transmit the generated single audio stream payload for transcription immediately following the processing of the last or Nth audio stream. For example, audio stream processor <NUM> may transmit the single audio stream payload <NUM> after insertion of the N-<NUM> audio stream separator into the concatenated audio stream.

In an embodiment, buffer processor <NUM> of audio stream processor <NUM> makes a request of audio signal processor <NUM> to onboard the next set of N audio streams onto buffer <NUM>. Audio stream processor <NUM> may transmit the single audio stream payload no later than an end of a wait time, the wait time starting from a request for onboarding buffer <NUM> with one or more of the N audio streams and ending at a subsequent request to re-onboard buffer <NUM> with the next batch (or set) of N audio streams.

Buffer <NUM> may be filled to capacity upon building payload <NUM> but in some embodiments, payload <NUM> may not consume the entire capacity of buffer <NUM>. For example, the time window for receiving all N audio streams may close before the buffer is full or the audio streams may be too short to fill buffer <NUM> to capacity. In some embodiments, audio stream processor <NUM> transmits payload <NUM> to ASR services <NUM> from buffer <NUM> despite the density of buffer <NUM> to prevent noticeable delay by ASR services <NUM> in receiving the text file.

The N audio streams from sources <NUM> may have different audio stream sizes. In some embodiments, payload <NUM> is transmitted when the specific time window ends despite the audio stream sizes. Prior to the transmitting step <NUM>, in <FIG>, audio signal processor <NUM> of audio stream processor <NUM> may determine, for each of the N audio streams, whether the audio stream can be transmitted in its entirety during the time window and based on the determination, audio signal processor <NUM> may then transmit only a subset of the N audio streams, those that have been received in their entirety while leaving a remaining subset of partially-received audio streams for future transmission.

<FIG> depicts an illustrative flowchart of a process for determining a time-based STT service approach. In <FIG>, a process <NUM> is shown in flowchart form for determining a time-based STT services approach in accordance with disclosed methods. The steps and determinations of process <NUM> will be described relative to <FIG> but it is understood that process <NUM> or any derivations thereof may be equally applicable to embodiments other than that of <FIG>.

At step <NUM>, an audio stream is received from an audio source, such as source <NUM>. The audio stream includes speech content, such as audio stream content 302a, 302b and 302c of <FIG>. The audio stream may be received through audio stream interface <NUM> of audio stream processor <NUM>. At <NUM>, the audio signal processor <NUM> determines whether ASR services are to be performed pursuant to a time-based approach. Audio signal processor <NUM> may make the determination at <NUM> based on various factors, such as but not limited to those discussed above. At <NUM>, a determination may be based on whether at least one of the received audio streams requires time-based speech-to-text services. The foregoing determination may be based at least in part on an average size of N number of audio streams or an average size of the set of audio streams in an application receiving N audio streams.

If a time-based approach is determined not to be implemented ("No" at <NUM>), process <NUM> ends at <NUM>, otherwise ("Yes" at <NUM>), process <NUM> continues to step <NUM>. In some embodiments, the determination at <NUM> may be performed as described relative to <FIG>.

At step <NUM>, audio stream compactor <NUM> compacts the received audio stream and process <NUM> continues to step <NUM>. Next, at step <NUM>, audio signal processor <NUM> transmits the compacted audio stream, through audio stream interface <NUM>, to ASR services <NUM> for transcription. At step <NUM>, in response to transmitting the compacted audio stream to ASR services <NUM>, audio stream processor <NUM> receives, from ASR services <NUM> through audio stream interface <NUM>, a text file with text content of the of the audio stream speech content.

<FIG> is an illustrative block diagram showing an audio processing system, in accordance with some embodiments of the disclosure. In <FIG>, audio processing system is configured as audio processing system <NUM> in accordance with some embodiments of the disclosure. In an embodiment, a part or the entirety of system <NUM> may be configured as either of systems <NUM>, <NUM>, of <FIG>, <FIG>, respectively. Although <FIG> shows a certain number of components, in various examples, system <NUM> may include fewer than the illustrated number of components and/or multiples of one or more of the illustrated number of components.

System <NUM> is shown to include a server <NUM>, a computing device <NUM>, an audio stream processor <NUM>, an ASR services <NUM>, and a communication network <NUM>. Each of the server <NUM>, computing device <NUM>, audio stream processor <NUM>, and ASR services <NUM> is communicatively coupled to communication network <NUM>. In an embodiment, server <NUM> may be configured as one or more network elements in communication network <NUM>. In an embodiment, server <NUM> resides externally to communication network <NUM>, as shown in <FIG>.

In some embodiments, computing device <NUM> may be configured as all or part of one of the audio stream sources <NUM> of <FIG> or one of the audio stream sources <NUM> of <FIG>. Alternatively, any of the sources <NUM> may form a part of computing device <NUM>. In an embodiment, server <NUM> may be configured as audio stream processor <NUM> or audio stream processor <NUM>.

Communication network <NUM> may comprise one or more network systems, such as, without limitation, an Internet, LAN, WiFi or other network systems suitable for audio processing applications. In some embodiments, system <NUM> excludes server <NUM> and functionality that would otherwise be implemented by server <NUM> is instead implemented by other components of system <NUM>, such as one or more components of communication network <NUM>. In still other embodiments, server <NUM> works in conjunction with one or more components of communication network <NUM> to implement certain functionality described herein in a distributed or cooperative manner. Similarly, in some embodiments, system <NUM> excludes audio stream processor <NUM> and functionality that would otherwise be implemented by audio stream processor <NUM> is instead implemented by other components of system <NUM>, such as one or more components of communication network <NUM> or server <NUM>. In still other embodiments, audio stream processor <NUM> works in conjunction with one or more components of communication network <NUM> or server <NUM> to implement certain functionality described herein in a distributed or cooperative manner.

Server <NUM> includes control circuitry <NUM> and server interface <NUM>, and control circuitry <NUM> includes storage <NUM> and processing circuitry <NUM>. Computing device <NUM>, which may be a personal computer, a laptop computer, a tablet computer, a smartphone, entertainment equipment, or any other type of computing device, includes control circuitry <NUM>, speaker <NUM>, display <NUM>, hardware interface <NUM>, and computing device interface <NUM>. Control circuitry <NUM> includes storage <NUM> and processing circuitry <NUM>. Control circuitry <NUM> and/or <NUM> may be based on any suitable processing circuitry such as processing circuitry <NUM> and/or <NUM>. As referred to herein, processing circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores). In some embodiments, processing circuitry may be distributed across multiple separate processors, for example, multiple of the same type of processors (e.g., two Intel Core i9 processors) or multiple different processors (e.g., an Intel Core i7 processor and an Intel Core i9 processor). In some embodiments, control circuitry <NUM> and/or control circuitry <NUM> are configured to implement an audio processing system, such as system <NUM> or system <NUM>, or parts thereof, such as audio stream processor <NUM> and audio stream processor <NUM>, and/or any plugins thereof, each of which is described above in connection with <FIG>, <FIG>, respectively.

In some embodiments, audio stream processor <NUM> may be configured as audio stream processor <NUM> or audio stream processor <NUM>, of <FIG> and <FIG>, respectively. In some embodiments where audio stream processor <NUM> is standalone, audio stream processor <NUM> may be directly communicatively coupled to computing device <NUM> and/or server <NUM>. In some embodiments where audio stream processor <NUM> is standalone, audio stream processor <NUM> may be indirectly communicatively coupled to computing device <NUM> and/or server <NUM>, through communication network <NUM>. Similarly, audio stream processor <NUM> may be directly communicatively coupled to ASR services <NUM> or indirectly coupled to ASR services <NUM> through communication network <NUM>. In some embodiments, ASR services <NUM> is analogous to ASR services <NUM> of <FIG> or ASR services <NUM> of <FIG>.

In some embodiments, while not shown in <FIG>, audio stream processor <NUM>, as a standalone unit, may be configured with similar components as server <NUM> and/or computing device <NUM>. For example, audio stream processor <NUM> may have a control circuitry analogous to control circuitry <NUM>, a storage device analogous to storage <NUM> and a processing circuit analogous to processing circuit <NUM>, and an interface analogous to interface <NUM>. For example, storage <NUM> and storage <NUM> (<FIG>) and/or storage <NUM> (<FIG>) may be analogously configured. Processing circuit <NUM> and audio signal processor <NUM> (<FIG>) and/or audio signal processor <NUM> (<FIG>) may be analogously configured; and interface <NUM> and audio stream interface <NUM> (<FIG>) and/or audio stream interface <NUM> (<FIG>) may be analogously configured. In some embodiments, buffer processor <NUM> of <FIG> may be configured as processing circuit <NUM> and buffer <NUM> may be configured as storage <NUM>. In a non-limiting operational example, one or more of the server interface <NUM>, interface of the audio stream processor <NUM>, hardware interface <NUM>, and computing device interace <NUM> may send N number of audio streams to buffer <NUM> (<FIG>).

Each of storage <NUM>, storage <NUM>, and/or storages of other components of system <NUM> (e.g., storages <NUM> and <NUM> and/or the like) may be an electronic storage device. As referred to herein, the phrase "electronic storage device" or "storage device" should be understood to mean any device for storing electronic data, computer software, or firmware, such as random-access memory, read-only memory, hard drives, optical drives, digital video disc (DVD) recorders, compact disc (CD) recorders, BLU-RAY disc (BD) recorders, BLU-RAY 3D disc recorders, digital video recorders (DVRs, sometimes called personal video recorders, or PVRs), solid state devices, quantum storage devices, gaming consoles, gaming media, or any other suitable fixed or removable storage devices, and/or any combination of the same. Each of storage <NUM>, storage <NUM>, and/or storages of other components of system <NUM> may be used to store various types of content, metadata, and or other types of data. Non-volatile memory may also be used (e.g., to launch a boot-up routine and other instructions). Cloud-based storage may be used to supplement storages <NUM>, <NUM> or instead of storages <NUM>, <NUM>. In some embodiments, control circuitry <NUM> and/or <NUM> executes instructions for an application stored in memory (e.g., storage <NUM> and/or <NUM>). Specifically, control circuitry <NUM> and/or <NUM> may be instructed by the application to perform the functions discussed herein. In some implementations, any action performed by control circuitry <NUM> and/or <NUM> may be based on instructions received from the application. For example, the application may be implemented as software or a set of executable instructions that may be stored in storage <NUM> and/or <NUM> and executed by control circuitry <NUM> and/or <NUM>. In some embodiments, the application may be a client/server application where only a client application resides on computing device <NUM>, and a server application resides on server <NUM>.

The application may be implemented using any suitable architecture. For example, it may be a stand-alone application wholly implemented on computing device <NUM>. In such an approach, instructions for the application are stored locally (e.g., in storage <NUM>), and data for use by the application is downloaded on a periodic basis (e.g., from an out-of-band feed, from an Internet resource, or using another suitable approach). Control circuitry <NUM> may retrieve instructions for the application from storage <NUM> and process the instructions to perform the functionality described herein. Based on the processed instructions, control circuitry <NUM> may determine what action to perform when input is received from interface <NUM>.

In client/server-based embodiments, control circuitry <NUM> may include communication circuitry suitable for communicating with an application server (e.g., server <NUM>) or other networks or servers. The instructions for carrying out the functionality described herein may be stored on the application server. Communication circuitry may include a cable modem, an Ethernet card, or a wireless modem for communication with other equipment, or any other suitable communication circuitry. Such communication may involve the Internet or any other suitable communication networks or paths (e.g., communication network <NUM>). In another example of a client/server-based application, control circuitry <NUM> runs a web browser that interprets web pages provided by a remote server (e.g., server <NUM>). For example, the remote server may store the instructions for the application in a storage device. The remote server may process the stored instructions using circuitry (e.g., control circuitry <NUM>) and/or generate displays. Computing device <NUM> may receive the displays generated by the remote server and may display the content of the displays locally via display <NUM>. This way, the processing of the instructions is performed remotely (e.g., by server <NUM>) while the resulting displays, such as the display windows described elsewhere herein, are provided locally on computing device <NUM>. Computing device <NUM> may receive inputs from the user via computing device interface <NUM> and transmit those inputs to the remote server for processing and generating the corresponding displays.

A user may send instructions to control circuitry <NUM> and/or <NUM> using user input interface <NUM>. Interface <NUM> may be any suitable user interface, such as a remote control, trackball, keypad, keyboard, touchscreen, touchpad, stylus input, joystick, voice recognition interface, a gaming controller, or other user input interfaces. In an embodiment, interface <NUM> is configured as at least a part of an audio source, such as audio source <NUM> or audio source <NUM>. Interface <NUM> may be integrated with or combined with display <NUM>, which may be a monitor, a television, a liquid crystal display (LCD), electronic ink display, or any other equipment suitable for displaying visual images. In an embodiment, display <NUM> may be a part of an audio source, such as audio source <NUM> or audio source <NUM>.

Server <NUM> and computing device <NUM> may transmit and receive content and data via interfaces <NUM> and <NUM>. For instance, interface <NUM> may include a communication port configured to receive audio streams via communication network <NUM>, and/or to communicate payload and text file information to and from ASR services <NUM>. Control circuitry <NUM>, <NUM> may be used to send and receive commands, requests, and other suitable data using interfaces <NUM>, <NUM>, respectively.

In some embodiments, a part or the entirety of system <NUM> carries out the steps and determinations of the flowcharts of <FIG>, in addition to other steps and determinations not shown or discussed herein.

Claim 1:
A method (<NUM>) of processing audio streams, the method comprising:
receiving (<NUM>) N number of audio streams during a predetermined time window, each audio stream received from an independent source and including speech content, wherein N is a plurality;
generating (<NUM>) N-<NUM> number of audio stream delineation indicators;
concatenating (<NUM>) a set of the audio streams to generate a concatenated audio stream;
inserting (<NUM>) an audio stream delineation indicator between every two adjacent audio streams of the concatenated audio stream to generate a single audio stream payload, each audio stream delineation indicator delineating a beginning of a next audio stream and an end of a preceding audio stream in the concatenated audio stream;
transmitting (<NUM>) the single audio stream payload from a buffer for transcription of the speech content of the set of audio streams to text content, to obtain a transcription of each of the speech content of the set of audio streams; and
in response (<NUM>) to transmitting the single audio stream payload for transcription, receiving a text file including the text content delineated with the audio stream delineation indicators.