Patent Description:
In audio data processing, it is often desirable to identify a speaker in the audio data. Speaker identification is the labeling of an extent of audio data (recording or stream) with the identity of a specific person whose speech is contained within the audio. Conventionally, speaker identification can require a large amount of processing resources due to the amount of audio data typically being analyzed to make the identification determination. <CIT> describes a matching system that receives probe audio samples for comparison to references of a data store. Comparisons are generated between a first segment of a probe audio sample and corresponding time segments of a plurality of reference audio samples to identify a plurality of sufficiently matching reference audio samples based upon a first set of consistency scores. Matching references are retained, unless they meet a score threshold. Comparisons are continually generated with a second segment of the probe audio sample and corresponding time segments of the sufficiently matching reference audio samples to generate a second set of consistency scores. The retained results are outputted based on the first and second set of consistency scores. <CIT> discloses a system and method of identifying a sound signal, by receiving a sound signal; converting the sound signal into an audio signal; identifying a first keypoint in the audio signal; calculating first keypoint data from the audio signal, the first keypoint data being associated with the first keypoint; comparing the first keypoint data to at least one stored keypoint data, the at least one stored keypoint data each being associated with at least one audio recording; determining a first similarity value for each audio recording based on said comparison; assigning a likeness score to each audio recording whose first simlarity value is within a threshold range; identifying a second keypoint in the audio signal; calculating second keypoint data from the audio signal, the second keypoint data being associated with the second keypoint; comparing the second keypoint data to the at least one stored keypoint data; determining a second similarity value for each audio recording based on said comparison of the second keypoint data; updating each likeness score based on the second similarity value; identifying at least one of the audio recordings as a candidate audio recording, based on the updated likeness score; and providing the identity of the identified candidate audio recording as the identity of the received sound signal. <CIT> describes a method for performing speaker recognition that comprises: estimating respective uncertainties of acoustic coverage of respective speech utterance(s) by first and second speakers, the acoustic coverage representing respective sounds used by the speakers when speaking; representing the respective uncertainties of acoustic coverage in a manner that allows for efficient memory usage by discarding dependencies between uncertainties of different sounds for the speakers; representing the respective uncertainties of acoustic coverage in a manner that allows for efficient computation by representing an inverse of the respective uncertainties of acoustic coverage and then discarding the dependencies between the uncertainties of different sounds for the speakers; and computing a score between the speech utterance(s) by the speakers in a manner that leverages the respective uncertainties of the acoustic coverage during the comparison, the score being indicative of a likelihood that the speakers are the same speaker.

According to one aspect, a system for identifying audio data is provided. The system includes a feature extraction module for receiving an unknown input audio data stream from a source to be identified and dividing the unknown input audio data stream into a plurality of segments of unknown input audio data. A similarity module receives the plurality of segments of the unknown input audio data and receives a known audio data stream from a known source, the known audio data stream being divided into a plurality of segments of known audio data. The similarity module performs comparisons between the segments of unknown input audio data and respective segments of known audio data and generates a respective plurality of similarity values representative of similarity between the segments of the comparisons, the comparisons being performed serially. The similarity module terminates the comparisons if the similarity values indicate insufficient similarity between the segments of the comparisons, prior to completing comparisons for all segments of the unknown input audio data.

In some exemplary embodiments, each segment of unknown audio data comprises an unknown data vector comprising a plurality of data values associated with the segment of unknown input audio data; and each segment of known audio data comprises a known data vector comprising a plurality of data values associate with the segment of known audio data. The similarity module, in generating each similarity value, can compute a vector distance between the unknown data vector and the known data vector of the segments of the comparison being performed.

In some exemplary embodiments, the feature extraction module comprises a deep neural network.

In some exemplary embodiments, the similarity module performs a probabilistic linear discriminant analysis (PLDA) in generating the plurality of similarity values.

In some exemplary embodiments, extents of each segment are defined by a time duration. In some exemplary embodiments, extents of each segment are defined by a quantity of data in the segment.

According to another aspect, a method for identifying audio data is provided. The method includes: (i) receiving, at a feature extraction module, an unknown input audio data stream from a source to be identified and dividing the unknown input audio data stream into a plurality of segments of unknown input audio data; and (ii) receiving, at a similarity module, the plurality of segments of the unknown input audio data and receiving, at the similarity module, a known audio data stream from a known source, the known audio data stream being divided into a plurality of segments of known audio data, the similarity module performing comparisons between the segments of unknown input audio data and respective segments of known audio data and generating a respective plurality of similarity values representative of similarity between the segments of the comparisons, the comparisons being performed serially, the similarity module terminating the comparisons if the similarity values indicate insufficient similarity between the segments of the comparisons, prior to completing comparisons for all segments of the unknown input audio data.

In some exemplary embodiments, each segment of unknown audio data comprises an unknown data vector comprising a plurality of data values associated with the segment of unknown input audio data; and each segment of known audio data comprises a known data vector comprising a plurality of data values associate with the segment of known audio data. In some exemplary embodiments, the similarity module, in generating each similarity value, computes a vector distance between the unknown data vector and the known data vector of the segments of the comparison being performed.

The present disclosure is further described in the detailed description which follows, in reference to the noted plurality of drawings by way of non-limiting examples of embodiments of the present disclosure, in which like reference numerals represent similar parts throughout the several views of the drawings.

Speaker identification (SID) is the labeling of an extent of audio data (recording or stream) with the identity of a specific target person whose speech is contained within the audio. Approaches to SID typically include an offline component (training) and a runtime component (recognition). Current SID systems perform recognition by computing a low-dimensional feature vector representing the entire recording duration and comparing the feature vector, using a learned similarity function, to a previously computed feature vector from a known example of the target speaker, which is generated during training, and which is commonly known as the enrollment vector. The similarity between the two vectors is reported as the probability of detection, which is compared to a user-supplied threshold, the result of this comparison being used to determine whether to label or not label the recording with the enrollment speaker's identity. Some modern approaches use deep neural networks to extract data-driven features that enable comparison of new audio to audio from known speakers. The approach of the present disclosure reduces the amount of audio data needed at runtime to make a correct non-target designation, which directly reduces the processing cost and resources in terms of computation cycles.

According to the present disclosure, this improvement is achieved by an incremental computing approach that "fails fast" in asserting that a recording does not contain the identity of a target speaker before all audio is processed. Under the "fail fast" approach of the present disclosure, rather than use the entire audio recording, overall computation, which is proportional to the amount of audio processed, is reduced by computing the feature vector on increasing initial subsets of the unknown audio. The similarity function is applied to these vectors, and the trajectory of the distance function output is observed. If subsequent segments of audio fail to increase the similarity to the target speaker, the audio is rejected as being that of the known speaker, and processing is stopped early, i.e., before all of the audio data is processed. Typical conventional systems will make an identification decision only after computing over the entire audio file. The approach of the present disclosure stops processing early if an additional portion or segment or "chunk" of audio fails to reduce the system comparison function distance to the target speaker. The "fail fast" approach of the present disclosure can be applied to any speaker identification system using a feature-extraction/similarity-function paradigm.

Thus, the approach of the present disclosure is an improvement to deep-learning-based approaches to speaker identification. Audio of variable length is converted, via a deep neural network, to a low-dimensional vector of numbers with the property that multiple audio samples from the same speaker will be close in the resulting vector space. The feature vector is extracted from one of the internal network layers, commonly described as a bottleneck layer, an example being shown as layer l<NUM> in <FIG>, described in detail below. The output values of this layer become the feature vector for given input audio. These feature vectors are described as x-vectors, based on the work by Snyder et al, (cited below), which describes a successful neural network architecture applied to speaker identification. <FIG> is an illustration of Snyder's architecture, with the bottleneck layer / x-vector described as the "speaker embedding.

According to the disclosure, the identity of the target speaker is provided by an additional recording which is converted to the low-dimensional space, i.e., compute an enrollment x-vector. To speed up processing, the approach of the disclosure computes x-vectors on incremental portions of the unknown audio. The distance between the unknown recordings and enrollment x-vector is computed using, for example, a Probabilistic Linear Discriminative Analysis (PLDA). To reject non-target speakers, the trajectory of the PLDA distance is followed, such that, if the subsequent segments of audio fail to reduce the distance to the target speaker, the audio is rejected, and processing is stopped early.

Thus, according to the present disclosure, the use of a trajectory-based approach to reduce the expected computation for non-target speakers is implemented in a speaker identification system. By computing x-vectors (embeddings) on sequential segments of audio, the approach of the disclosure, processing can be stopped early when the distance fails to converge on the target speaker.

<FIG> includes a schematic functional block diagram of a speaker identification system <NUM>, according to some exemplary embodiments. Referring to <FIG>, system <NUM> takes in audio data <NUM> from an unknown speaker of any length, which, in some particular exemplary embodiments, is in the form of a. System <NUM> also takes in a model or model data <NUM> for a known speaker, referred to herein as known speaker "Bob. " Model <NUM> represents what makes the known speaker's voice distinct from others' voices. Output <NUM> from system <NUM> is a yes or no answer to the question as to whether audio data <NUM> from the unknown speaker contains the voice of the known speaker. That is, a yes output identifies the speaker in audio data <NUM> as being the known speaker of model <NUM>.

According to some exemplary embodiments, the approach to speaker identification is carried out in two stages, namely, a feature extraction stage and a similarity function stage. <FIG> includes a schematic functional block diagram illustrating speaker identification system <NUM> of <FIG>, according to some exemplary embodiments. Referring to <FIG>, speaker identification system <NUM> includes feature extraction module <NUM> and similarity function module <NUM>. Audio input data <NUM> is provided to feature extraction module <NUM>, which outputs a feature vector <NUM>, denoted X, which in some exemplary embodiments is a fixed-length list of decimal values xi. In some exemplary embodiments, feature X-vector <NUM><NUM> decimal values x<NUM>, x<NUM>,. X-vector <NUM> is then input to similarity function module <NUM>, which measures the distance between feature X-vector <NUM> and known model data <NUM>-vector <NUM>. The "fail fast" approach of the present disclosure enhances any existing speaker identification system using the framework illustrated in <FIG>.

It will be understood that either or both of feature extraction module <NUM> and similarity function module <NUM> include all of the processing capabilities required to carry out their individual functions and the overall functions of speaker identification system <NUM>, as described herein in detail. These processing capabilities can be implemented in either or both modules <NUM>, <NUM>, and can include for example, one or more dedicated processors, memories, input/output devices, interconnection devices, and any other required devices or subsystems. Alternatively, these modules <NUM>, <NUM> and system <NUM> can be implemented on a general purpose computer executing instructions to implement the technology described herein.

Feature Extraction module <NUM> takes audio <NUM> as input, and outputs a fixed-dimensional feature vector <NUM>, i.e., X-vector as shown in <FIG>. By fixed-dimensional, what is meant is that audio of any length (5sec, 10sec, 60sed) is transformed into a vector made up of the same number of decimal values (typically <NUM>-<NUM>).

Similarity Module takes two vectors, i.e., unknown audio X-vector <NUM> and known model audio M-vector <NUM> as input and outputs a single numeric value that captures the "closeness" of the vectors. For example, a similarity function can be defined as: <MAT> The intuition to this function is that the larger the value of S, the "closer", i.e., more similar, X and Y are. Two common geometric similarity functions are the Euclidian distance and Cosine similarity. The distance function is turned into a similarity function by subtracting from <NUM>. The Euclidean distance in two dimensions is another name for the Pythagorean Theorem. <MAT> <MAT> <MAT>.

The Cosine similarity captures the angle between two vectors and is a common metric used in high dimensions (greater than <NUM>). <FIG> is a curve illustrating an example of the Euclidean similarity function, and <FIG> is a curve illustrating an example of the Cosine similarity function. The Cosine similarity quantifies the angle between the two X vectors, starting at the origin, (<NUM>,<NUM>), and terminating at the points marked X in <FIG>, and the vector terminating at the letter B, which represents the speaker model. These metrics are illustrated in two dimensions but have well-defined constructions in higher dimensions.

Probability functions, which return a value of <NUM> to <NUM>, are also an intuitive set of similarity functions. If there is a probability that X and Y are the same, then the higher the probability S= P(X,Y), the "closer" or more similar X and Y are. In some exemplary embodiments, the similarity function <NUM> is Probabilistic Linear Discriminant Analysis (PLDA). PLDA is a probability-based metric that is a log-likelihood ratio, a comparison of two probabilities: <MAT> PLDA and other log-likelihood ratios range from -∞ to ∞, with <NUM> being the point at which we are completely unsure whether the unknown audio is Bob. Positive values indicate that it is more likely than not to be Bob, and negative values indicate that it is more likely than not that the unknown speaker is not Bob. This fits the requirement for a similarity metric in that larger values of S mean "closer" or "more similar.

According to some exemplary embodiments, feature extraction module <NUM> uses the neural network model developed at Johns Hopkins University, which is commonly referred to as an x-vector extractor. This model is well-known in the art, and is described in, for example, D. Snyder, et al. , "X-vectors: Robust DNN embeddings for speaker recognition," in Proc. ICASSP, <NUM>. The x-vector neural network is an extraction function that satisfies the condition for equation <NUM> but is internally implemented with a neural network. <FIG> includes a schematic functional block diagram which illustrates the x-vector neural network structure. <FIG> is taken from <FIG> of Zhu, et al. , "Self-attentive Speaker Embeddings for Text-Independent Speaker Verification," INTERSPEECH <NUM>. Referring to <FIG>, the "speaker embedding" refers to the output feature vector, and X (x<NUM>, x<NUM>,. , xT) refers to the audio feature input. The network of <FIG> is a typical feed-forward network, in that the inputs to each layer (denoted l<NUM> through l<NUM>) are multiplied with a weight matrix, followed by the application of a nonlinear function to each output dimension. The output dimensionality for a layer is represented by the number of nodes in that layer. Each node has a weight associated with each input dimension, so each layer can be represented by an M × N dimensional weight matrix, where M is the number of inputs and N the number of outputs. The outputs of layer i are in turn used as the inputs to layer i+<NUM>. In <FIG>, the "speaker embedding" is the output of layer l<NUM>.

As illustrated below in detail, adding additional audio data into feature extraction module <NUM>, without re-initializing the network, gives a better estimate of feature vector <NUM>. It is related to the effect in statistical estimation that more data points give a more accurate measurement estimate.

The "fail fast" approach to using feature extraction <NUM> and similarity function <NUM> according to the exemplary embodiments, as described in connection with <FIG>, inputs unknown audio <NUM> and known target speaker model audio <NUM> from a known speaker ("Bob"), and outputs a yes/no answer as to whether the known speaker's voice is in the unknown audio.

<FIG> includes a schematic logical flow diagram of the logical flow of the approach of the disclosure, according to some exemplary embodiments. Referring to <FIG>, in step S100, unknown input audio A <NUM> is broken up or divided or segmented into multiple adjacent data "chunks" or segments c. In some exemplary embodiments, the segments c can be defined by a certain predetermined time duration. For example, each segment c can have a time duration of <NUM> seconds or <NUM> seconds or <NUM> seconds, or other time duration. In some embodiments, a quantity of N segments ci are generated as c<NUM>, c<NUM>,.

In step S102, the index i of segments c is initialized to a value of <NUM>. Next, in step S104, a determination is made as to whether the last segment, i.e., cN, has been reached. If not, in step S105, a feature vector X[i] is generated for the current segment ci of audio data by feature extraction module <NUM>. Next, in step S106, the similarity score S[i] between X[i] and target speaker model M-vector, S(X[i],M), is computed. In step <NUM>, a determination is made as to whether the similarity for the current segment S[i] has increased over that of the previous segment S[i-<NUM>]. If so, then processing continues with the next segment, since it cannot yet be concluded that the unknown audio does not contain the voice of the known target speaker. To that end, the index i is incremented in step S110, and flow returns to the top of the loop at step S104.

If the determination in step S108 is that the current similarity has not increased over the previous similarity, then a determination is made in step S112 as to whether the current similarity score is positive, recalling that a positive similarity score indicates that it is more likely than not that the unknown audio includes the target speaker's voice. If the current similarity score is positive, then a YES decision is issued in step S116. That is, it is determined that the unknown audio includes the voice of the target speaker. However, if the similarity score is not positive, then a NO decision is issued in step S114. That is, it is determined that the unknown audio does not include the voice of the target speaker.

Hence, as long as the similarity score continues to increase from segment to segment, the process loops to check each segments until N segments have been processed. At that point, the determination in step S104, after the index i has been incremented in step S110, will be that index i has exceeded N. The index i will then be decremented in step S118 back to that of the last audio segment processed. Flow will then continue to step S112, where a determination is made as to whether the current, i.e., last, similarity score is positive. If the current similarity score is positive, then a YES decision is issued in step S116. However, if the similarity score is not positive, then a NO decision is issued in step S114.

As an alternative to the binary YES/NO decisions, some embodiments can interpret the final output, decision, or score as a probability. Speaker identification is often treated as a detection problem. The binary decision represents the application of a specific threshold. While the threshold of <NUM> is a natural one, given its probabilistic interpretation as a likelihood ratio test between two competing hypotheses, by outputting the raw similarity score, the system allows for external components to make their own thresholding decisions.

<FIG> include schematic block diagrams which illustrate a positive identification of the target speaker in the unknown audio, and <FIG> include schematic block diagrams which illustrate a negative identification of the target speaker in the unknown audio. Referring to <FIG>, and with continuing reference to the logical flow diagram of <FIG>, unknown audio <NUM> is split into three segments or chunks; i.e., for the illustrations of <FIG>, and of <FIG>, N = <NUM>. <FIG> illustrates the processing of the first segment, <FIG> illustrates the processing of the second segment, and <FIG> illustrates the processing of the third segment. The similarity score for the first segment in <FIG> is negative, i.e., -<NUM>. The similarity score for the second segment in <FIG> is -<NUM>, which is greater than -<NUM>; therefore, processing continues to the third segment. The similarity score for the third segment in <FIG> is <NUM>, which is greater than the score for the second segment. Processing ends here because the third, i.e., Nth, segment has been processed. Because <NUM> is a positive number, a YES decision <NUM> is issued. It should be noted that, in this scenario, if the score for the third segment were negative, for example, -<NUM>, a NO decision would have been issued, even though the score increased with each segment.

Continuing to refer to <FIG>, it should be noted that no decision <NUM> was issued until all of the data had been processed. That is, all segments were processed to obtain a similarity score, and then the final positive YES decision was issued after all processing was completed. <FIG> illustrate processing according to which a negative NO decision is reached. In this scenario, the "fail fast" technique of the present disclosure is implemented, resulting in only two of the three segments needing to be processed. That is, the NO decision is reached after only two segments are processed, resulting in reduced use of processing resources.

Referring to <FIG> and <FIG>, the similarity score for the first segment is -<NUM>. The similarity score for the second segment is -<NUM>. Since the score decreased from the first to the second segment, loop processing stopped at step S108, and the determination as to whether the score is positive was made in step S112. Since the score for the second segment was negative, the NO decision was issued without all segments needing to be processed.

Hence, the approach of the invention saves considerable processing cost and time because of its ability to eliminate the processing of large amounts of audio data, in the case of a non-identification of a target speaker in unknown audio. In particular, reducing usage of feature extraction module <NUM> is beneficial, especially since, under operation, that is where the bulk of processing time and cost is expended.

<FIG> includes a graph illustrating vectors and trajectory for the two scenarios illustrated in <FIG> and <FIG>, according to some exemplary embodiments. Referring to <FIG>, the trajectory of the scenario of <FIG> is shown in the path on the left, the trajectory of the scenario of <FIG> is shown in the path on the right, and the known model vector M is labeled B (Bob), which is shown as a point in the two-dimensional graph. In general, <FIG> illustrates that the system of the disclosure, with each chunk of additional data, is refining the representation with more data. For the "YES" scenario of <FIG>, the feature vector X gets closer to target feature vector B which each chunk. For the "NO" scenario of <FIG>, more data moves the vector X away from B, so processing can be stopped early.

Whereas many alterations and modifications of the disclosure will become apparent to a person of ordinary skill in the art after having read the foregoing description, it is to be understood that the particular embodiments shown and described by way of illustration are in no way intended to be considered limiting. Further, the subject matter has been described with reference to particular embodiments, but variations within the scope of the disclosure will occur to those skilled in the art. It is noted that the foregoing examples have been provided merely for the purpose of explanation and are in no way to be construed as limiting of the present disclosure.

Claim 1:
A system for identifying audio data, comprising:
a feature extraction module (<NUM>) for receiving an unknown input audio data stream from a source to be identified and dividing the unknown input audio data stream into a plurality of segments of unknown input audio data; and
a similarity module (<NUM>) for receiving the plurality of segments of the unknown input audio data and for receiving a known audio data stream from a known source, the known audio data stream being divided into a plurality of segments of known audio data, the similarity module (<NUM>) performing comparisons between the segments of unknown input audio data and respective segments of known audio data and generating a respective plurality of similarity values representative of similarity between the segments of the comparisons, the comparisons being performed serially, the similarity module (<NUM>) terminating the comparisons if the similarity values indicate insufficient similarity between the segments of the comparisons, prior to completing comparisons for all segments of the unknown input audio data.