Feature extraction for anonymized speech recognition

Various of the disclosed embodiments relate to systems and methods for extracting audio information, e.g. a textual description of speech, from a speech recording while retaining the anonymity of the speaker. In certain embodiments, a third party may perform various aspects of the anonymization and speech processing. Certain embodiments facilitate anonymization in compliance with various legislative requirements even when third parties are involved.

FIELD OF THE INVENTION

Various of the disclosed embodiments relate to systems and methods for extracting user information from a speech recording while retaining the anonymity of the user, sometimes while working with a third party in compliance with legislative requirements.

BACKGROUND

Various regulatory agencies are working to protect the privacy of online users, particularly children, by imposing certain guidelines regarding the acquisition and storage of speech data. For example, the FTC's recent implementation of the Children's Online Privacy and Protection Act (COPPA) concerns itself in part with children under 13 years old. Changes to the Act may require that audio files that contain a child's voice be considered “personal information” deserving of special protection. Companies which provide child-oriented applications, such as entertainment applications, may need to gather consent from parents before letting children use their service, particularly where the service acquires and processes personal information in the course of entertaining the child audience.

Legislation, such as COPPA, may require that consent be acquired in one of two categories: “rigorous” and “mild”. Rigorous forms of consent may include, e.g., processing a credit card payment, maintaining a call center for parents to contact, requiring a parent to print, sign, and mail a physical approval form, etc. Mild forms of consent may include, e.g., “email plus” activities. COPPA's “email plus” method allows a company to request (in the direct notice to the parent) that the parent provide consent in an email message. For various business and growth reasons, including potentially offering customers a free trial to their services, many organizations may prefer to gather consent via a mild method, such as the “email plus” method.

Legislation such as COPPA is also concerned with the “disclosure” of personal information, and may compel companies to use the more rigorous forms of consent when personal information is shared with third parties. However, there is an important exception (16 C.F.R. 312.2. “disclosure”0.1) that allows companies to implement consent via “email plus” as long as a third party provider is only using the information for supporting the internal operations of the company's online service.

Accordingly, there exists a need for systems and methods which comply with the legislative changes, while providing the functionality necessary to perform certain interactive operations.

SUMMARY

Certain embodiments contemplate a computer-implemented method for speech data processing comprising: receiving a raw waveform containing speech data of a user; providing the raw waveform to a third party processing system; directing the third party processing system to perform an operation upon the waveform, the operation making identification of the user from the waveform more difficult; and receiving a textual depiction of speech from the third party system.

In some embodiments, making identification of the user from the waveform more difficult comprises rendering at least one technique for identifying the user from the waveform less effective in identifying the user. In some embodiments, the plurality of features in the waveform to be redacted comprise frequency components. In some embodiments, directing the third party processing system to perform an operation comprises identifying features in the waveform for the third party system to redact. In some embodiments, the identified features comprise frequency components. In some embodiments, identifying features in the waveform comprises providing metadata with the raw waveform to the third party system, the metadata identifying the plurality of features in the waveform to be redacted. In some embodiments, the third party processing system comprises a speech processing software module. In some embodiments, the third party processing system comprises a server computer. In some embodiments, the textual depiction comprises a string of characters. In some embodiments, the textual depiction includes a Natural Language Processing (NLP) description.

Certain embodiments contemplate a computer-implemented method for processing speech data comprising: receiving a raw waveform containing speech data of a user; generating an anonymized intermediate representation of the speech data based on the raw waveform; and determining a plurality of textual speech data based on the anonymized intermediate representation.

In some embodiments, generating an anonymized intermediate representation of the speech data comprises transmitting the anonymized intermediate representation to a processing system operated by a third party. In some embodiments, generating an anonymized intermediate representation comprises generating a spectral decomposition of the raw waveform and removing at least one frequency component. In some embodiments, generating an anonymized intermediate representation comprises generating a feature vector representation of a plurality of frequencies derived from the raw waveform. In some embodiments, determining a plurality of textual speech data comprises generating an NLP description of at least a portion of the waveform.

Certain embodiments contemplate a computer-implemented method for anonymizing speech data comprising: receiving a raw waveform containing speech data of a user; segmenting the raw waveform into a plurality of intervals; calculating a frequency domain representation of each interval; applying a post-processing function to smooth the frequency representation; extracting features from at least one of the plurality of intervals; and providing the features for processing by an analysis system.

In some embodiments the method further comprises performing at least one of noise removal, speaker removal, or vocal tract length normalization. In some embodiments, the post-processing function removes at least a portion of speaker identifying data from the waveform. In some embodiments, the post-processing function comprises the addition of certain nonspecific components into the waveform data. In some embodiments, extracting the features comprises analyzing data provided from a speech database.

DETAILED DESCRIPTION

System Overview

Certain of the present embodiments contemplate anonymization methods for software products, e.g., software products for mobile devices aimed at the family entertainment market. Certain of these embodiments may employ cloud-based speech recognition of children and adults as part of the runtime experience offered by the product.

For conversational entertainment, speech recognition (e.g., the identification of words in speech) may be fundamental to the internal operations of the application providing the entertainment. The designer, or implementer, of the application system may desire to use “email plus” consent if the system engages a third party as a speech recognition provider and the information is only used for the limited purpose of speech recognition. By anonymizing user data, in certain embodiments it may not be necessary to acquire “rigorous” consent, or any consent at all, to achieve legislative compliance.

However, it may also be common for third party speech recognition providers to want to capture and maintain large libraries of voice recordings to improve their recognition capabilities and to help develop new speech recognition products. These new products may then be offered to the market at large. Since legislation, such as that described above, may consider an audio file of a child's voice to be “personal information”, the third party speech recognition provider's desire to keep recordings of children's voices for future use may require the application system to first gather consent through one of the “rigorous” methods.

To avoid these difficulties, certain embodiments contemplate systems and methods to recognize, and to perform operations based upon speech without using and/or retaining raw audio files provided by users. Both the application system and the third party may benefit since the application system would not be “disclosing” any personal information to the third party. Without the acquisition and distribution of personal information, legislation, such as COPPA, would not require the more rigorous consent.

Accordingly, certain embodiments recognize what a child is saying in a runtime experience, such as a conversational system wherein the child engages in conversation with a plurality of synthetic characters, without using or storing raw audio files that contain that child's voice.

FIG. 1illustrates a general network topology of certain embodiments of the system. In these embodiments, an application server101may host a service to run an entertainment program, such as a conversational dialogue with a plurality of synthetic characters, on a user device108via a network109a. Application server101may provide assets and instructions to a program running on user device108. Networks109aand109bmay be the same network, such as the Internet, or different networks, such as a local area network and the Internet. Application server101may send data to user device108, so that user device108may execute the dialogue through a web browser, a stand-alone application, etc. In some embodiments, application system101may not be located on a server system across a network109a, but may instead be located on user device108. System tools104may include a plurality of software programs to implement various aspects of the disclosed embodiments and to maintain operation of the server101. User device108may include a mobile phone, table PC, desktop computer, laptop computer, internet connected TV set, etc. The user device108may be located in a kiosk, e.g. at a hospital or nursery, or may be part of an entertainment system in a home, or in a minivan, etc. As used herein, the “application system” may refer to the application server101, speech information database107, application analysis server103, and the application running on user device108, together as a whole, or with reference to only one or more of these elements, depending on the embodiment discussed. In some embodiments, these individual elements may be grouped together into fewer than all the depicted devices. For example, in some embodiments the operations of the application server101and/or database107and/or application analysis server103may be performed on user device108.

As the user112responds in the dialogue, the user interface110may record the user's responses and transmit them to the application server101or directly to the processing analysis server102. Processing analysis server102may be run by a third party separate from the party in control of application server101and the conversation application running locally on user device108. For example, the company running processing analysis server102may be a third party service provider dedicated only to speech recognition. The application server101or application running on user device108may transmit audio files of the user's voice to the processing and analysis server102. The processing may, for example, convert the audio to a Natural Language Processing (NLP) format upon which subsequent analysis may be performed. Following processing, the processing analysis server102may transmit the processed data back to application server101or directly to speech information database107, application analysis server103, or in some embodiments to user device108. Application server101may use the processed information to prepare a response to the user at the user interface110. For example, having determined the words spoken by the user112in response to a query, a program running on application server101or on user device108may determine an appropriate response and deliver the response to the user112via one or more of the synthetic characters depicted on user device108.

An application analysis server103may perform an analysis of the processed audio data, possibly “offline” and not in real-time during the conversational interaction between the user112and synthetic characters. For example, where the processed audio data includes NLP information, the system may apply an Artificial Intelligence engine or Machine Learning method, to extract statistical data. In some embodiments, the NLP or speech recognition information may be used directly by application server101or user device108to inform the operation of the conversational dialogue application running on user device108.

Application Interface

FIG. 2illustrates an example screenshot of a graphical user interface (GUI)200of a virtual environment as may be implemented in certain embodiments. In some embodiments, the GUI200may appear on an interface110of the user device108, such as on a display screen of a mobile phone, or on a touch screen of a mobile phone or of a tablet device. As illustrated in this example, the GUI200may include a first201adepiction of a synthetic character, a second201bdepiction of a synthetic character, a menu bar202having a user graphic204a, a real-time user video204b, and a speech interface203. Through interaction with synthetic characters in the GUI, a user may engage in a conversational dialogue. The user's oral responses and statements may be transmitted by the user device108to application server101.

Menu202may depict speech tools available to the user. Speech interface203may be used to respond to inquiries from synthetic characters201a-b. For example, in some embodiments the user may touch the interface203to activate a microphone to receive their response. In other embodiments the interface203may illuminate or otherwise indicate an active state when the user selects some other input device. In some embodiments, the interface203may illuminate automatically when recording is initiated by the application running on the user device108.

In some embodiments, real-time user video204bdepicts a real-time, or near real-time, image of a user as they use a user device, possibly acquired using a camera in communication with the user device. As indicated inFIG. 2, the depiction of the user may be modified by the system, for example, by overlaying facial hair, wigs, hats, earrings, etc. onto the real-time video image.

Synthetic characters201a-bmay perform a variety of animations, both to indicate that they are speaking as well as to interact with other elements of the virtual environment. Through these interactions synthetic characters201a-bmay encourage the user to participate in dialogue with them and to provide auditory responses through the interface203. The system may transmit these audio responses to application server101for processing and response. In some embodiments the applications operating on application server101are instead operating locally on user device108, and the processing may accordingly take place there.

User Interaction

FIG. 3is a flowchart depicting certain steps in a user interaction process300with the virtual environment as may be implemented in certain embodiments. The process300may be implemented on the user device108or on application server101in certain embodiments. At step301the system may present the user with a virtual environment, such as an environment depicted inFIG. 2. The virtual environment may be divided into a plurality of areas, where each area offers unique conversational interactions between the user and synthetic characters. At step302, the system may receive a user selection for an interactive area. In some instances, the input may comprise a touch or swipe action relative to a graphical icon, but in other instances the input may be an oral response by the user, such as a response to an inquiry from a synthetic character (e.g., “Where would you like to go?”). At step303, the system may present the user with the selected interactive area.

At step304, the system may engage the user in a dialogue sequence based on criteria. The criteria may include previous conversations with the user and a database of statistics generated based on social information or past interactions with the user. The criteria may be statistics that have been generated based on an analysis of content in speech information database107. At step305, the system may determine whether the user wishes to repeat an activity associated with the selected interactive area. For example, a synthetic character may inquire as to the user's preferences. If the user elects, perhaps orally or via tactile input, to pursue the same activity, the system may repeat the activity using the same criteria as previously, or at step306may modify the criteria to reflect the previous conversation history.

Alternatively, if the user does not wish to repeat the activity the system can determine whether the user wishes to quit at step307, again possibly via interaction with a synthetic character. If the user does not wish to quit the system can again determine which interactive area the user wishes to enter at step302.

Anonymization Overview

Some legislation, such as COPPA, considers any audio file that contains child speech to be personal information. Various embodiments therefore propose that audio files be transformed into a different form before they are used by the application system or by a third party to make the third party's own audio models. This different form may not be an audio format and it may not be possible to revert the data back to the original audio waveform. In this manner, certain embodiments may remove personal information from the data as defined by COPPA, because the file would not be one from which all the original audio data, particularly data concerning identity, could be reproduced. Certain embodiments may remove, alter, substitute, or destroy elements of the original, raw audio file to accomplish the anonymization.

FIG. 4illustrates a process flow diagram for a method to generate anonymized speech data in certain embodiments. At step401the system may receive a raw waveform from a user, such as receiving a waveform at the user device108or at the application server101. At step402, the system may then generate an anonymized intermediate representation from the raw waveform. At step403, the system may then use the anonymized intermediate representation for storage and/or analysis and/or for processing by a third party. The anonymized intermediate form may be the form provided to a third party system, such as processing analysis server102.

Anonymization in the Processing Pipeline

Certain embodiments contemplate performing these anonymization operations as part of one or more speech processing operations, e.g. a speech recognition process, such as a speech audio to speech text conversion process. In some embodiments, the speech processing operations may be performed within the same system performing the character animations and entertainment application, e.g., user device108or at the application server101. In certain embodiments, a third party service, dedicated to speech processing may receive the acquired waveforms and return textual or other post-processed versions of the audio file. The system or the third party may desire to use the audio data to build derivative acoustic models for their speech recognition service.

The first phase of a speech recognition process may include the performance of feature extraction, which may involve converting the waveform data into the frequency domain and extracting key frequencies. This may have the effect of transforming the data into a non-audio file format, e.g. a format which is not based on a time-sequence of values. This phase of feature extraction may also destroy most of the frequency content of the recording, making it impossible to exactly reconstruct the original waveform or to reconstruct the original waveform sufficiently to identify an individual by their voice.

Certain embodiments therefore contemplate that the application system, or the third party speech processing service, use the feature extracted frequency information, instead of original audio files, for the purposes of building acoustic models. Such an operation would be in agreement with legislative mandates. Furthermore, in some embodiments the system may specify certain features be retained from the audio file, which facilitate various post-processing operations, e.g. identifying emotional cues, without retaining sufficient information to reconstruct the raw audio file and/or to determine a speaker's identity. In this manner, these embodiments may address privacy concerns over the use of audio data collected from children, while still acquiring sufficient information for a data intensive dialogue-based entertainment system.

Anonymization in the Processing Pipeline—Application System Based Anonymization

In some embodiments, the application system may perform feature extraction first and may then only send the third party the extracted data. The third party data could then be used to make acoustic models both for the system and for the third party's own use. In some situations, this approach may be preferred because the third party never receives any audio files (and hence never receives any personal information) in the first place. In the examples ofFIGS. 5 and 6, and the corresponding embodiments, the operator of the application system may take advantage of the COPPA exception that allows a company to acquire consent via “email plus”, because the third party provider is only using the information for supporting the internal operations of the company's online service or because the third party receives only anonymized data.

FIG. 5illustrates a flow diagram for a process500, involving a third party in part, that analyzes and anonymizes a raw speech waveform, as may be implemented in certain embodiments. At step501the system may receive raw audio data from a user. For example, the system may receive audio at a microphone attached to a user device108or the server system101may receive a waveform of the captured audio from a user device.

At step502, the system may perform anonymization processing of the raw audio waveform. For example, the system may convert the raw waveform to a frequency representation and remove specific frequencies that could be used to identify a user. In some embodiments, the system may convert the result back to a time-based file. Certain methods for anonymizing speech data are discussed in greater detail herein.

At step503, the system may provide the anonymized data to a third party speech processing system. For example, the system may transmit the data to a separately located server, such as processing analysis server102. In some embodiments, the third party may provide an interface, or a local processing module or application, that can be included on application server101. Accordingly, in some embodiments though the data may not physically leave server101, the data may enter into the control of an application operated by a third party.

Steps505and506may be part of third party based operations504which are not in the direct control of the application system(s), or the operator of the application system(s), performing the remaining steps of the process. Third party based operations504may be performed on a separate system, e.g. analysis server102, or as part of a local interface or program provided by the third party. In either instance, in the depicted example the application system receiving the waveform has surrendered control of a portion of the waveform (or a copy of a portion of the waveform) to the third party for third party based operations504.

In some embodiments, the third party and the operator of the application system may have negotiated a protocol for providing anonymized waveforms to the third party system. For example, after converting the waveform to a frequency representation at step502, the application system may not convert the waveform back to a time-based representation, but may provide the frequency version to the third party system. The third party system, anticipating that the waveform will abide by the agreed protocol, may perform operations505,506upon the waveform assuming the waveform comprises frequency-based data. The third party system may also be informed by the application server of the character of the components that were removed from the frequency-based data as part of process step502(e.g., as part of the protocol or by transmitting accompanying metadata).

At step505, the third party system may perform speech processing services, for example, identifying NLP and/or textual elements using a Hidden Markov Model, etc. In some embodiments, the identified NLP and/or textual elements may also be specified in a protocol determined between operators of the application system and operators of the third party system or in metadata provided from the application system with the waveform to the third party system.

At step506, the third party system may copy elements relevant to internal operations of the third party's business. For example, the third party may wish to build derivative acoustic models for their speech recognition service. While these models may be retained for the third party's use, in some embodiments the third party may deliver these models, or other determined information, to analytics server103which may be in communication with, or the same as, application server101. This data may be included as part of analysis server103's study of the speech information database107. In some embodiments, pursuant to the agreed protocol, the third party may adjust the operation of step506as compared to waveforms received from other than the application system. For example, the adjustments may be performed in view of the anonymization processing performed at step502. The anonymization performed by the application system may be of a different character or quality from the operations performed by other systems. In some embodiments, the application system may describe aspects of the processing at step502to the third party system using metadata or specifications in the agreed protocol.

At step507, the system, e.g. application server101, may receive the processed data from the third party and at step508may perform system operations using the processed data from the third party. For example, the processed data may include a textual version of the speech in the raw audio waveform. The application server101, or user device108, may perform natural language processing, artificial intelligence, and/or machine learning techniques upon this textual data. From the analysis, the server101, or user device108, may determine a responsive action, such as, e.g., directing a synthetic character at the user device108.

The system operations performed at508, may include transmitting data to speech information database107or application analysis server103.

Anonymization in the Processing Pipeline—Third Party Based Anonymization

In another disclosed approach the third party may perform the anonymization of the data, e.g. feature extraction. The third party may be expressly contracted to perform anonymization of the application system or may perform the anonymization as part of the third party's operations performed on behalf of all its customers. An agreement may make it clear that any use of the audio files is solely for the purpose of doing the data extraction for the implementer of the application system and for no other purpose. However, in either this embodiment or the embodiment described in relation toFIG. 5, once the data is anonymized, the implementer of the application system could then license or sell the anonymized data to the third party to use however it wishes. As the extracted data contains no personal information the data would not run afoul of the privacy protection legislation. Here, in the example ofFIG. 6the third party would receive personal information when it receives the audio files to do the extraction, but only in its role of providing support for the internal operations of the implementer of the application system.

FIG. 6illustrates a flow diagram for a process600, involving a third party in part, that analyzes and anonymizes a raw speech waveform, as may be implemented in certain embodiments. At step601the system may receive raw audio data from a user. For example, the system may receive audio at a microphone attached to a user device or the server system101may have received the waveform from a user device.

At step602, the system may provide the anonymized data to a third party speech processing system. For example, the system may transmit the data to a separately located server, such as processing analysis server102. In some embodiments, the third party may provide an interface, or a local processing system, that can be included on server101. Accordingly, in some embodiments though the data may not physically leave server101, the data may enter into the control of an application, or module, operated by a third party.

Steps604-606may be part of third party based operations603. Like third party operations504, these steps may be performed on a separate system, e.g. analysis server102, or as part of a local interface or program provided by the third party. In some embodiments, the third party and the operator of the application system may have negotiated a protocol for providing anonymized waveforms to the third party system.

At step604, the system may perform anonymization processing of the raw audio waveform. As discussed in greater detail herein, this processing may occur normally in the third party's processing operations, for example, during conversion to a frequency domain representation and removal of selected frequencies. Certain of the anonymizing operations may be specified by the application system, either previously, via contract or protocol, or by the application system including metadata with the raw audio, specifying which elements are to be redacted. The system may convert the raw waveform to a frequency representation, remove specific frequencies that could be used to identify a user, and convert the result back to a time-based file. Certain methods for anonymizing speech data are discussed in greater detail herein.

At step605, the third party system may perform speech processing services, for example, identifying NLP and/or textual elements using a Hidden Markov Model, etc. As discussed above, in some embodiments, the identified NLP and/or textual elements may also be specified in a protocol determined between operators of the application system and operators of the third party system or in metadata provided from the application system with the waveform to the third party system.

At step606, the third party system may copy elements relevant to internal operations. For example, the third party may wish to build derivative acoustic models for their speech recognition service. While these models may be retained for the third party's use, in some embodiments the third party may deliver these models, or other determined information, to analytics server103which may be in communication with, or the same as, application server101. This data may be included as part of analysis server103's study of the data in speech information database107.

At step607, the system, e.g. application server101, may receive the processed data from the third party and at step608may perform system operations using the processed data from the third party. For example, the processed data may include a textual version of the speech in the raw audio waveform. The application server101, or user device108, may perform natural language processing, or similar analysis, of this textual data. From the analysis, the server101, or user device108, may determine a responsive action, such as, e.g., directing a synthetic character at the user device108.

The system operations performed at step608, may include transmitting data to speech information database107or application analysis server103.

In addition to the operations depicted inFIGS. 5 and 6, the system may transcribe all audio files before making them available to the third party. As part of this process, the system may remove any files, or portions of files, that contain personal information that is uttered verbally by the child/user. Such portions may be recognized using textual analysis, e.g. Bayesian methods as applied to contextual identification. This step may further ensure that the third party will not be exposed to personal information as defined by legislation such as COPPA.

Feature Extraction Overview

In some embodiments, generation of an anonymized intermediate representation at step402of process400may include conversion of the raw audio waveform to a form possessing a plurality of “features”, e.g. frequency components following a Fourier Transform, principal components following a decomposition, etc. These features may then be identified for significance and relevance to post-processing, and those elements relevant to processing retained, while the other elements, including elements which could be used to recover identity information regarding the speaker, are removed. As discussed above, certain speech processing operations may inherently remove certain of the personally identifying qualities of the speech waveform. Accordingly, anonymization methods at step402may build upon these speech processing operations and remove features associated with personal identification that were not previously redacted by the speech processing operations.

In some embodiments, a speech recognition system may be based on a Hidden Markov Model (HMM) where the speech waveform is viewed as a piecewise linear signal. The first stage of a recognition process may be to calculate this sequence of signals for the HMM. This first stage may be called feature extraction.

At a high level, feature extraction may involve taking the raw speech waveform and converting the waveform from the time domain into the frequency domain, then building a feature vector by finding the major frequency peaks at regular intervals. In some embodiments an alternative feature identification method, such as Principal Component Analysis, Support Vector Machines, etc. may be used. The feature vector may then be passed to the HMM stage of the recognition process.

With reference to the example process700ofFIG. 7, at step701the system may receive a raw audio waveform. At step702, the system may optionally apply a pre-emphasis filter function on the original waveform data to emphasize higher frequency components. At step703, the system may segment the audio signal into a sequence of short intervals, using an appropriate windowing function. At step704, for each window of the audio waveform, the system may calculate the window's frequency-domain spectrum by running a Fourier transform on the audio data. At step705, the system may apply a post processing function to smooth the frequency data. For example, the system may use the Mel-frequency cepstral (MFC) method, which involves mapping the Fourier transformed data onto a perceptual scale of pitches called the Mel scale. The system may then take the logarithm and discrete cosine transform (DCT) of the resulting data.

At step706, the system may extract the features as the frequencies of the first most significant peaks (or formants) in the spectrum. If using the MFC, the resulting features may be called Mel-frequency cepstral coefficients (MFCCs). These extracted features may be useful for identifying the spoken words in the audio, but may have little or minimal information regarding the identity of the speaker.

At step707, the extracted features may be optionally stored for later use and/or analysis. For example, at step708, the system may provide the extracted features to a HMM for analysis. Any relevant recognition and analysis method may be used, however. For example, Bayesian methods and analysis may be applied in lieu of, or in conjunction with, the HMM approach.

Application Specific Feature Extraction

FIG. 8illustrates a process flow diagram for a method to extract relevant speech features in certain embodiments. At step801the system may identify analysis relevant criteria. For example, the system may wish to identify emotional cues based on the character of the speech waveform from the user. By consulting database107, the system may determine that certain features are relevant to this determination. At step802the system may determine the features to be extracted as those features which correspond to the relevant criteria. At step803the system may extract those features. Although step803refers to extracting the features that are to be retained, one will readily understand that certain embodiments may perform the converse. For example, rather than identify features to retain, the system may identify features to remove and may remove all but the desired features at step803.

At step804of the depicted example, the system may discard the remaining features or a portion of the remaining features to anonymize the speech data. At step805the system may store the extracted features for further processing possibly by a third party as discussed above.

Additional Applications of Feature Extraction

In certain embodiments, by applying feature extraction to an audio waveform the system may remove frequencies that are not essential to the speech recognition process. Particularly, the system may remove personal information from an audio file. In certain embodiments, the feature extraction may also transform the audio data into a form that is not an audio format, e.g., it cannot be played back by standard audio players such as iTunes, QuickTime Player, or Windows Media Player.

An expert skilled in the field of signal processing may attempt to reverse the feature extraction process by applying an inverse Fourier transform on the feature vector and use a technology like a vocoder to resynthesize an audio waveform. The resulting audio file may be distinguishable as speech. However, because most of the frequencies have been removed from the spectra, the resulting speech may no longer sound anything like the original speaker. The modification may render it prohibitively difficult and/or impossible to uniquely identify the original speaker.

In effect, feature extraction followed by resynthesis may be an information destroying process. Nonetheless, certain embodiments contemplate further steps to further distance the resynthesized audio from the original speaker. For example:

1. Before passing the feature vector to the HMM stage the system may apply a normalization post process to account for the presence of noise, different speakers, different genders, or vocal tract length normalization. The system may apply one or more of these normalization steps to the feature vector in order to make the reconstructed audio sound like a more generic nonspecific human speaker. Application server101may direct the third party system how many times to perform the normalization of a given audio data.

2. In some embodiments, as an alternative to trying to normalize the data, certain embodiments contemplate the addition of new nonspecific components into the data to further mask the profile of the original speaker. For example, white noise frequencies could be added in the frequency domain to distort and obfuscate the identity of the original speaker when transformed back into the time domain. Application server101may direct the third party system regarding what components to add and in what quantity to a given audio data.

3. Some embodiments may employ speaker de-identification that attempts to transform spectral features in such a manner that any reconstructed voice is made to sound intelligible but does not reveal the identity of the speaker. These techniques may be employed in conjunction with the techniques disclosed herein.

Computer System Overview

Various embodiments include various steps and operations, which have been described above. A variety of these steps and operations may be performed by hardware components or may be embodied in machine-executable instructions, which may be used to cause a general-purpose or special-purpose processor programmed with the instructions to perform the steps. Alternatively, the steps may be performed by a combination of hardware, software, and/or firmware. As such,FIG. 9is an example of a computer system900with which various embodiments may be utilized. Various of the disclosed features may be located on computer system900. According to the present example, the computer system includes a bus905, at least one processor910, at least one communication port915, a main memory920, a removable storage media925, a read only memory930, and a mass storage935.

Processor(s)910can be any known processor, such as, but not limited to, an Intel® Itanium® or Itanium 2® processor(s), or AMD® Opteron® or Athlon MP® processor(s), or Motorola® lines of processors. Communication port(s)915can be any of an RS-232 port for use with a modem based dialup connection, a 10/100 Ethernet port, or a Gigabit port using copper or fiber. Communication port(s)915may be chosen depending on a network such a Local Area Network (LAN), Wide Area Network (WAN), or any network to which the computer system900connects.

Main memory920can be Random Access Memory (RAM), or any other dynamic storage device(s) commonly known in the art. Read only memory930can be any static storage device(s) such as Programmable Read Only Memory (PROM) chips for storing static information such as instructions for processor910.

Mass storage935can be used to store information and instructions. For example, hard disks such as the Adaptec® family of SCSI drives, an optical disc, an array of disks such as RAID, such as the Adaptec family of RAID drives, or any other mass storage devices may be used.

Bus905communicatively couples processor(s)910with the other memory, storage and communication blocks. Bus905can be a PCI/PCI-X or SCSI based system bus depending on the storage devices used.

Removable storage media925can be any kind of external hard-drives, floppy drives, IOMEGA@ Zip Drives, Compact Disc-Read Only Memory (CD-ROM), Compact Disc—Re-Writable (CD-RW), Digital Video Disk-Read Only Memory (DVD-ROM).

The components described above are meant to exemplify some types of possibilities. In no way should the aforementioned examples limit the scope of the invention, as they are only exemplary embodiments.

Remarks

While the computer-readable medium is shown in an embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that stores the one or more sets of instructions. The term “computer-readable medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the computer and that cause the computer to perform any one or more of the methodologies of the presently disclosed technique and innovation.

Moreover, while embodiments have been described in the context of fully functioning computers and computer systems, various embodiments are capable of being distributed as a program product in a variety of forms, and the disclosure applies equally regardless of the particular type of computer-readable medium used to actually effect the distribution.

Aspects of the disclosure can be modified, if necessary, to employ the systems, functions, and concepts of the various references described above to provide yet further embodiments of the disclosure.

These and other changes can be made to the disclosure in light of the above Detailed Description. While the above description describes certain embodiments of the disclosure, and describes the best mode contemplated, no matter how detailed the above appears in text, the teachings can be practiced in many ways. Details of the system may vary considerably in its implementation details, while still being encompassed by the subject matter disclosed herein. As noted above, particular terminology used when describing certain features or aspects of the disclosure should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the disclosure with which that terminology is associated. In general, the terms used in the following claims should not be construed to limited the disclosure to the specific embodiments disclosed in the specification, unless the above Detailed Description section explicitly defines such terms. Accordingly, the actual scope of the disclosure encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the disclosure under the claims.