System and method for reconstructing metadata from audio outputs

The disclosed invention provides system and method to reconstruct metadata of audio outputs in which portions of metadata are missing. The system and method of the disclosed invention utilizes characteristics of speakers in audio outputs, voiceprints to identify speakers, and transcripts of the audio outputs to further analyze the audio outputs through machine learning process. The metadata reconstruction system performs operations that include isolating the metadata of the audio output, detecting missing portions of the metadata, detecting characteristics of speakers involved in the audio output, identifying the speakers from the characteristics of the speakers by utilizing voiceprints of speakers, creating a transcript of the audio output, analyzing the transcript by using natural language processing (NLP), annotating the transcript with identified speakers, constructing metadata with the identified speakers and results of the analysis of the transcript, and recombining the constructed metadata with the audio output to produce reconstructed audio output.

BACKGROUND

Audio recordings, which are without metadata or with missing metadata have disadvantages for obtaining information contained in the audio contents compared to audio recordings that include metadata in which contents are metadata-tagged. With various reasons, metadata for audios could be missing. For example, recording of audios may begin before metadata is associated with the voice-captured output (i.e. files). Though spoken words may remain fairly reliable when listening to a conversation, acquiring information of recorded audios could be a challenge without metadata. Audio recordings without metadata may not be able provide information such as when the conversation happened, who was speaking, and applicable sentiment during the conversation.

Traditionally metadata restoration might have been done in isolated circumstances where post-processing of a recording is required. For example, adding metadata about when the file was created, or which hardware or software was used to create the file, or perhaps pulling in respective user-information that created the file may be required. These examples concern the most straightforward techniques to resolve metadata deficiency using standard computer network environment techniques and possibly advanced macros. Other restoration techniques may be used to analyze an audio file and update the metadata accordingly. In these techniques, data-points, which might include bit stream, duration, etc., may be analyzed. However, even in these techniques, data concerning, for example, how many speakers presented, who spoke the most, etc. are less common data restoration examples.

SUMMARY

The disclosed invention provides system and method to reconstruct or restore metadata of audio processing. The system and method utilize technique to extract data from the audio source to construct metadata. Data may include characteristics such as speaker-identity, transcript, sentiment analysis, language spoken, etc. The system and method utilizes characteristics of speakers in audio outputs, voiceprints to identify speakers, and transcripts of the audio outputs to further analyze the audio outputs through machine learning process.

These and others advantages may be provided by, for example, a system for reconstructing metadata of an audio output that is output from one or more audio sources. The metadata reconstruction system is coupled with the one or more audio sources and one or more databases including voiceprints of users. The system includes at least one non-transitory storage medium to store executable instructions, and at least one processor to execute the executable instructions that cause the at least one processor to perform operations to reconstruct the metadata of the audio output. The operations includes isolating the metadata of a recorded conversation of the audio output, detecting missing portions of the metadata from the isolated metadata, detecting characteristics of speakers involved in the audio output, identifying the speakers from the characteristics of the speakers, creating a transcript of the audio output, analyzing the transcript by using natural language processing (NLP), annotating the transcript with identified speakers, constructing metadata with the identified speakers and results of the analysis of the transcript, in which the missing portions of the metadata is replaced with the constructed metadata, and recombining the constructed metadata with the audio output to produce reconstructed recorded conversation.

The audio output may be a recorded audio stream. The audio output may be an audio stream produced in real time by the one or more audio sources. The audio output may be an audio stream produced in non-real time by the one or more audio sources. The identifying the speakers may include accessing the one or more databases to retrieve the voiceprints of users, and matching the characteristics of the speakers in the audio output with the voiceprints. The characteristics of the speakers may include frequencies, amplitudes and durations of voices of the speakers. The analyzing the transcript may include detecting sentiments of the speakers by using the NLP. The operation may further include detecting intervals between words in the audio output for each speaker to determine responsiveness and stress level of the speaker. The operation may further include detecting languages of the audio output. The operation may further include recognizing entities that includes names and personal information attributable to the speakers.

These and others advantages may be also provided by, for example, a method for reconstructing metadata of an audio output that is output from one or more audio sources. The method includes isolating the metadata of a recorded conversation of the audio output, detecting missing portions of the metadata from the isolated metadata, detecting characteristics of speakers involved in the audio output, identifying the speakers from the characteristics of the speakers, creating a transcript of the audio output, analyzing the transcript by using natural language processing (NLP), annotating the transcript with identified speakers, constructing metadata with the identified speakers and results of the analysis of the transcript, in which the missing portions of the metadata is replaced with the constructed metadata, and recombining the constructed metadata with the audio output to produce reconstructed recorded conversation.

DETAILED DESCRIPTIONS

It is to be understood that the figures and descriptions of the present invention may have been simplified to illustrate elements that are relevant for a clear understanding of the present invention. It is also to be understood that the drawings included herewith only provide diagrammatic representations of the presently preferred structures of the present invention and that structures falling within the scope of the present invention may include structures different than those shown in the drawings. It is also to be understood that the invention is not limited in its applications to the details of the particular arrangements shown since the invention is capable of other embodiments. Also, the terminology used herein is for the purpose of description and not of limitation. In the Summary above and in the Detailed Descriptions and in the accompanying drawings, reference is made to particular features of the invention. It is to be understood that the disclosure of the invention in this specification does not include all possible combinations of such particular features. For example, where a particular feature is disclosed in the context of a particular aspect or embodiment of the invention, that feature can also be used, to the extent possible, in combination with and/or in the context of other particular aspects and embodiments of the invention, and in the invention generally.

Metadata is a type of data that provides information about other data that may include content data. In audio recording during audio processing such as phone calls, audio/video conference and voicemail, metadata related to the audio processing may be generated and stored for future references. The information in the metadata may include start time and end time of the audio processing, identifications of users or speaker who attend the conversation, personal information such as speakers' phone numbers, and locations of the speakers. The metadata may be used to create voiceprints that may include characteristics of voice that uniquely identify an individual, and may be used for analyses of audio processing later. The metadata may be stored internally in recording devices, data repository systems coupled to the recording devices, or systems that include audio recording and management capabilities. Alternatively, the metadata may be stored in external storages such as servers or databases in remote locations.

For some instances, however, recording of audio processing may be performed without generating metadata associated with the audio processing, or recording of audio processing may begin before metadata is generated. In this situation, even though the audio processing such as conversations or audio/video conferences is recorded, analyses of the recoded audios may not be complete because of lack of information such as identifications of speaks, places where the conversation took place, and emotions of speakers.

The disclosed invention provides system and method to reconstruct or restore metadata of audio processing which has missing metadata entirely or partially. For this purpose, the system and method of the disclosed invention employ technique that reconstructs metadata for audio outputs that has absent metadata. The system and method of the disclosed invention may provide standard audio files with more metadata definition such as who is speaking, transcripts, emotional analysis, etc. to enhance static audio files by creating rich information media records. The system and method of the disclosed invention work on any audio output such as a voicemail or in an environment where the initial identification of people is beneficial once a recording is made. Applications of the disclosed invention are limitless, and recorded audio output may be used as a means to generate in depth metadata for the audio output. For example, voiceprints may be used to identify speakers individually. Therefore even indistinct recordings of audio output without metadata can be transformed into useful audio recordings by reconstructing metadata of the indistinct recordings. The system and method of the disclosed invention may be used for real-time calls so that call information, such as caller identification and call duration, is updated and the metadata construct is populated as a voice call progresses.

With reference toFIGS. 1A and 1B, shown are exemplary situations of conference calls100A and100B, respectively, in which three speakers SP1, SP2and SP3are involved. An audio stream or output110is processed or recorded. Gray areas in the audio stream110illustrates periods in which a speaker SP1, SP2or SP3speaks.FIG. 1Ashows metadata120A that is generated all the time during the audio processing. Metadata120A, which is associated with the audio output110, are generated during the entire audio processing.FIG. 1Bshows metadata120B that are partially generated during the audio processing. The metadata120B has a time period121during which metadata is not generated. If this audio stream110is recorded, the time period with no metadata121may prevent a complete analysis of the audio stream110later, or may cause incorrect analysis of the audio stream110. As shown inFIG. 1B, for example, while the speaker SP2is speaking, the metadata is not constructed. In the analysis of the recorded audio output110later, the speaker SP2may not be identified, or the analysis may lead to a conclusion that the speaker SP2has not attended the conference call100B. As illustrated in the example above, analysis without metadata may be a challenge because of the lack of information about the audio outputs. In this or similar situations, the system and method of the disclosed invention may be used to reconstruct metadata for the period121in which metadata is missing.

The system and method of the disclosed invention for reconstructing metadata utilize algorithms or processes to extract audio data from audio outputs that are provided by audio sources to construct metadata. The audio data may include information or characteristics of speakers, transcript, sentiment analysis, language spoken, etc. The method of the disclosed invention may be applied to an existing recording, where the audio output once analyzed may produce data that can lead to the reconstruction of metadata. Alternatively, the method of the disclosed invention may be applied in real time calls or audio conferences.

The method of the disclosed invention reconstructs metadata by transcribing audio outputs such as voice streams, by separating speakers, and then by applying machine learning to correlate metadata to a specific speaker. The process adds information that would otherwise be missing from the transcribed audio output file. Furthermore, extra data extracted from audio files, while metadata is reconstructed, may be stored in a database for the purposes of leveraging customer relationship management (CRM) type applications. The extracted data and reconstructed metadata may also be utilized to be adapted for language translations or for other utilities such as call suppression when a speaker's identity once derived from an audio file triggers, for example, General Data Protection Regulation (GDPR) compliance.

When an audio output file is produced through audio capturing processes with absent metadata, the method of the disclosed invention applies processes to reconstruct the missing metadata. These processes may include detecting characteristics of the speaker and producing output based on those learned characteristics. The metadata builder is a component within the process that may apply different methods to reconstruct the missing metadata. If the audio from a recording is analyzed and then associated with voiceprints that are stored in a database, speaker may be recognized and identified, which means that each indistinct speaker in the audio output file can be identified and the transcript can then be precisely annotated (tagged) for each speaker. Other steps may include processes such as analyzing transcript using natural language processing (NLP) techniques to generate sentiment bias. In this case, sentiment (i.e. emotional) analysis is an additional metadata point that can be associated with each segment of the original audio file or tags within the transcript. Furthermore, the transcript can be translated, in which case the reconstructed metadata based on an original first language can provide alternate languages through one or more of the output formats.

With reference toFIG. 2, shown is an embodiments of a method200to reconstruct metadata for audio output that includes missing portions of metadata. In order to reconstruct the metadata, an audio output is generated, block201. The audio output may be generated by an audio capturing devices connected to audio sources, and may be stored at one or more storage media as audio files. The audio output is examined to check if the audio output is suitable for metadata analysis and restoration, block202. An audio quality score, which may represent quality of the audio output, may be used to evaluate the quality of the audio output. In order to calculate the audio quality score, the audio output may be transcribed to create a transcript. However, the quality of the audio output may be evaluated with any known methods. If the audio quality score is below a predetermined value or the quality of the audio output is not acceptable for metadata analysis, block203, the audio output may be enhanced, block204. Any known method may be used to enhance the audio output. When the quality of the audio output is improved to be suitable for metadata analysis, metadata of recorded conversations of the audio output is detected and is isolated from the audio output, block205. The isolated metadata may be separated from the audio output. The isolated metadata is examined to determine whether the metadata have any missing portions, block206. If the metadata have missing portions, block207, the metadata is reconstructed to properly restore missing metadata, block208. In the reconstruction of metadata, absent fields of the metadata, which may be the missing portions of the metadata, may be added or filled with information constructed through the metadata reconstruction processes. The missing portions of the metadata may be replaced with constructed metadata. The steps for reconstructing metadata are further described referring toFIG. 3. If the metadata do not have any missing portions, the metadata is recombined with the audio output, block209. However, the metadata may be further enhanced to include more information of the audio output, which is described referringFIG. 4.

Once the metadata is reconstructed, the metadata is recombined with the recorded conversations of the audio output to produce reconstructed recorded conversations, block209, producing reconstructed audio output that includes reconstructed metadata. The reconstructed audio output is examined to check if additional processing is necessary for certain applications. If additional processing is necessary, block210, the method applies additional processing to the reconstructed audio output, block211. In this step, more information related to the audio output may be added to the metadata. Once the audio output and the metadata are enhanced through the additional processing211, the metadata is validated for future reference and other applications, block212. The validated metadata may be stored in external databases for other applications such as CRM applications.

With reference toFIG. 3, shown is a workflow diagram for step208shown inFIG. 2, which includes a method to reconstruct missing metadata. The method includes detecting characteristics of speakers, block301. Once the characteristics of the speakers are detected, the speakers are identified, block302. In this step, voiceprints of speakers may be used to identify the speakers with the detected characteristics of the speakers in the audio output. The voiceprints may be retrieved from one or more databases that store the voiceprints of speakers or users. The one or more databases may be located remotely or located internally in the metadata reconstruction system. The voice prints are retrieved by the metadata reconstruction system to identify speakers. If the databases are located remotely, the metadata reconstruction system may communicate with the databases through a network. A voiceprint is a set of measurable characteristics of a human voice that uniquely identifies an individual. These characteristics may be expressed as a mathematical formula, and may include audio characteristics such as frequency, amplitude and duration. When the audio output is analyzed and is associated with voiceprints that are stored in a database, speakers may be identified by matching characteristics of the audio output with voiceprints, which means that each indistinct speaker in the audio output can be identified. Machine learning processes may be further applied to correlate metadata to specific speaker. Once the speakers are identified, the transcript may be precisely annotated (tagged) for each speaker.

In order to further process the reconstruction of missing metadata, a transcript of the audio output may be created, block311. The transcript may be created referring to the identifications of the speakers that is generated in step302. As the speakers are identified in step302, the transcript may be annotated with the identified speakers. The transcript is further analyzed by using NLP, block312, to detect more information such as sentiment, block313. NLP utilizes different techniques to interpret human languages. For example, NLP breaks down language into elementary pieces, and constructs relationships between the pieces and explores how the pieces work together to create meaning. NLP has the capability to provide content categorization such as search and indexing, contextual extraction in which structured information from texts is extracted, sentiment analysis in which mood may be identified from texts, and topic modeling in which meaning in text collections may be captured.

The transcript may be further analyzed to collect more information such as to detect language, block314, and to recognize entities, block315. The entities may include names and personal information of the speakers such as phone numbers and/or addresses. Based on the recognized entities, speakers' profiles such as contact information may be updated. The identifications of speakers constructed in step302and the data obtained through analysis of transcripts in steps312-315are used to reconstruct missing metadata, block321. Absent fields of the metadata, which is the missing portions of the metadata, may be filled or added with information constructed through steps shown inFIG. 3. The transcript is annotated with identified speakers, and more information of the speakers (entities) are added to the metadata.

With reference toFIG. 4, shown is a workflow diagram for step211shown inFIG. 2, which are to perform additional processes that may be necessary for the reconstructed audio output. These processes may include translation of the transcript to alternate language, block401, detection of intervals between words, block411, and determining responsiveness or stress levels of speakers, block412. These analyses in steps411and412may be used to further enhance the reconstructed metadata. Through these steps, the reconstructed metadata and annotation are enhanced, block421. Once the reconstructed metadata is enhanced, the metadata may be stored in databases for further applications. For example, the reconstructed and enhanced metadata may be used to apply call controls or recording controls such as voice suppression or transcript suppression when these controls are required.

With reference toFIG. 5, the disclosed invention provides a metadata reconstruction system510that communicates with one or more audio sources520and one or more databases530. The metadata reconstruction system510may also communicate with audio recorder540if necessary. The one or more audio sources520include hardware (systems or devices) that is capable of producing or processing audios. Examples of the audio sources include but are not limited to phone systems for real time voice calls, video/audio conference systems for video/audio conferences involving multiple speakers in real time, audio systems that can play back recorded audios, and storage media such as hard disk driver and memories which can stores audio files. The recorded audios may include voice mails, recorded video/audio conferences, and any recorded voices of speakers. The audio recorder540records audio outputs from any types of audio sources that may include sound in indoor and outdoor environment. The audio recorder540may have a storage medium or memories to store the recorded audios. The database530stores voiceprints of users (speakers). Voiceprints in the database530are retrieved by the metadata reconstruction system510to identify the speakers in the audio output. The database530may store other information of users that may be retrieved by the metadata reconstruction system510to perform additional processing (block211inFIG. 2). The other information of users may include users' profiles, contact information, and personal information attributable to the speakers.

In one embodiment, the metadata reconstruction system510interacts with the audio sources520to retrieve the recorded audio stored in the audio sources. In another embodiment, the metadata reconstruction system510interacts with the audio sources520to directly retrieve audio outputs in real time. This embodiment enables construction of metadata in real time during voice call or audio conferences. Voiceprints may be retrieved from databases in real time during the voice call or call conferences to identify speakers, and the audio output may be transcribed to produce a transcript in real time. The transcript may be further analyzed using NLP in real time. Optionally, the metadata reconstruction system510may include an internal audio capturing element515shown inFIG. 6to store the audio output from the audio sources520while the audio output is processed to construct metadata.

With reference toFIG. 6, the metadata reconstruction system510includes one or more storage media512that may include memories and/or hard disk drives. The storage media512stores software or computer programs including instructions that perform operations to construct metadata when executed by one or more processors511. The operations have functionalities to examine audio outputs that are retrieved from the audio sources520, and to reconstruct metadata of the audio output if the audio output has missing portions of metadata. As described above, the operations may include isolating the metadata of the audio output, detecting missing portions of the metadata from the isolated metadata, detecting characteristics of speakers involved in the audio output, identifying the speakers from the characteristics of the speakers by utilizing voiceprints of speakers, creating a transcript of the audio output, analyzing the transcript with NLP, annotating the transcript with identified speakers, constructing metadata with the identified speakers and results of the analysis of the transcript, and recombining the constructed metadata with the audio output to produce reconstructed audio output.

The metadata reconstruction system510may further include input/output adapter514to communicate with the audio sources520and the external audio recorder540. The metadata reconstruction system510may include an internal audio capturing element515such as audio recorder to capture audio streams from the audio sources520if this capturing is desired. The metadata reconstruction system510may store the recorded audio streams in one of the storage media512. The metadata reconstruction system510includes a networking adapter513to communicate with one or more databases or other external systems. As described above, in the process of reconstructing missing metadata, the metadata reconstruction system510may interact with one or more databases530that store voiceprints and additional data of speakers such as speakers' phone numbers. The metadata reconstruction system510detects characteristics of speakers in the audio output, and then may access voiceprints stored in the databases530to match characteristics of the audio output with voiceprints. Voiceprints may be used to identify the speakers with the characteristics of the speakers. Accessing the database through the networking adapter513may be performed wirelessly or with wires. The metadata reconstruction system510may be connected to audio sources520and/or the audio recorder540wirelessly or with wires.

The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. Those skilled in the art will recognize that many variations are possible within the spirit and scope of the invention and the embodiments described herein.