Patent Description:
With development of voice technology, voiceprint recognition, as a biometric authentication technology, is gradually applied to various intelligent devices. Common applications include a voiceprint recommendation system, a voiceprint lock function, voiceprint payment, and the like. Voice usually contains channel information, environmental noise information and speaker information. For the voiceprint recognition technology, only the speaker information needs to be acquired. However, it may be difficult to completely separate these types of information, so that a voiceprint recognition system may be greatly affected by channels. Since different voice recording devices may introduce different channel information, the effect may be relatively poor in cross-device applications.

A method is needed to better separate speaker information (that is, voiceprint features of a speaker) from various types of noise information.

Related technology is known from <CIT>, <CIT>, <CIT> and <NPL>.

In order to overcome the problems in related technologies, the disclosure provides a method for training a voiceprint extraction model, as well as a device and a medium thereof.

According to a first aspect of embodiments of the disclosure, a method for training a voiceprint extraction model is provided. The voiceprint extraction model may be a neural network model. The method includes:.

The loss function is a weighted sum of a classification loss function, a reconstruction loss function, a difference loss function and a similarity loss function. The difference loss function is for representing orthogonality between a shared space and a private space. The shared space and the private space are respectively established for data from different channels, the shared space being shared by the different channels and configured to learn shared features which are voiceprint features, the private space being limited by each of the different channels and configured to learn private features which are channel noises generated by each of the T devices, wherein the orthogonality between the shared space and the private space represents independency between the shared space and the private space. The classification loss function is for predicting a final output label. The reconstruction loss function is for ensuring that the private features of the private space can act on learning objectives. The similarity loss function is for ensuring that a network can extract similarity features of different devices.

The acquiring the loss function of each training of the neural network model may include:.

The processing the voiceprint feature information to remove the relevance between the voiceprint feature information and the channel to which the voiceprint feature information belongs may include:
processing the voiceprint feature information by gradient inversion.

According to a second aspect of embodiments of the disclosure, a method for voiceprint recognition is provided. The method may implement the voiceprint extraction model trained by the above method for training a voiceprint extraction model. The method includes:.

The method may further include:
storing the determined correspondence between the voiceprint features and the user information in memory.

The acquiring the voiceprint features of the user based on the information of the user may include:
acquiring the voiceprint features of the user from the memory based on the information of the user.

According to a third aspect of embodiments of the disclosure, a device for training a voiceprint extraction model is provided. The device is applicable to a neural network model and includes:.

The loss function acquisition module may be further configured to:.

The loss function acquisition module may be further configured to:
process the voiceprint feature information by gradient inversion to remove the relevance between the voiceprint feature information and the channel to which the voiceprint feature information belongs.

According to a fourth aspect of embodiments of the disclosure, a non-transitory computer-readable storage medium has stored instructions therein that, when executed by a processor of a device, cause the device to implement a method for voiceprint recognition, the method including:.

The disclosure provides a method for training a voiceprint extraction model. After the voiceprint extraction model is trained by this method, voiceprint features of a user are extracted from the acquired voice of the user, so as to achieve better separation of the voiceprint features from channel noise.

The unified trained voiceprint extraction model is applicable to different devices, so that the workload can be greatly reduced, and a system is easier to be maintained. In addition, when a user needs to use the model, the user only needs to register on one device and then can use the model on other devices, thereby improving the use experience.

It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the disclosure.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation models described in the following exemplary embodiments do not represent all implementation models consistent with the disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the disclosure as detailed in the appended claims.

The terms used in the present disclosure are for describing particular embodiments only, and are not intended to limit the present disclosure. The singular forms "a/an", "the" and "this" used in the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates other meanings. It is to be understood that the term "and/or" as used herein refers to and includes any or all possible combinations of one or more associated listed items.

It is to be understood that although the terms first, second, third, etc. may be used to describe various information in the present disclosure, the information should not be limited to these terms. The terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "during" or "when" or "in response to determination".

A voiceprint recognition system may be greatly affected by channels. Since different voice recording devices may introduce different channel information, the effect is often poor in cross-device applications.

To solve the above problems, a proposed method is establishing a voiceprint mapping model by learning a mapping relationship between different channels, extracting speaker features from voice obtained by a voice recording device, and binding the speaker features with a user, so as to obtain depth features of the voice on other devices, that is, speaker feature representations of the user on other voice recording devices, according to the voiceprint mapping model between different devices.

However, this method requires additional learning of the mapping relationships between different devices one by one. When there are more devices, more voiceprint mapping models need to be established. With the number of intelligent devices increasing, it is required to establish a voiceprint model for each of the devices, and in addition, continuous maintenance is required, which involves huge workload. Moreover, when using these devices, a user needs to register on each of these devices. The operations are repeated and complicated.

The disclosure provides a method for voiceprint recognition. In this method, voice of a user is acquired by a voice acquisition device which is provided with a trained voiceprint extraction model. Voiceprint features of the user are acquired by the trained voiceprint extraction model. A correspondence between the voiceprint features and user information is determined. Information of the user is acquired when the user performs voiceprint recognition. Voiceprint features of the user are acquired based on the information of the user. Then voiceprint recognition is performed on the user based on the voiceprint features of the user. The disclosure further provides a method for training a voiceprint extraction model. After the voiceprint extraction model is trained by this method, voiceprint features of a user are extracted from the acquired voice of the user, so as to achieve better separation of the voiceprint features from channel noise.

In the method for voiceprint recognition of the disclosure, a unified trained voiceprint extraction model is applicable to different devices, so that the workload can be greatly reduced, and a system is easier to be maintained. In addition, when a user needs to use the model, the user only needs to register on one device and then can use the model on other devices, thereby improving the user experience.

<FIG> is a flowchart of a method for training a voiceprint extraction model according to an exemplary embodiment. The voiceprint extraction model is a neural network model. As shown in <FIG>, the method includes the following operations.

Operation <NUM> includes that: voice data of a user on T devices is acquired, and acoustic features are extracted based on the voice data to construct a training data set. T may be a positive integer greater than or equal to <NUM>.

Operation <NUM> includes that: the training data set is input into the neural network model, the neural network model is trained, and a loss function of each training of the neural network model is acquired.

Operation <NUM> includes that: it is determined that the training of the neural network model is completed when the loss function converges.

The loss function may be a weighted sum of a classification loss function, a reconstruction loss function, a difference loss function and a similarity loss function.

The voiceprint extraction model in this method is a neural network model known to those skilled in the art, such as a deep neural network model. The voiceprint extraction model may be a unified model independent of devices. In this model, a shared space and a private space are respectively established for data from different channels. The shared space is shared by different channels and configured to learn shared features, that is, the voiceprint features of a speaker. The private space is limited by each of channels and configured to learn the private features of each of devices, that is, the channel noise generated by each of devices. Therefore, the representations of audio data in the shared space may be speaker voiceprint features independent of channels, that is, speaker voiceprint features may be independent of devices.

The deep neural network model mentioned in the disclosure is shown in <FIG>, and the descriptions of each of modules are as follows:.

The above modules, other than the difference loss function and the similarity loss function, may be acquired by those skilled in the art by a deep neural network model, which will not be repeated here.

A loss function L of the deep neural network model of the disclosure may be expressed as:
<MAT>.

λ, α, β and γ respectively represent weights of a classification loss function, a reconstruction loss function, a difference loss function and a similarity loss function. These weights may be updated during a training process of the deep neural network model. The loss function of each round of training may be acquired by updating the weights. Here, the updating of weights may be implemented by those skilled in the art based on the training of the deep neural network model, which will not be repeated here.

In this method, by introducing the difference loss function and the similarity loss function into the loss function of the deep neural network model, better separation of voiceprint features from channel noise can be implemented.

In an optional implementation, the operation that the loss function of each training of the neural network model is acquired may include that:.

voiceprint feature information and channel noise information are acquired from each training of the neural network model; and.

the difference loss function is calculated based on the voiceprint feature information and the channel noise information.

The difference loss function Ldifference may be calculated by the following formula:
<MAT><MAT> represents the square of the F-norm. Other types of norms may also be used for calculation. <MAT> and <MAT> respectively represent a voiceprint feature matrix and a channel noise matrix obtained during each training of the neural network model. The orthogonality between the voiceprint features and the channel noise may be obtained by the difference loss function. The orthogonality between the two parameters represents the independency between the two parameters. Therefore, when the orthogonality between the voiceprint features and the channel noise is smaller, the separation of the voiceprint features from the channel noise is better.

In order to ensure that different subspaces are transferable, it is required to ensure that the distribution of the shared features <MAT> has similarity. Accordingly, the voiceprint feature information may be processed to remove the relevance between the voiceprint feature information and the channel to which the voiceprint feature information belongs, and then, the processed voiceprint feature information may be input into the domain classifier, so that the domain classifier is unable to distinguish which subspace the input comes from, so as to ensure that a network can extract similarity features of different devices, that is, acquiring common features of the input, namely shared features. The similarity loss function is defined as follows:
<MAT>
wherein d represents a classification result of each of sentences in voice of a speaker by the domain classifier (a shared feature is classified to a device to which the shared feature belongs, namely, a channel). The value may be <NUM> or <NUM>. When the classification is correct, the value is <NUM>. When the classification is incorrect, the value is <NUM>. <MAT> represents a probability value of correct classification output by the neural network model, and the value may be between <NUM> and <NUM>.

In an optional implementation, the operation that the voiceprint feature information is processed to remove the relevance between the voiceprint feature information and the channel to which the voiceprint feature information belongs may include that:
the voiceprint feature information is processed by gradient inversion.

The gradient inversion may be performed synchronously in a training process of a network. After the gradient inversion is performed on voiceprint feature information, the entire network may not distinguish which device the input voice comes from, so as to achieve the purpose of acquiring shared features of the input voice.

In addition, in a classification loss function Lclass:
<MAT>
wherein N represents the number of sentences contained in acquired voice of a speaker, <MAT> represents input real voice, and <MAT> represents voice predicted by a model.

The reconstruction loss function Lrecon may be calculated by the following formula:
<MAT>
wherein Lsi_mse is a scale-invariant mean square error loss function which may be calculated by the following formula:
<MAT>
wherein k represents a dimension of input x, <NUM>k represents a matrix with an element <NUM> and a length k, and <MAT> represents an L<NUM>-norm. Other types of norms may also be used for calculation.

Both the classification loss function and the reconstruction loss function may be calculated by existing methods. The classification loss function may be softmax, or other optimized varieties of softmax, such as A-Softmax, L-Softmax and AM-Softmax. Similarly, the reconstruction loss function may be other functions other than the scale-invariant mean square error, which is not limited here.

The disclosure further provides a method for voiceprint recognition. As shown in <FIG> (not covered by the invention), the method includes the operations as below.

Operation <NUM> includes that voice of a user is acquired by a voice acquisition device. The voice acquisition device is provided with a trained voiceprint extraction model.

Operation <NUM> includes that voiceprint features of the user are acquired by the trained voiceprint extraction model, and a correspondence between the voiceprint features and user information is determined.

Operation <NUM> includes that information of the user is acquired when the user performs voiceprint recognition.

Operation <NUM> includes that voiceprint features of the user are acquired based on the information of the user.

Operation <NUM> includes that voiceprint recognition is performed on the user based on the voiceprint features of the user.

The method for voiceprint recognition is implemented by using the voiceprint extraction model trained by the above method for training a voiceprint extraction model. That is, the trained voiceprint extraction model in the operation <NUM> is trained by the above method for training a voiceprint extraction model. The above trained voiceprint extraction model may well separate voiceprint features from channel noise. Therefore, after voice of a user is input into the trained voiceprint extraction model, voiceprint features of the user may be acquired. The voiceprint features may be in one-to-one correspondence to the information of the user. Then, the above trained voiceprint extraction model may be shared on different voice acquisition devices. Based on information registered by the user on a certain device, voiceprint features in one-to-one correspondence to the information may be acquired, and voiceprint recognition may be performed on the user based on the voiceprint features.

In an optional implementation, the method may further include that the determined correspondence between the voiceprint features and the user information is stored in memory.

The operation that the voiceprint features of the user are acquired based on the information of the user may include that the voiceprint features of the user are acquired from the memory based on the information of the user.

Here, the memory may be cloud memory. When a user performs voiceprint recognition on other devices, the user information in a cloud terminal may be taken to conduct matching to recognize the user. Corresponding operations may be completed according to voice instructions of the user.

The following describes specific embodiments according to the disclosure in combination with specific application scenarios. In the present embodiment, the voiceprint extraction model is a deep neural network model. As shown in <FIG>, the method further includes the following operations.

Operation <NUM> includes that voice data of a user on <NUM> devices is acquired, and a training data set is constructed based on these voice data.

Operation <NUM> includes that the training data set is input into the deep neural network model, and the neural network model is trained.

Operation <NUM> includes that a loss function of each training is acquired.

Operation <NUM> includes that it is determined that the training of the deep neural network model is completed when the loss function converges.

Operation <NUM> includes that the trained deep neural network model is disposed on a plurality of voice acquisition devices.

Operation <NUM> includes that voice of the user is acquired by the voice acquisition devices provided with the trained deep neural network model.

Operation <NUM> includes that voiceprint features of the user are acquired by the trained voiceprint extraction model, and a correspondence between the user and voiceprint features thereof is stored in a cloud memory.

Operation <NUM> includes that voiceprint recognition is performed on the user based on the voiceprint features of the user, and then, corresponding operations are performed.

The disclosure further provides a device for training a voiceprint extraction model. The device is applicable to a neural network model. As shown in <FIG>, the device includes:.

In an optional implementation, the loss function acquisition module <NUM> is further configured to:.

In an optional implementation, the loss function acquisition module <NUM> is further configured to:
process the voiceprint feature information by gradient inversion to remove the relevance between the voiceprint feature information and the channel to which the voiceprint feature information belongs.

The disclosure further provides a voiceprint recognition device. As shown in <FIG> (not covered by the invention), the device further includes:.

In an optional implementation, the device may further include:
a storage module, configured to store the determined correspondence between the voiceprint features and the user information in memory.

The voiceprint acquisition module is further configured to acquire the voiceprint features of the user from the memory based on the information of the user.

With respect to the device in the above embodiment, the specific manners for performing operations for individual modules therein have been described in detail in the embodiment regarding the method, which will not be elaborated herein.

The disclosure provides a method for voiceprint recognition. In this method, voice of a user is acquired by a voice acquisition device which is provided with a trained voiceprint extraction model. Voiceprint features of the user are acquired by the trained voiceprint extraction model. A correspondence between the voiceprint features and user information are determined. Information of the user is acquired when the user performs voiceprint recognition. Voiceprint features of the user are acquired based on the information of the user. Then voiceprint recognition is performed on the user based on the voiceprint features of the user. The disclosure further provides a method for training a voiceprint extraction model. After the voiceprint extraction model is trained by this method, voiceprint features of a user is extracted from the acquired voice of the user, so as to achieve better separation of the voiceprint features from channel noise.

<FIG> is a block diagram of a device for training a voiceprint extraction model <NUM> according to an exemplary embodiment.

Referring to <FIG>, the device <NUM> may include one or more of the following components: a processing component <NUM>, memory <NUM>, a power component <NUM>, a multimedia component <NUM>, an audio component <NUM>, an Input/Output (I/O) interface <NUM>, a sensor component <NUM>, and a communication component <NUM>.

The processing component <NUM> generally controls overall operations of the device <NUM>, such as operations related to displaying, telephone calls, data communications, camera operations, and recording operations. The processing component <NUM> may include one or more processors <NUM> to execute instructions to complete all or part of the operations of the method described above. In addition, the processing component <NUM> may include one or more modules to facilitate the interaction between the processing component <NUM> and other components. For example, the processing component <NUM> may include a multimedia module to facilitate the interaction between the multimedia component <NUM> and the processing component <NUM>.

Examples of such data include instructions for any applications or methods operated on the device <NUM>, contact data, phonebook data, messages, pictures, videos, etc. The memory <NUM> may be implemented using any type of volatile or non-volatile memory devices, or a combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

The power component <NUM> is configured to provide power to various components of the device <NUM>. The power component <NUM> may include a power management system, one or more power sources, and any other components associated with the generation, management and distribution of power in the device <NUM>.

The multimedia component <NUM> may include a screen that provides an output interface between the device <NUM> and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes the TP, the screen may be implemented as a touch screen to receive an input signal from a user. The TP includes one or more touch sensors to sense touch, swipe, and gestures on the TP. The touch sensor may not only sense a boundary of a touch or swipe action, but also detect duration and pressure related to the touch or swipe operation. The front camera and the rear camera may receive external multimedia data while the device <NUM> is in an operation mode, such as a photographing mode or a video mode. Each front camera and each rear camera may be fixed optical lens systems or may have focal lengths and optical zoom capabilities.

The audio component <NUM> is configured to output and/or input audio signals. For example, the audio component <NUM> includes a microphone (MIC), and the microphone is configured to receive external audio signals when the device <NUM> is in an operation mode, such as a calling mode, a recording mode, and a voice identification mode. The received audio signals may be further stored in the memory <NUM> or transmitted via the communication component <NUM>. In some embodiments, the audio component <NUM> may further include a speaker configured to output audio signals.

The I/O interface <NUM> is configured to provide an interface between the processing component <NUM> and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but not limited to: a home button, a volume button, a start button, and a lock button.

The sensor component <NUM> may include one or more sensors configured to provide various aspects of state assessment for the device <NUM>. For example, the sensor component <NUM> may detect an open/closed status of the device <NUM>, and relative positioning of components. For example, the components are the display and the keypad of the device <NUM>. The sensor component <NUM> may also detect a change in position of the device <NUM> or a component of the device <NUM>, a presence or absence of user contact with the device <NUM>, an orientation or an acceleration/deceleration of the device <NUM>, and a change in temperature of the device <NUM>. The sensor component <NUM> may include a proximity sensor configured to detect the presence of objects nearby without any physical contact. The sensor component <NUM> may also include light sensors, such as CMOS or CCD image sensors, for use in imaging applications. In some embodiments, the sensor component <NUM> may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component <NUM> is configured to facilitate wired or wireless communication between the device <NUM> and other devices. The device <NUM> may access a wireless network based on a communication standard, such as WiFi, <NUM> or <NUM>, or a combination thereof. In an exemplary embodiment, the communication component <NUM> receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component <NUM> may further include a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module can be implemented based on the Radio Frequency Identification (RFID) technology, the Infrared Data Association (IrDA) technology, the Ultra-Wideband (UWB) technology, the Bluetooth (BT) technology and other technologies.

In exemplary embodiments, the device <NUM> may be implemented by one or more Application-Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

In exemplary embodiments, a non-transitory computer readable storage medium storing instructions, such as memory <NUM> storing instructions, is also provided. The instructions may be executed by the processor <NUM> of the device <NUM> to complete the above methods. For example, the non-transitory computer-readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a soft disk, an optical data storage device or the like.

A non-transitory computer-readable storage medium is provided. When an instruction in the storage medium is executed by a processor of a mobile terminal, the mobile terminal may execute a method for training a voiceprint extraction model, including: voice data of a user on T devices is acquired, and acoustic features are extracted based on the voice data to construct a training data set, where T is a positive integer greater than or equal to <NUM>; the training data set is input into the neural network model, the neural network model is trained, and a loss function of each training of the neural network model is acquired; and it is determined that the training of the neural network model is completed when the loss function converges, wherein the loss function is a weighted sum of a classification loss function, a reconstruction loss function, a difference loss function and a similarity loss function.

<FIG> is a block diagram of a device for training a voiceprint extraction model <NUM> according to an exemplary embodiment. For example, the device <NUM> may be provided as a server. Referring to <FIG>, the device <NUM> includes a processing component <NUM>, further including one or more processors and memory resources represented by memory <NUM> for storing instructions executable by the processing component <NUM>, such as application programs. The application program stored in the memory <NUM> may include one or more modules which each corresponds to a set of instructions. Furthermore, the processing component <NUM> is configured to execute the instructions to execute the above method: voice data of a user on T devices is acquired, and acoustic features are extracted based on the voice data to construct a training data set, wherein T is a positive integer greater than or equal to <NUM>; the training data set is input into the neural network model, the neural network model is trained, and a loss function of each training of the neural network model is acquired; and it is determined that the training of the neural network model is completed when the loss function converges, wherein the loss function is a weighted sum of a classification loss function, a reconstruction loss function, a difference loss function and a similarity loss function.

The device <NUM> may also include a power component <NUM> configured to execute power management of the device <NUM>, a wired or wireless network interface <NUM> configured to connect the device <NUM> to the network, and an Input/Output (I/O) interface <NUM>. The device <NUM> may operate an operating system stored in the memory <NUM>, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. It is intended that the specification and embodiments be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims.

Claim 1:
A method for training a voiceprint extraction model, the voiceprint extraction model being a neural network model, the method characterized by comprising:
acquiring (<NUM>) voice data of a user on T devices, T being a positive integer greater than or equal to <NUM>;
extracting (<NUM>) acoustic features based on the voice data to construct a training data set;
inputting (<NUM>) the training data set into the neural network model;
training (<NUM>) the neural network model;
acquiring (<NUM>) a loss function of each training of the neural network model; and
determining (<NUM>) that the training of the neural network model is completed when the loss function converges,
wherein the loss function is a weighted sum of a classification loss function, a reconstruction loss function, a difference loss function and a similarity loss function,
wherein the difference loss function is for representing orthogonality between a shared space and a private space, wherein the shared space and the private space are respectively established for data from different channels, the shared space being shared by the different channels and configured to learn shared features which are voiceprint features, the private space being limited by each of the different channels and configured to learn private features which are channel noises generated by each of the T devices, wherein the orthogonality between the shared space and the private space represents independency between the shared space and the private space,
wherein the classification loss function is for predicting a final output label,
wherein the reconstruction loss function is for ensuring that the private features of the private space can act on learning objectives, and
wherein the similarity loss function is for ensuring that a network can extract similarity features of different devices.