Patent Description:
This application relates to the field of Internet technologies, and specifically, to a speech data processing method and apparatus, an electronic device, and a computer-readable storage medium.

The essence of speech enhancement is speech noise reduction. Speeches acquired by a microphone are generally speeches with different noise, and a main objective of speech enhancement is to recover speeches without noise from speeches with noise. Various interfering signals may be effectively suppressed and a target speech signal may be enhanced through speech enhancement, thereby not only improving the speech intelligibility and speech quality, but also helping improve speech recognition.

When speech enhancement is performed on a speech to be processed, a universal noise reduction model is generated first through training, and adaptive training is then performed, for different speakers, on the entire noise reduction model or some layers in the model by using speech data corresponding to the speakers, to obtain noise reduction models corresponding to the different speakers respectively and store the noise reduction models. In practice, corresponding noise reduction models are obtained for different speakers, and noise reduction processing is performed on speech data of the speakers by using the noise reduction models.

<CIT>(A1) discloses a following solution. An audio signal generated by a device based on audio input from a user may be received. The audio signal may include at least a user audio portion that corresponds to one or more user utterances recorded by the device. A user speech model associated with the user may be accessed and a determination may be made background audio in the audio signal is below a defined threshold. In response to determining that the background audio in the audio signal is below the defined threshold, the accessed user speech model may be adapted based on the audio signal to generate an adapted user speech model that models speech characteristics of the user. Noise compensation may be performed on the received audio signal using the adapted user speech model to generate a filtered audio signal with reduced background audio compared to the received audio signal.

<CIT>) discloses an intelligent voice recognition method, a voice recognition device, and an intelligent computing device. A method of intelligently recognizing a voice in a voice recognition device includes obtaining an ambient noise signal from the microphone detection signal when a voice is detected from the microphone detection signal; updating a previously learned noise cancellation model based on the ambient noise; and removing the ambient noise signal from the microphone detection signal. Therefore, a voice recognition performance of the voice recognition device can be maximized. At least one of the voice recognition device, the intelligent computing device and the server of the present disclosure may be associated with an Artificial Intelligence module, a drone (Unmanned Aerial Vehicle, UAV), robot, Augmented Reality (AR) device, virtual reality (VR) device and a device related to the <NUM> service.

The present invention is provided in the appended independent claims. Advantageous embodiments are provided in the appended dependent claims.

To describe the technical solutions in the embodiments of this disclosure more clearly, the following briefly describes the accompanying drawings required for describing the embodiments of this disclosure.

Embodiments of this disclosure are described in detail below, and examples of the embodiments are shown in accompanying drawings, where the same or similar elements or the elements having same or similar functions are denoted by the same or similar reference numerals throughout the description. The embodiments that are described below with reference to the accompanying drawings are exemplary, and are only used to interpret this disclosure and cannot be construed as a limitation to this disclosure.

A person skilled in the art may understand that, the singular forms "a", "an", "said", and "the" used herein may include the plural forms as well, unless the context clearly indicates otherwise. It is to be further understood that, the terms "include" and/or "comprise" used in this specification of this disclosure refer to the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or combinations thereof. It is to be understood that, when an element is "connected" or "coupled" to another element, the element may be directly connected to or coupled to the another element, or an intermediate element may exist. In addition, the "connection" or "coupling" used herein may include a wireless connection or a wireless coupling. The term "and/or" used herein includes all of or any of units and all combinations of one or more related listed items.

Artificial intelligence (AI) is a theory, method, technology, and application system that uses a digital computer or a machine controlled by the digital computer to simulate, extend, and expand human intelligence, perceive an environment, obtain knowledge, and use knowledge to obtain an optimal result. In other words, artificial intelligence is a comprehensive technology in computer science and attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, to enable the machines to have the functions of perception, reasoning, and decision-making.

The artificial intelligence technology is a comprehensive discipline, and relates to a wide range of fields including both hardware-level technologies and software-level technologies. The basic artificial intelligence technologies generally include technologies such as a sensor, a dedicated artificial intelligence chip, cloud computing, distributed storage, a big data processing technology, an operating/interaction system, and electromechanical integration. Artificial intelligence software technologies mainly include several major directions such as a computer vision technology, a speech processing technology, a natural language processing technology, and machine learning or deep learning.

Key technologies of the speech technology include an automatic speech recognition (ASR) technology, a text-to-speech (TTS) technology, and a voiceprint recognition technology. To make a computer capable of listening, seeing, speaking, and feeling is the future development direction of human-computer interaction, and speech has become one of the most promising human-computer interaction methods in the future.

To make the objectives, technical solutions, and advantages of this disclosure clearer, the following further describes embodiments of this disclosure in detail with reference to the accompanying drawings.

As described above, when speech enhancement is performed on a speech to be processed, for different speakers, noise reduction models corresponding to the speakers need to be obtained, and noise reduction processing is performed on speech data of the speakers by using the noise reduction models. In this way, the noise reduction model corresponding to each speaker needs to be stored, which has a relatively high storage capacity requirement.

Therefore, the embodiments of this disclosure provide a speech data processing method, an electronic device, and a computer-readable storage medium, to resolve the foregoing technical problems in the related art.

The following describes the technical solutions of this disclosure and how to resolve the foregoing technical problems according to the technical solutions of this disclosure in detail by using specific embodiments. The following several specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described repeatedly in some embodiments. The following describes the embodiments of this disclosure with reference to the accompanying drawings.

<FIG> is a diagram of a system architecture to which a speech data processing method is applicable according to an embodiment of this disclosure. Referring to <FIG>, the system architecture diagram includes a server <NUM>, a network <NUM>, and terminal devices <NUM> and <NUM>, where the server <NUM> establishes connection to the terminal device <NUM> and the terminal device <NUM> through the network <NUM>.

In some examples of this disclosure, the server <NUM> is a backend server that receives speech data transmitted by a sender and then processes the received speech data. The server <NUM> provides services for a user together with the terminal device <NUM> and the terminal device <NUM>. For example, after processing speech data transmitted by the terminal device <NUM> (which may also be the terminal device <NUM>) corresponding to the sender, the server <NUM> transmits obtained speech enhancement data to the terminal device <NUM> (which may also be the terminal device <NUM>) corresponding to a recipient so as to provide the speech enhancement data to the user. The server <NUM> may be an independent server or may be a server cluster including multiple servers.

The network <NUM> may include a wired network and a wireless network. As shown in <FIG>, on a side of an access network, the terminal device <NUM> and the terminal device <NUM> may access the network <NUM> in a wireless or wired manner. On a side of a core network, the server <NUM> is usually connected to the network <NUM> in a wired manner. Certainly, the server <NUM> may alternatively be connected to the network <NUM> in a wireless manner.

The terminal device <NUM> and the terminal device <NUM> may be smart devices having a data calculation processing function such as playing processed speech enhancement data provided by the server. The terminal device <NUM> and the terminal device <NUM> include, but are not limited to, a smartphone (in which a communication module is installed), a palmtop computer, a tablet computer, or the like. Operating systems are respectively installed on the terminal device <NUM> and the terminal device <NUM>, which include, but are not limited to, an Android operating system, a Symbian operating system, a Windows mobile operating system, and an Apple iPhone OS operating system.

Based on the system architecture diagram shown in <FIG>, an embodiment of this disclosure provides a speech data processing method, and the processing method is performed by the server <NUM> in <FIG>. As shown in <FIG>, the method includes the following steps S101 to S103.

In a case that first speech data transmitted by a sender is received, obtain a speech enhancement parameter.

In some embodiments, in a process of obtaining the speech enhancement parameter, a pre-stored speech enhancement parameter corresponding to the sender is obtained; and in a case that the speech enhancement parameter corresponding to the sender is not obtained, a preset speech enhancement parameter is obtained.

In practice, this embodiment of this disclosure may be applicable to an application scenario based on network speech communication, such as a telephone conference or a video conference. The sender may be one party transmitting speech data. For example, if a user A makes a speech by using the terminal device <NUM>, the terminal device <NUM> may be the sender, and speech content of the user A may be the first speech data. The first speech data is transmitted to the server through a network, and after receiving the first speech data, the server may obtain the speech enhancement parameter and further perform speech enhancement on the first speech data. A long short-term memory (LSTM) model may be run on the server, and the model may be used for performing speech enhancement on speech data.

Perform speech enhancement on the first speech data based on the obtained speech enhancement parameter to obtain first speech enhancement data, and determine a first speech enhancement parameter based on the first speech data.

In some embodiments, in a case that the speech enhancement parameter corresponding to the sender is not obtained, speech enhancement is performed on the first speech data based on the preset speech enhancement parameter to obtain the first speech enhancement data.

In a case that the speech enhancement parameter corresponding to the sender is obtained, speech enhancement is performed on the first speech data based on the speech enhancement parameter corresponding to the sender to obtain the first speech enhancement data.

In practice, in a case that the speech enhancement parameter corresponding to the sender is not obtained, speech enhancement is performed on the first speech data based on the preset speech enhancement parameter; and in a case that the speech enhancement parameter corresponding to the sender is obtained, speech enhancement is performed on the first speech data based on the speech enhancement parameter corresponding to the sender.

In some embodiments, in a case that the speech enhancement parameter corresponding to the sender is not obtained, the performing speech enhancement on the first speech data based on the preset speech enhancement parameter to obtain the first speech enhancement data and determining a first speech enhancement parameter based on the first speech data includes: performing feature sequence processing on the first speech data by using a trained speech enhancement model to obtain a first speech feature sequence, the speech enhancement model being provided with the preset speech enhancement parameter; performing batch processing calculation on the first speech feature sequence by using the preset speech enhancement parameter to obtain a processed first speech feature sequence and the first speech enhancement parameter; and performing feature inverse transformation processing on the processed first speech feature sequence to obtain the first speech enhancement data.

In a case that the speech enhancement parameter corresponding to the sender is obtained, the performing speech enhancement on the first speech data based on the speech enhancement parameter corresponding to the sender to obtain the first speech enhancement data and determining a first speech enhancement parameter based on the first speech data includes: performing feature sequence processing on the first speech data by using a trained speech enhancement model to obtain a second speech feature sequence; performing batch processing calculation on the second speech feature sequence by using the speech enhancement parameter corresponding to the sender to obtain a processed second speech feature sequence and a second speech enhancement parameter; and performing feature inverse transformation processing on the processed second speech feature sequence to obtain processed second speech enhancement data, and using the processed second speech enhancement data as the first speech enhancement data.

Transmit the first speech enhancement data to a recipient, and update the obtained speech enhancement parameter by using the first speech enhancement parameter to obtain an updated speech enhancement parameter, where the updated speech enhancement parameter is used for performing, in a case that second speech data transmitted by the sender is received, speech enhancement on the second speech data based on the updated speech enhancement parameter.

In some embodiments, in a case that the speech enhancement parameter corresponding to the sender is not obtained, the obtained preset speech enhancement parameter is updated based on the first speech enhancement parameter to obtain the updated speech enhancement parameter, and the first speech enhancement parameter is used as the speech enhancement parameter corresponding to the sender.

In a case that the speech enhancement parameter corresponding to the sender is obtained, the speech enhancement parameter corresponding to the sender is updated by using the first speech enhancement parameter to obtain the updated speech enhancement parameter.

Specifically, after the first speech enhancement parameter is determined based on the first speech data, if a storage does not store the speech enhancement parameter corresponding to the sender, the first speech enhancement parameter may be used as the speech enhancement parameter corresponding to the sender and stored in the storage; and if the storage has stored the speech enhancement parameter corresponding to the sender, the first speech enhancement parameter may be used to replace the stored speech enhancement parameter. In addition, the server transmits the first speech enhancement data obtained through speech enhancement to the recipient, and the recipient plays the first speech enhancement data after receiving the first speech enhancement data.

In some embodiments, the trained speech enhancement model is generated by: obtaining first speech sample data including noise and performing speech feature extraction on the first speech sample data, to obtain a first speech feature sequence; obtaining second speech sample data which does not include noise and performing speech feature extraction on the second speech sample data, to obtain a second speech feature sequence; and training a preset speech enhancement model by using the first speech feature sequence, to obtain a first speech feature sequence outputted by the trained speech enhancement model, and calculating a similarity between the first speech feature sequence outputted by the trained speech enhancement model and the second speech feature sequence until the similarity exceeds a preset similarity threshold, to obtain the trained speech enhancement model.

In some embodiments, a manner of performing speech feature extraction includes: performing speech framing and windowing on speech sample data to obtain at least two speech frames of the speech sample data; performing fast Fourier transformation on each of the speech frames to obtain discrete power spectra respectively corresponding to the speech frames; and performing logarithmic calculation on each of the discrete power spectra to obtain logarithmic power spectra respectively corresponding to the speech frames, and using the logarithmic power spectra as a speech feature sequence of the speech sample data.

In the embodiments of this disclosure, in a case that first speech data transmitted by a sender is received, a speech enhancement parameter is obtained; speech enhancement is then performed on the first speech data based on the obtained speech enhancement parameter to obtain first speech enhancement data, and a first speech enhancement parameter is determined based on the first speech data; and then the obtained speech enhancement parameter is updated by using the first speech enhancement parameter to obtain an updated speech enhancement parameter, the updated speech enhancement parameter is used for performing, in a case that second speech data transmitted by the sender is received, speech enhancement on the second speech data based on the updated speech enhancement parameter, and the first speech enhancement data is transmitted to a recipient. In this way, the server may perform speech enhancement on speech data of the sender based on the speech enhancement parameter corresponding to the sender. Speech enhancement parameters corresponding to different senders are different, so that speech enhancement effects achieved by performing speech enhancement for the different senders are also different. In this way, in a case of not requiring multiple models, speech enhancement is still targeted and only speech enhancement parameters need to be stored without storing multiple models, thereby having a relatively low storage capacity requirement.

A speech data processing method shown in <FIG> is described in detail in the embodiments of this disclosure.

During actual disclosure, this embodiment of this disclosure may be applicable to an application scenario based on network speech communication, such as a telephone conference or a video conference. The sender may be one party transmitting speech data. For example, if a user A makes a speech by using the terminal device <NUM>, the terminal device <NUM> may be the sender, and speech content of the user A may be the first speech data. The first speech data is transmitted to the server through a network, and after receiving the first speech data, the server may obtain the speech enhancement parameter and further perform speech enhancement on the first speech data.

A long short-term memory (LSTM) model may be run on the server, and the model may be used for performing speech enhancement on speech data.

A basic structure of the LSTM model may be shown in <FIG> and include a frontend LSTM layer, a batch processing layer, and a backend LSTM layer, where X is each frame of speech in speech data, and t is a time window.

One frame of speech refers to a short segment in a speech signal. Specifically, a speech signal is unstable macroscopically but stable microscopically and has short-term stability (the speech signal may be considered approximately unchanged within <NUM> to <NUM>). Therefore, the speech signal may be divided into some short segments for processing, and each short segment is referred to as one frame. For example, in a speech of <NUM>, if a length of one frame of speech is <NUM>, the speech includes <NUM> frames.

When the LSTM model processes speech data, the frontend LSTM layer, the batch processing layer, and the backend LSTM layer may perform calculation on speech frames in different time windows at the same time, where the batch processing layer is configured to calculate a speech enhancement parameter corresponding to the speech data, such as an average value and a variance.

Further, in the embodiments of this disclosure, the terminal device <NUM> and the terminal device <NUM> may further have the following features.

In an exemplary embodiment of this disclosure, the obtaining a speech enhancement parameter includes:.

Specifically, after receiving the first speech data, the server may perform speech enhancement on the first speech data by using a trained LSTM model. The trained LSTM model is a universal model and includes a preset speech enhancement parameter, namely, a speech enhancement parameter in the trained LSTM model. The trained LSTM model may perform speech enhancement on speech data of any user.

In the embodiments of this disclosure, to provide targeted speech enhancement for different users, the trained LSTM model may be trained by using speech data of a user to obtain a speech enhancement parameter of the user. In this way, when speech enhancement is performed on speech data of the user, speech enhancement may be performed on the speech data of the user by using the speech enhancement parameter of the user.

For example, the trained LSTM model is trained by using speech data of a user A, to obtain a speech enhancement parameter of the user A. When speech enhancement is performed on subsequent speech data of the user A, the trained LSTM model may perform speech enhancement by using the speech enhancement parameter of the user A.

Therefore, when first speech data of a user is received, the server may first obtain a speech enhancement parameter of the user. In the embodiments of this disclosure, speech enhancement parameters corresponding to users may be stored in a storage of the server, or may be stored in a storage of another device, which is not limited in the embodiments of this disclosure.

If the server has not obtained the speech enhancement parameter of the user, it indicates that the server receives the speech data of the user for the first time, and the server obtains a preset speech enhancement parameter in this case.

In a preferable embodiment of this disclosure, in a case that the speech enhancement parameter corresponding to the sender is not obtained, the step of performing speech enhancement on the first speech data based on the obtained speech enhancement parameter to obtain first speech enhancement data, and determining a first speech enhancement parameter based on the first speech data includes:.

Specifically, if the speech enhancement parameter corresponding to the sender is not obtained, the first speech data may be inputted into the trained LSTM model. The trained LSTM model performs feature sequence processing on the first speech data to obtain a first speech feature sequence corresponding to the first speech data, where the first speech feature sequence includes at least two speech features; then performs batch processing calculation on the first speech feature sequence by using the preset speech enhancement parameter to obtain a processed first speech feature sequence; and then performs feature inverse transformation processing on the processed first speech feature sequence to obtain the first speech enhancement data. That is, speech enhancement is performed on the first speech data by using the trained LSTM model (universal model). The batch processing calculation may be performed by using the following formula (<NUM>) and formula (<NUM>): <MAT> ; and <MAT>
µB is an average value in the speech enhancement parameter, <MAT> is a variance in the speech enhancement parameter, xi is an inputted speech feature, yi is a speech feature outputted after speech enhancement, and ε, γ, and β are variable parameters.

In addition, the trained LSTM model is trained by using the first speech data to obtain the first speech enhancement parameter, namely, the speech enhancement parameter corresponding to the sender, and the first speech enhancement parameter is then stored. The trained LSTM model may be trained by using the following formula (<NUM>) and formula (<NUM>): <MAT> ; and <MAT>
µB is the average value in the speech enhancement parameter, <MAT> is the variance in the speech enhancement parameter, xi is the inputted speech feature, and m is a quantity of speech features.

An execution sequence of performing speech enhancement on the first speech data based on the obtained speech enhancement parameter and determining a first speech enhancement parameter based on the first speech data may be sequential execution or parallel execution. In practice, the execution sequence may be adjusted according to an actual requirement, and is not limited in the embodiments of this disclosure.

In a case that the speech enhancement parameter corresponding to the sender is obtained, the step of performing speech enhancement on the first speech data based on the obtained speech enhancement parameter to obtain first speech enhancement data, and determining a first speech enhancement parameter based on the first speech data includes:.

Specifically, if the speech enhancement parameter corresponding to the sender is obtained, the first speech data may be inputted into the trained LSTM model. The trained LSTM model performs feature sequence processing on the first speech data to obtain a second speech feature sequence corresponding to the first speech data, where the second speech feature sequence includes at least two speech features; then performs batch processing calculation on the second speech feature sequence by using the speech enhancement parameter corresponding to the sender to obtain a processed second speech feature sequence; and then performs feature inverse transformation processing on the processed second speech feature sequence to obtain second speech enhancement data. That is, the speech enhancement parameter in the trained LSTM model is replaced with the speech enhancement parameter corresponding to the sender. Speech enhancement is then performed on the second speech data by using an updated LSTM model. The batch processing calculation may also be performed by using the formula (<NUM>) and the formula (<NUM>), and details are not described herein again.

In addition, the updated LSTM model is trained by using the first speech data to obtain a second speech enhancement parameter, namely, the latest speech enhancement parameter corresponding to the sender, and the second speech enhancement parameter is then stored. The updated LSTM model may also be trained by using the formula (<NUM>) and the formula (<NUM>), and details are not described herein again.

In a preferable embodiment of this disclosure, the trained speech enhancement model is generated by:.

Specifically, first speech sample data including noise is obtained, and speech feature extraction is performed on the first speech sample data, to obtain a first speech feature a. In addition, second speech sample data not including noise is obtained, and speech feature extraction is performed on the second speech sample data, to obtain a second speech feature b. The speech feature a is inputted into an original LSTM model, and the original LSTM model is unidirectionally trained by using the speech feature b as a training target, namely, all parameters in the LSTM model are unidirectionally adjusted, to obtain a trained first speech feature a', where the parameters include a speech enhancement parameter. A similarity between the trained first speech feature a' and the second speech feature b is then calculated until the similarity between the trained first speech feature a' and the second speech feature b exceeds a preset similarity threshold, to obtain the trained LSTM model.

The similarity may be calculated by using a similarity measure such as an included angle cosine or a Pearson correlation coefficient, or may be calculated by using a distance measure such as a Euclidean distance or a Manhattan distance. Certainly, another calculation manner may be alternatively used, and a specific calculation manner may be set according to an actual requirement, which is not limited in the embodiments of this disclosure.

In a preferable embodiment of this disclosure, a manner of speech feature extraction includes:.

Specifically, the speech sample data is a speech signal, and the speech signal is a time domain signal. A processor cannot directly process the time domain signal, so that speech framing and windowing need to be performed on the speech sample data, to obtain the at least two speech frames of the speech sample data, so as to convert the time domain signal into a frequency domain signal that may be processed by the processor, as shown in <FIG>. Fast Fourier transformation (FFT) is then performed on each speech frame to obtain the discrete power spectra corresponding to the speech frames, and logarithmic calculation is then performed on the discrete power spectra to obtain the logarithmic power spectra respectively corresponding to the speech frames, so as to obtain the speech features respectively corresponding to the speech frames. A set of all the speech features is a speech feature sequence corresponding to the speech sample data. Feature inverse transformation processing is performed on the speech feature sequence, to convert the speech feature sequence in a frequency domain into a speech signal in a time domain.

A manner in which feature extraction is performed on the first speech sample data and a manner in which feature extraction is performed on the second speech sample data are the same. Therefore, for ease of description, the first speech sample data and the second speech sample data are collectively referred to as speech sample data in the embodiments of this disclosure.

Transmit the first speech enhancement data to a recipient, and update the obtained speech enhancement parameter by using the first speech enhancement parameter to obtain an updated speech enhancement parameter, the updated speech enhancement parameter is used for performing, in a case that second speech data transmitted by the sender is received, speech enhancement on the second speech data based on the updated speech enhancement parameter.

Generally, when a noise reduction model corresponding to a speaker is obtained through training, adaptive training needs to be performed, and adaptive training requires a relatively large data volume, so that adaptive training requires a relatively long time and has a relatively low efficiency.

In the embodiments of this disclosure, it is only required to update the obtained speech enhancement parameter by using the first speech enhancement parameter to obtain the updated speech enhancement parameter. In this way, adaptive training does not need to be performed.

Specifically, after the first speech enhancement parameter is determined based on the first speech data, if a storage does not store the speech enhancement parameter corresponding to the sender, the first speech enhancement parameter may be used as the speech enhancement parameter corresponding to the sender and stored in the storage; and if the storage has stored the speech enhancement parameter corresponding to the sender, the first speech enhancement parameter may be used to replace the stored speech enhancement parameter.

In a case that the second speech data transmitted by the sender is received, speech enhancement may be performed on the second speech data based on the first speech enhancement parameter, namely, the updated speech enhancement parameter. In this way, the server may unidirectionally train the trained LSTM model continuously based on the latest speech data transmitted by the sender, to update the speech enhancement parameter corresponding to the sender continuously, so that a degree of matching between the speech enhancement parameter and the sender is increasingly high, and a speech enhancement effect for the sender is also increasingly good.

In addition, the server transmits the first speech enhancement data obtained through speech enhancement to the recipient, and the recipient plays the first speech enhancement data after receiving the first speech enhancement data.

An execution sequence of a step in which the server updates the speech enhancement parameter and a step in which the server transmits the speech enhancement data may be a sequential sequence or a parallel sequence. In practice, the execution sequence may be set according to an actual requirement, and is not limited in the embodiments of this disclosure.

For ease of understanding, detailed description is made by using the following example in the embodiments of this disclosure.

It is assumed that a user A, a user B, and a user C perform a telephone conference, a trained LSTM model is being run on a server, the trained LSTM model includes a universal speech enhancement parameter, and neither a storage in the server nor another storage includes a speech enhancement parameter of the user A.

In this case, after the user A speaks out a first statement, a terminal device corresponding to the user A transmits the first statement to the server. After receiving the first statement of the user A, the server searches for a speech enhancement parameter corresponding to the user A. The speech enhancement parameter of the user A cannot be obtained since neither the storage of the server nor another storage includes the speech enhancement parameter of the user A. Therefore, the universal speech enhancement parameter of the trained LSTM model is obtained, and speech enhancement is performed on the first statement by using the universal speech enhancement parameter to obtain an enhanced first statement, and the enhanced first statement is transmitted to terminal devices corresponding to the user B and the user C. In addition, the trained LSTM model is unidirectionally trained by using the first statement, to obtain a first speech enhancement parameter of the user A and store the first speech enhancement parameter.

After the user A speaks out a second statement, the terminal device corresponding to the user A transmits the second statement to the server. After receiving the second statement of the user A, the server searches for the speech enhancement parameter corresponding to the user A. In this case, the speech enhancement parameter is found successfully, the first speech enhancement parameter of the user A is obtained, and the universal speech enhancement parameter in the trained LSTM model is replaced with the first speech enhancement parameter to obtain an updated LSTM model. Speech enhancement is then performed on the second statement by using the updated LSTM model to obtain an enhanced second statement, and the enhanced second statement is transmitted to the terminal devices corresponding to the user B and the user C. In addition, the updated LSTM model is unidirectionally trained by using the second statement, to obtain a second speech enhancement parameter of the user A, and the first speech enhancement parameter is replaced with the second speech enhancement parameter. For speech enhancement processes on subsequent statements, the rest may be deduced by analogy, and details are not described herein again.

Further, the server may further unidirectionally train the trained LSTM model continuously based on the latest speech data transmitted by the sender, to update the speech enhancement parameter corresponding to the sender continuously, so that a degree of matching between the speech enhancement parameter and the sender is increasingly high, and a speech enhancement effect for the sender is also increasingly good. In addition, in the continuous unidirectional training process, only the speech enhancement parameter needs to be trained, and it is unnecessary to train the entire trained LSTM model or an entire layer in the model, thereby reducing training costs and improving a training speed.

<FIG> is a schematic structural diagram of a speech data processing apparatus. As shown in <FIG>, the apparatus may include:.

In an exemplary embodiment of this disclosure, the obtaining module is specifically configured to:
obtain a pre-stored speech enhancement parameter corresponding to the sender; and obtain a preset speech enhancement parameter in a case that the speech enhancement parameter corresponding to the sender is not obtained.

In a case that the speech enhancement parameter corresponding to the sender is not obtained, the update module is further configured to update the obtained preset speech enhancement parameter based on the first speech enhancement parameter to obtain the updated speech enhancement parameter, and use the first speech enhancement parameter as the speech enhancement parameter corresponding to the sender.

In a case that the speech enhancement parameter corresponding to the sender is obtained, the update module is further configured to update the speech enhancement parameter corresponding to the sender by using the first speech enhancement parameter to obtain the updated speech enhancement parameter.

In some embodiments, in a case that the speech enhancement parameter corresponding to the sender is not obtained, the processing module is further configured to perform speech enhancement on the first speech data based on the preset speech enhancement parameter to obtain the first speech enhancement data.

The processing module includes a feature sequence processing submodule, a batch processing calculation submodule, and a feature inverse transformation processing submodule.

In a case that the speech enhancement parameter corresponding to the sender is not obtained, the feature sequence processing submodule is configured to perform feature sequence processing on the first speech data by using a trained speech enhancement model to obtain a first speech feature sequence, the speech enhancement model being provided with the preset speech enhancement parameter;.

In a case that the speech enhancement parameter corresponding to the sender is obtained, the processing module is further configured to perform speech enhancement on the first speech data based on the speech enhancement parameter corresponding to the sender to obtain the first speech enhancement data.

In a case that the speech enhancement parameter corresponding to the sender is obtained, the feature sequence processing submodule is configured to perform feature sequence processing on the first speech data by using a trained speech enhancement model to obtain a second speech feature sequence;.

The trained speech enhancement model is generated by:.

A manner of performing the speech feature extraction includes:.

The speech data processing apparatus may perform the speech data processing method shown in the first embodiment of this disclosure, implementation principles are similar, and details are not described herein again.

In a case that first speech data transmitted by a sender is received, a speech enhancement parameter is obtained; speech enhancement is then performed on the first speech data based on the obtained speech enhancement parameter to obtain first speech enhancement data, and a first speech enhancement parameter is determined based on the first speech data; and then the obtained speech enhancement parameter is updated by using the first speech enhancement parameter to obtain an updated speech enhancement parameter, the updated speech enhancement parameter is used for performing, in a case that second speech data transmitted by the sender is received, speech enhancement on the second speech data based on the updated speech enhancement parameter, and the first speech enhancement data is transmitted to a recipient. In this way, the server may perform speech enhancement on speech data of the sender based on the speech enhancement parameter corresponding to the sender. Speech enhancement parameters corresponding to different senders are different, so that speech enhancement effects achieved by performing speech enhancement for the different senders are also different. In this way, in a case of not requiring multiple models, speech enhancement is still targeted and only speech enhancement parameters need to be stored without storing multiple models, thereby having a relatively low storage capacity requirement.

An electronic device is provided in still another embodiment of this disclosure. The electronic device includes a memory and a processor, and at least one program is stored in the memory and is configured to, when executed by the processor, implement the following steps. In the embodiments of this disclosure, in a case that first speech data transmitted by a sender is received, obtaining a speech enhancement parameter; then, performing speech enhancement on the first speech data based on the obtained speech enhancement parameter to obtain first speech enhancement data, and determining a first speech enhancement parameter based on the first speech data; and then, updating the obtained speech enhancement parameter by using the first speech enhancement parameter to obtain an updated speech enhancement parameter, where the updated speech enhancement parameter is used for performing, in a case that second speech data transmitted by the sender is received, speech enhancement on the second speech data based on the updated speech enhancement parameter, and transmitting the first speech enhancement data to a recipient. In this way, the server may perform speech enhancement on speech data of the sender based on the speech enhancement parameter corresponding to the sender. Speech enhancement parameters corresponding to different senders are different, so that speech enhancement effects achieved by performing speech enhancement for the different senders are also different. In this way, in a case of not requiring multiple models, speech enhancement is still targeted and only speech enhancement parameters need to be stored without storing multiple models, thereby having a relatively low storage capacity requirement.

In some embodiments, an electronic device is provided. As shown in <FIG>, an electronic device <NUM> shown in <FIG> includes a processor <NUM> and a memory <NUM>. The processor <NUM> and the memory <NUM> are connected, for example, are connected by a bus <NUM>. The electronic device <NUM> may further include a transceiver <NUM>. In practice, there may be one or more transceivers <NUM>. The structure of the electronic device <NUM> does not constitute a limitation on this embodiment of this disclosure.

The processor <NUM> may be a CPU, a general-purpose processor, a DSP, an ASIC, a FPGA or another programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor may implement or perform various examples of logic blocks, modules, and circuits described with reference to content disclosed in this disclosure. The processor <NUM> may be alternatively a combination to implement a computing function, for example, may be a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

The bus <NUM> may include a channel, to transmit information between the foregoing components. The bus <NUM> may be a PCI bus, an EISA bus, or the like. The bus <NUM> may be classified into an address bus, a data bus, a control bus, and the like. For ease of description, the bus in <FIG> is represented by using only one bold line, but which does not indicate that there is only one bus or one type of bus.

The memory <NUM> may be a ROM or another type of static storage device that can store static information and a static instruction; or a RAM or another type of dynamic storage device that can store information and an instruction; or may be an EEPROM, a CD-ROM or another compact-disc storage medium, optical disc storage medium (including a compact disc, a laser disk, an optical disc, a digital versatile disc, a Blu-ray disc, or the like) and magnetic disk storage medium, another magnetic storage device, or any other medium that can be configured to carry or store expected program code in the form of an instruction or a data structure and that is accessible by a computer, but is not limited thereto.

The memory <NUM> is configured to store application program code for performing the solutions of this disclosure, and the application program code is executed under control of the processor <NUM>. The processor <NUM> is configured to execute the application program code stored in the memory <NUM> to implement the content shown in any one of the foregoing method embodiments.

The electronic device includes, but is not limited to, mobile terminals such as a mobile phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a portable Android device (PAD), a portable multimedia player (PMP), an in-vehicle terminal (such as an in-vehicle navigation terminal) and fixed terminals such as a digital television (TV) and a desktop computer.

According to still another embodiment of this disclosure, a computer-readable storage medium is provided, storing a computer program, the computer program, when run on a computer, causing the computer to perform the method described in the foregoing method embodiments. In the embodiments of this disclosure, in a case that first speech data transmitted by a sender is received, a speech enhancement parameter is obtained; speech enhancement is then performed on the first speech data based on the obtained speech enhancement parameter to obtain first speech enhancement data, and a first speech enhancement parameter is determined based on the first speech data; and then the obtained speech enhancement parameter is updated by using the first speech enhancement parameter to obtain an updated speech enhancement parameter, the updated speech enhancement parameter is used for performing, in a case that second speech data transmitted by the sender is received, speech enhancement on the second speech data based on the updated speech enhancement parameter, and the first speech enhancement data is transmitted to a recipient. In this way, the server may perform speech enhancement on speech data of the sender based on the speech enhancement parameter corresponding to the sender. Speech enhancement parameters corresponding to different senders are different, so that speech enhancement effects achieved by performing speech enhancement for the different senders are also different. In this way, in a case of not requiring multiple models, speech enhancement is still targeted and only speech enhancement parameters need to be stored without storing multiple models, thereby having a relatively low storage capacity requirement.

It is to be understood that, although the steps in the flowchart in the accompanying drawings are sequentially shown according to indication of an arrow, the steps are not necessarily sequentially performed according to a sequence indicated by the arrow. Unless explicitly specified in this specification, execution of the steps is not strictly limited in the sequence, and the steps may be performed in other sequences. In addition, at least some steps in the flowcharts in the accompanying drawings may include multiple substeps or multiple stages. The substeps or the stages are not necessarily performed at the same moment, but may be performed at different moments. The substeps or the stages are not necessarily performed in sequence, but may be performed in turn or alternately with another step or at least some of substeps or stages of the another step.

Claim 1:
A speech data processing method, performed by a server, the method comprising:
receiving first speech data transmitted by a sender and obtaining a speech enhancement parameter (S101);
performing speech enhancement on the first speech data based on the obtained speech enhancement parameter to obtain first speech enhancement data, and determining a first speech enhancement parameter based on the first speech data (S102); and
transmitting the first speech enhancement data to a recipient, and updating the obtained speech enhancement parameter by using the first speech enhancement parameter to obtain an updated speech enhancement parameter, wherein the updated speech enhancement parameter is used for performing, in a case that second speech data transmitted by the sender is received, speech enhancement on the second speech data based on the updated speech enhancement parameter (S103),
wherein in a case that the speech enhancement parameter corresponding to the sender is not obtained, the performing speech enhancement on the first speech data based on the obtained speech enhancement parameter to obtain first speech enhancement data (S102) comprises:
obtaining a preset speech enhancement parameter,
performing speech enhancement on the first speech data based on the preset speech enhancement parameter to obtain the first speech enhancement data,
performing feature sequence processing on the first speech data by using a trained speech enhancement model to obtain a first speech feature sequence, the speech enhancement model being provided with the preset speech enhancement parameter;
performing batch processing calculation on the first speech feature sequence by using the preset speech enhancement parameter to obtain a processed first speech feature sequence and the first speech enhancement parameter; and
performing feature inverse transformation processing on the processed first speech feature sequence to obtain the first speech enhancement data.