Patent Description:
The present Application relates to speech recognition and deep machine learning. More specifically, example embodiment(s) described below relate to improving neural network architecture for better separation of speech sources.

Identifying individual speech sources from mixture speech has been challenging. Learning from a large amount of data has led to some progress in such identification. It can be helpful to further utilize deep machine learning to improve the separation of speech sources.

The example embodiment(s) of the present invention are illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings in which:.

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the example embodiment(s). It will be apparent, however, that the example embodiment(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the example embodiment(s).

Embodiments are described in sections below according to the following outline:.

A speech separation method according to claim <NUM>, a computer-readable storage medium according to claim <NUM>, and a system according to claim <NUM> are disclosed.

<FIG> illustrates example components of a speech separation server computer. The figure is for illustration purposes only, and the server <NUM> can comprise fewer or more functional or storage components. Each of the functional components can be implemented as software components, general or specific-purpose hardware components, firmware components, or any combination thereof. A storage component can be implemented using any of relational databases, object databases, flat file systems, or JavaScript Object Notation (JSON) stores. A storage component can be connected to the functional components locally or through the networks using programmatic calls, remote procedure call (RPC) facilities or a messaging bus. A component may or may not be self-contained. Depending upon implementation-specific or other considerations, the components may be centralized or distributed functionally or physically.

The server <NUM> can comprise data collection instructions <NUM>, neural model management instructions <NUM>, neural model training instructions <NUM>, and neural model execution instructions <NUM>. In addition, the server <NUM> can comprise a server database <NUM>.

The data collection instructions <NUM> enable collection of training data, validation data, or actual data comprising mixture audio waveforms from which individual waveforms corresponding to different speech sources are to be identified. The training data and validation data, including both the mixture audio waveforms and the corresponding individual waveforms, can be retrieved from public data sources. The data collection instructions <NUM> further enable collection of various user preferences or system configurations related to the generation, processing, or use of the training data, validation data, or actual data.

The neural model management instructions <NUM> enable management of different neural models, including storing or updating multiple convolutional neural networks having different structures. Each neural "model" in this context refers to an electronic digitally stored set of executable instructions and data values, associated with one another, which are capable of receiving and responding to a programmatic or other digital call, invocation, or request for resolution based upon specified input values, to yield one or more stored or calculated output values that can serve as the basis of computer-implemented recommendations, output data displays, or machine control, among other things. The management of different neural models includes retrieving and storing system configurations or user preferences related to the training or execution of the neural models, such as the selection of the training algorithm or the objective function.

The neural model training instructions <NUM> enable training or building of the different neural models. Based on the data obtained via the data collection instructions <NUM>, and the relevant parameter values for the training process obtained via the neural model management instructions <NUM>, the different neural models can be trained, and the results of the training, including values of various parameters of the neural models, can be stored for future use via the neural model management instructions <NUM>.

The neural model execution instructions <NUM> enable execution of the different neural models. Using the trained neural models obtained via the neural model training instructions <NUM> based on the relevant parameter values for the execution process obtained via the neural model management instructions <NUM>, actual data of mixture audio waveforms can be separated into individual waveforms corresponding to different speech sources. The neural model execution instructions <NUM> further enable storing or transmitting the results of executing the neural models, such as the individual waveforms.

The server database <NUM> is programmed or configured to manage storage of and access to relevant data, such as the training data, validation data, actual data, neural models, components of training procedures, system configurations, user preferences, or results of training or executing the neural models.

Recent publications have reported promising results from tackling source separation in the waveform domain. Conv-TasNet is an end-to-end neural network for speech source separation in the waveform domain, as described in <NPL>.

<FIG> illustrates the Conv-TasNet neural network. The Conv-TasNet neural network ("system" or "Conv-TasNet) comprises an encoder <NUM>, a separator <NUM>, and a decoder <NUM>. The system is programmed to take a mixture audio waveform <NUM> as input and produces individual waveforms <NUM>. Specifically, the system is programmed to separate C individual speech sources sc ∈ RT, where c ∈ {<NUM>, <NUM>,. , C}, from a single-channel mixture of speech x ∈ RT, where T represents the length of the waveform and <MAT>.

The encoder <NUM> linearly maps the mixture audio waveform <NUM> into a learned latent space. Specifically, the mixture audio waveform <NUM> is segmented into K overlapping frames xk ∈ RL, k = <NUM>, <NUM>,. , K, each of length L with an overlap with S with the previous frame. Then, the linear transform is defined as: <MAT> where U ∈ RN×L contains N learnable basis filters or kernels of size L in its rows, X ∈ RL×K stores the K frames of length L in columns, and E ∈ RN×K denotes the latent space representation of the mixture audio waveform <NUM>, where each frame is represented by N values. This encoder can thus be implemented as a <NUM>-D convolution with N kernels. In one example, N = <NUM>, L = <NUM>, and S = <NUM>, corresponding to <NUM> basis filters at a sample rate of <NUM>, and <NUM> overlap.

The separator <NUM> predicts a representation for each speech source by learning a mask in this latent space. The separator <NUM> comprises stacked dilated convolutional blocks with exponentially increasing dilation factors, and each stack can be repeated multiple times. A deep stack of dilated convolutions enables the separator to have a large temporal context with a compact model size. The element-wise multiplication <NUM> then applies the individual masks produced by the separator <NUM> to the mixture audio waveform <NUM> to generate the individual latent space representations.

The decoder linearly transforms the latent space representation of each estimated clean source c = <NUM>, <NUM>,. , C to the time domain: <MAT> where V ∈ RN×L contains N decoder basis filters (not tied with the encoder U), Dc ∈ RN×K is the representation of the cth estimated source predicted by the separator, and Ŝc ∈ RK×L contains K frames of the reconstructed signal. The entire time domain waveform ŝc is obtained by overlap-and-add of the rows of Ŝc. Similar to the encoder, the decoder can be implemented as a <NUM>-D (transposed) convolution. All the Conv-TasNet building blocks are jointly optimized.

<FIG> illustrates an example neural network for speech separation in accordance with the disclosed embodiments. In some embodiments, the neural network comprises an encoder and a decoder. The encoder or the decoder can be combined with the separator in Conv-TasNet to form an end-to-end neural network for speech separation or with another separator configured to receive the same types of input data and produce the same types of output data as the separator in Conv-TasNet, such as later improvements of Conv-TasNet.

In some embodiments, the encoder comprises a <NUM>-D convolution <NUM>, which can be identical or similar to the encoder <NUM> in Conv-TasNet, which receives a mixture audio waveform and produces the latent space representation of the mixture audio waveform. The <NUM>-D convolution <NUM> is followed by a first stack of convolutional layers, such as convolutional layers <NUM>, <NUM>, and <NUM>. Each convolutional layer comprises a <NUM>-D convolution and a nonlinear activation function. For example, each convolutional layer may have a <NUM>-D convolution followed by a PReLU, the PReLU being known to someone skilled in the art: <MAT> where * denotes the convolution operator, i = <NUM>, <NUM>,. , I denotes the convolutional layer index, Ui ∈ RN×N×g denotes N learnable kernels, E<NUM> is the output of the <NUM>-D convolution <NUM>, and Ei ∈ RN×K is output of the ith convolutional layer. As each Ei denotes a latent space representation of the mixture audio waveform <NUM>, where each frame is represented by N values, each kernel effectively has a size of Nxg. The convolution with respect to the mixture audio waveform <NUM> continues over the time dimension. A stride of S can be used in the convolution. For example, the value of g can be <NUM>, the value of S can be <NUM>, and the size of the stack can be <NUM>. This first stack of convolutional layers hierarchically transforms the mixture audio waveform <NUM> into a nonlinear latent space.

As the kernels in these convolutional layers have relatively small sizes for convolution in the time dimension, the initial convolutional layers extract more high-resolution or specific patterns in the mixture audio waveform <NUM>, while the subsequent convolutional layers extract more low-resolution or abstract patterns. Such abstract, nonlinear latent representations allow the separator to make predictions in a space in which the salient components of the mixture are more easily distinguished and extracted. The small strides tend to improve the temporal resolution and can be combined with small kernel sizes to achieve good coverage. The controlled number of filters and size of the first stack (number of convolutional layers) then help achieve a good tradeoff between accuracy and speed. Compared to the convolutional blocks in the separator <NUM> in Conv-TasNet, for example, the structure of each convolutional layer and of the entire first stack of convolutional layers is relatively simple to help retain the performance advantage of the encoder, leaving more of the execution time to a separator module. Such deep learning to effectively and efficiently characterize the speech signals for speech separation is missing from prior encoders or decoders.

In the invention, the structure of the decoder mirrors the structure of the encoder. The predicted representations generated from the separator <NUM>, which can be identical or similar to the separator <NUM> in Conv-TasNet, are first processed by a second stack of convolutional layers, such as convolutional layers <NUM>, <NUM>, and <NUM>. Each convolutional layer comprises a dimension-preserving <NUM>-D transposed convolution with a nonlinear activation function and otherwise operates in the same fashion as a convolutional layer in the encoder. The kernels in these convolutional layers of the decoder can be learned separately from the kernels in those convolutional layers of the encoder. Finally, the second stack of convolutional layers is followed by <NUM>-D transposed convolution <NUM>, which can be identical or similar to the decoder <NUM> in Conv-TasNet, to produce the time-domain estimated source signals.

<FIG> illustrates an example neural network for speech separation in accordance with the disclosed embodiments. In some embodiments, the neural network comprises an encoder and a decoder, similar to the neural network illustrated in <FIG>. One difference is that a GLU known to someone skilled in the art or a modified GLU, as discussed below, is used in place of each PReLU. A GLU generally has many more learnable parameters than a PReLU. Similar to attention, a GLU relies on a learned gate to model the relative importance of the kernels. Therefore, the encoder comprises a <NUM>-D convolution <NUM>, which can be identical or similar to the encoder <NUM> in Conv-TasNet, followed by a first stack of convolutional layers, such as convolutional layers <NUM>, <NUM>, and <NUM>. Each convolutional layer comprises a <NUM>-D convolution and a nonlinear activation function, such as the GLU. The structure of the decoder mirrors the structure of the encoder. The predicted representations from the separator <NUM>, which can be identical or similar to the separator <NUM> in Conv-TasNet, are first processed by a second stack of convolutional layers, such as <NUM>, <NUM>, and <NUM>. Each convolutional layer comprises a dimension-preserving <NUM>-D transposed convolution with a nonlinear activation function, such as the GLU. Finally, the second stack of convolutional layers is followed by <NUM>-D transposed convolution <NUM>, which is the decoder in Conv-TasNet.

<FIG> illustrates an example convolutional layer having a modified GLU in accordance with the disclosed embodiments. In some embodiments, the convolutional layer, such as <NUM> illustrated in <FIG>, comprises a GLU, with a first pathway through the <NUM>-D convolution <NUM> and a second pathway through the <NUM>-D convolution <NUM> and the Sigmoid activation function <NUM> to fulfill the gating mechanism <NUM>. The convolutional layer further comprises a global layer normalization <NUM> known to someone skilled in the art before the Sigmoid activation function <NUM> known to someone skilled in the art, forming a modified GLU, to speed up training.

In some embodiments, the number of convolutional layers having nonlinear activation included in a neural network for an encoder or decoder, such as the one illustrated in <FIG> or <FIG>, can be more than three. In one embodiment, the nonlinear activation function included in the neural network can instead be a leaky Rectified Linear Unit (ReLU), a Sigmoid function, a TanH function, a Gaussian error linear unit (GELU), a softplus function, an exponential linear unit (ELU), an arcTan function, a square nonlinearity function (SQNL), or other nonlinear activation functions known to someone skilled in the art. In one embodiment, multiple convolutional layers have different nonlinear activation functions. In one embodiment, every convolutional layer has nonlinear activation. In one embodiment, not every convolutional layer following the first convolutional layer, which has linear activation or no linear activation, has nonlinear activation.

<FIG> illustrates an example neural network for speech separation in accordance with the disclosed embodiments. In some embodiments, the neural network comprises an encoder and a decoder. The encoder comprises a <NUM>-D convolution <NUM>, which can be identical or similar to the encoder in Conv-TasNet. The <NUM>-D convolution <NUM> is followed by a first stack of convolutional layers, such as <NUM>, <NUM>, and <NUM>, regulated by a nonlinear unit, such as a PReLU or a GLU <NUM>. Each convolutional layer in the encoder is set up with skip and residual connections, such skip and residual connections known to someone skilled in the art. Skip connections are known to avoid vanishing gradients or eliminate singularities in deep learning networks, making the learning process possible or easier. The residual connection leads to the next convolutional layer and the skip connection leads to a summation <NUM> feeding into the nonlinear unit <NUM>. The structure of the decoder mirrors the structure of the encoder. The predicted representations from the separator <NUM>, which can be identical or similar to the separator in Conv-TasNet, are first processed by a second stack of convolutional layers, such as <NUM>, <NUM>, and <NUM>, regulated by a nonlinear unit <NUM>. Each convolutional layer in the decoder is set up with skip and residual connections, as in the encoder. Finally, the second stack of convolutional layers is followed by <NUM>-D transposed convolution <NUM>, which can be identical or similar to the decoder <NUM> in Conv-TasNet.

<FIG> illustrates an example convolutional layer with skip and residual connections in accordance with the disclosed embodiments. In some embodiments, the convolutional layer, such as <NUM> illustrated in <FIG>, comprises a depth-wise convolution <NUM>, the depth-wise convolution or depth-wise separable convolution known to someone skilled in the art, followed by a nonlinear activation function <NUM>, such as a PReLU. The depth-wise convolution <NUM> can have the same parameters as the <NUM>-D convolutions discussed. For example, the kernel used in each convolution operation has a size of g. The nonlinear activation function <NUM> is then followed by a first path through a first point convolution <NUM>, the point convolution known to someone skilled in the art for the residual connection <NUM> and a second path through the second point convolution <NUM> for the skip connection <NUM>. In other embodiments, the depth-wise convolution <NUM> the the PReLU <NUM> may be replaced by another nonlinear activation function, such as a GLU or modified GLU illustrated in <FIG>.

In some embodiments, the encoder and the decoder are trained together with the separator. A procedure for such end-to-end training is described in the Conv-TasNet publication cited above. Instead of the scale-invariant signal-to-noise ratio (SI-SNR) applied in that procedure, the SI-SNR with a power-law term is used as the objective function. Specifically, in an effort to constrain the scale of the predicted sources (in terms of the amplitude of the waveforms) from the deep encoder or decoder, the SI-SNR objective function is augmented with a power-law term that encourages the model being trained to predict spectra that are of similar magnitude to the ground truth. Power laws are known to correlate with human perception. The augmented objective function is written as: <MAT> where <MAT> ŝc and sc are discussed above. STFT stands for short-time Fourier transform, α is a perceptual exponent that maps the raw signal energy in each time-frequency bin to a perceptual domain. α is meant to map the spectra to a domain in which human perception of loudness is more linear, e.g., doubling the value in this domain should double the perceived loudness of the signal. β is a weighting factor that represents the weight of the P-law in the total loss. The value of α or β is between <NUM> and <NUM>. In certain embodiments, β is set to <NUM> and α is set to <NUM>.

In some embodiments, the training data spans a certain duration, such as <NUM> hours, and the validation data spans a certain duration, such as <NUM> hours. The training data or the validation data can be obtained from randomly mixing utterances from a number of speakers at randomly selected signal-to-noise ratios (SNRs). For example, the training data and the validation data can be generated from randomly mixing utterances from <NUM> speakers at randomly selected SNRs between -<NUM> and <NUM> dB. The waveforms can be sampled at a certain frequency, such as <NUM>,<NUM>.

<FIG> and <FIG> each illustrate an example process performed with a speech separation server computer in accordance with some embodiments described herein. <FIG> and <FIG> are each shown in simplified, schematic format for purposes of illustrating a clear example and other embodiments may include more, fewer, or different elements connected in various manners. <FIG> and <FIG> are each intended to disclose an algorithm, plan or outline that can be used to implement one or more computer programs or other software elements which when executed cause performing the functional improvements and technical advances that are described herein. Furthermore, the flow diagrams herein are described at the same level of detail that persons of ordinary skill in the art ordinarily use to communicate with one another about algorithms, plans, or specifications forming a basis of software programs that they plan to code or implement using their accumulated skill and knowledge.

Referring to <FIG>, in step <NUM>, the server is programmed to receive a mixture audio signal comprising audio signals from a plurality of audio sources. Each of the plurality of audio signals can comprise a waveform in a time domain.

In step <NUM>, the server is programmed to transform the mixture audio signal into an encoded representation by an encoder convolutional neural network (CNN) with multiple convolutional layers and nonlinear activation.

The encoder CNN comprises a convolutional layer with <NUM>-D convolutional operation.

In some embodiments, the nonlinear activation comprises a PReLU, a GLU, a GLU with normalization, a leaky ReLU, a Sigmoid function, or a TanH function. In other embodiments, the encoder CNN comprises three convolutional layers, each convolutional layer of the three convolutional layers having nonlinear activation.

In some embodiments, the encoder CNN comprises one or more convolutional layers with residual and skip connections. The encoder CNN can further comprise nonlinear activation following the one or more convolutional layers with residual and skip connections. In addition, at least three convolutional layers of the one or more convolutional layers can comprise one or more nonlinear activation functions.

In step <NUM>, the server is programmed to separate the encoded representation into a plurality of individual representations corresponding to the plurality of audio sources. The separating can be performed by a separator CNN comprising stacked dilated convolutional blocks.

In step <NUM>, the server is programmed to transform the plurality of individual representations into a plurality of audio signals corresponding to the plurality of audio sources by a decoder CNN with multiple convolutional layers and nonlinear activation. A structure of the decoder CNN corresponds to a structure of the encoder CNN.

The server is programmed to receive a plurality of sample audio signals corresponding to the plurality of audio sources. The server is programmed to further build the encoder CNN from the plurality of sample audio signals using an objective function comprising SI-SNR with permutation-invariant training.

Referring to <FIG>, in step <NUM>, the server is programmed to receive a mixture audio signal comprising audio signals from a plurality of audio sources, the mixture audio signal spanning a range of time in a time domain.

In step <NUM>, the server is programmed to divide the mixture audio signal into a plurality of segments of a specific length, at least two segments of the plurality of segments spanning overlapping sub-ranges of the range of time in the time domain.

In step <NUM>, the server is programmed to apply an encoder CNN to each segment of the plurality of segments to generate an encoded representation of the segment, the encoder CNN having multiple convolutional layers and nonlinear activation.

In some embodiments, the nonlinear activation comprises a PReLU, a GLU, a GLU with normalization, a leaky ReLU, a Sigmoid function, or a TanH function. The encoder CNN comprises three convolutional layers, each convolutional layer of the three convolutional layers having nonlinear activation. In yet other embodiments, the encoder CNN comprises one or more convolutional layers with residual and skip connections.

In step <NUM>, the server is programmed to transmit the plurality of encoded representations of the plurality of segments to a separator that generates a plurality of individual representations corresponding to the plurality of audio sources.

The server is programmed to receive the plurality of individual representations from the separator. The server is further programmed to apply a decoder CNN to each individual representation of the plurality of individual representations to generate an audio signal that spans the range of time, the decoder CNN having multiple convolutional layers and nonlinear activation. The server is next programmed to transmit the plurality of audio signals.

Experiments were conducted using the exemplary nonlinear deep encoder/decoder with I number of layers. The first layer was equivalent to the original Conv-TasNet encoder, where a linear transformation was applied to frames of length L and stride S. It was implemented via a <NUM>-D convolutional layer with N kernels. The first layer is followed by a stack of <NUM>-<NUM><NUM>-D convolutional layers, with each layer having N kernels of size Nx3 and a PReLU according to equation <NUM> above (represented below). <MAT> where * denotes the convolution operator, i = <NUM>, <NUM>,. , I denotes the convolutional layer index, Ui ∈ RN×N×g denotes N learnable kernels, E<NUM> is the output of the <NUM>-D convolution <NUM>, and Ei ∈ RN×K is output of the ith convolutional layer.

A first variant of the exemplary nonlinear deep encoder/decoder increases the temporal context of the deep layers by employing dilated convolutions.

A second variant of the exemplary nonlinear deep encoder/decoder further increases the capacity by using GLUs to replace the PReLUs as the activation function, as noted above. Furthermore, a global layer normalization was inserted before the sigmoid nonlinearity in the GLU to speed up training.

In a first experiment, the performance of the nonlinear deep encoder/decoder and its variants for speech source separation was evaluated on the commonly used WSJ0 <NUM>-speaker (WSJ0-2mix) database known to those skilled in the art. The results of the first experiment are summarized in Table <NUM>, below.

Training (<NUM> hours) and validation sets (<NUM> hours) were created by randomly mixing utterances from <NUM> speakers at randomly selected SNRs between -<NUM> and <NUM> dB. All waveforms were sampled at <NUM>. Table <NUM> summarizes the SI-SNR improvement (SI-SNRi) before and after speech separation on the test set (<NUM> hours, <NUM> unseen speakers). First, the original Conv-TasNet was successfully reproduced to build upon their result (row <NUM>, Table <NUM>). Second, note that the BigConv-TasNet (row <NUM>, Table <NUM>), a modified Conv-TasNet with double the number of kernel filters and four (instead of three) temporal convolutional networks (TCNs) in the separator, is not able to outperform the original Conv-TasNet. And third, the deep encoder/decoder (row <NUM>, Table <NUM>) provides <NUM> dB improvement over the baseline. This result denotes the importance of the architecture itself because the objective metrics did not improve by simply increasing the capacity of the model. Therefore, it can be seen in Table <NUM> that the nonlinear deep encoder/decoder and its variants significantly outperform both the original Conv-TasNet, and BigConv-TasNet.

According to one embodiment, the techniques described herein are implemented by at least one computing device. The techniques may be implemented in whole or in part using a combination of at least one server computer and/or other computing devices that are coupled using a network, such as a packet data network. The computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as at least one application-specific integrated circuit (ASIC) or field programmable gate array (FPGA) that is persistently programmed to perform the techniques, or may include at least one general purpose hardware processor programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the described techniques. The computing devices may be server computers, workstations, personal computers, portable computer systems, handheld devices, mobile computing devices, wearable devices, body mounted or implantable devices, smartphones, smart appliances, internetworking devices, autonomous or semi-autonomous devices such as robots or unmanned ground or aerial vehicles, any other electronic device that incorporates hard-wired and/or program logic to implement the described techniques, one or more virtual computing machines or instances in a data center, and/or a network of server computers and/or personal computers.

<FIG> is a block diagram that illustrates an example computer system with which an embodiment may be implemented. In the example of <FIG>, a computer system <NUM> and instructions for implementing the disclosed technologies in hardware, software, or a combination of hardware and software, are represented schematically, for example as boxes and circles, at the same level of detail that is commonly used by persons of ordinary skill in the art to which this disclosure pertains for communicating about computer architecture and computer systems implementations.

Computer system <NUM> includes an input/output (I/O) subsystem <NUM> which may include a bus and/or other communication mechanism(s) for communicating information and/or instructions between the components of the computer system <NUM> over electronic signal paths. The I/O subsystem <NUM> may include an I/O controller, a memory controller and at least one I/O port. The electronic signal paths are represented schematically in the drawings, for example as lines, unidirectional arrows, or bidirectional arrows.

At least one hardware processor <NUM> is coupled to I/O subsystem <NUM> for processing information and instructions. Hardware processor <NUM> may include, for example, a general-purpose microprocessor or microcontroller and/or a special-purpose microprocessor such as an embedded system or a graphics processing unit (GPU) or a digital signal processor or ARM processor. Processor <NUM> may comprise an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

Computer system <NUM> includes one or more units of memory <NUM>, such as a main memory, which is coupled to I/O subsystem <NUM> for electronically digitally storing data and instructions to be executed by processor <NUM>. Memory <NUM> may include volatile memory such as various forms of random-access memory (RAM) or other dynamic storage device. Memory <NUM> also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor <NUM>. Such instructions, when stored in non-transitory computer-readable storage media accessible to processor <NUM>, can render computer system <NUM> into a special-purpose machine that is customized to perform the operations specified in the instructions.

Computer system <NUM> further includes non-volatile memory such as read only memory (ROM) <NUM> or other static storage device coupled to I/O subsystem <NUM> for storing information and instructions for processor <NUM>. The ROM <NUM> may include various forms of programmable ROM (PROM) such as erasable PROM (EPROM) or electrically erasable PROM (EEPROM). A unit of persistent storage <NUM> may include various forms of non-volatile RAM (NVRAM), such as FLASH memory, or solid-state storage, magnetic disk or optical disk such as CD-ROM or DVD-ROM, and may be coupled to I/O subsystem <NUM> for storing information and instructions. Storage <NUM> is an example of a non-transitory computer-readable medium that may be used to store instructions and data which when executed by the processor <NUM> cause performing computer-implemented methods to execute the techniques herein.

The instructions in memory <NUM>, ROM <NUM> or storage <NUM> may comprise one or more sets of instructions that are organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps. The instructions may comprise an operating system and/or system software; one or more libraries to support multimedia, programming or other functions; data protocol instructions or stacks to implement TCP/IP, HTTP or other communication protocols; file processing instructions to interpret and render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. The instructions may implement a web server, web application server or web client. The instructions may be organized as a presentation layer, application layer and data storage layer such as a relational database system using structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system or other data storage.

Computer system <NUM> may be coupled via I/O subsystem <NUM> to at least one output device <NUM>. In one embodiment, output device <NUM> is a digital computer display. Examples of a display that may be used in various embodiments include a touch screen display or a light-emitting diode (LED) display or a liquid crystal display (LCD) or an e-paper display. Computer system <NUM> may include other type(s) of output devices <NUM>, alternatively or in addition to a display device. Examples of other output devices <NUM> include printers, ticket printers, plotters, projectors, sound cards or video cards, speakers, buzzers or piezoelectric devices or other audible devices, lamps or LED or LCD indicators, haptic devices, actuators or servos.

At least one input device <NUM> is coupled to I/O subsystem <NUM> for communicating signals, data, command selections or gestures to processor <NUM>. Examples of input devices <NUM> include touch screens, microphones, still and video digital cameras, alphanumeric and other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, slides, and/or various types of sensors such as force sensors, motion sensors, heat sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors and/or various types of transceivers such as wireless, such as cellular or Wi-Fi, radio frequency (RF) or infrared (IR) transceivers and Global Positioning System (GPS) transceivers.

Another type of input device is a control device <NUM>, which may perform cursor control or other automated control functions such as navigation in a graphical interface on a display screen, alternatively or in addition to input functions. Control device <NUM> may be a touchpad, a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor <NUM> and for controlling cursor movement on display <NUM>. The input device may have at least two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. Another type of input device is a wired, wireless, or optical control device such as a joystick, wand, console, steering wheel, pedal, gearshift mechanism or other type of control device. An input device <NUM> may include a combination of multiple different input devices, such as a video camera and a depth sensor.

In another embodiment, computer system <NUM> may comprise an internet of things (IoT) device in which one or more of the output device <NUM>, input device <NUM>, and control device <NUM> are omitted. Or, in such an embodiment, the input device <NUM> may comprise one or more cameras, motion detectors, thermometers, microphones, seismic detectors, other sensors or detectors, measurement devices or encoders and the output device <NUM> may comprise a special-purpose display such as a single-line LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator or a servo.

When computer system <NUM> is a mobile computing device, input device <NUM> may comprise a global positioning system (GPS) receiver coupled to a GPS module that is capable of triangulating to a plurality of GPS satellites, determining and generating geo-location or position data such as latitude-longitude values for a geophysical location of the computer system <NUM>. Output device <NUM> may include hardware, software, firmware and interfaces for generating position reporting packets, notifications, pulse or heartbeat signals, or other recurring data transmissions that specify a position of the computer system <NUM>, alone or in combination with other application-specific data, directed toward host <NUM> or server <NUM>.

Computer system <NUM> may implement the techniques described herein using customized hard-wired logic, at least one ASIC or FPGA, firmware and/or program instructions or logic which when loaded and used or executed in combination with the computer system causes or programs the computer system to operate as a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system <NUM> in response to processor <NUM> executing at least one sequence of at least one instruction contained in main memory <NUM>. Such instructions may be read into main memory <NUM> from another storage medium, such as storage <NUM>.

Non-volatile media includes, for example, optical or magnetic disks, such as storage <NUM>. Volatile media includes dynamic memory, such as memory <NUM>. Common forms of storage media include, for example, a hard disk, solid state drive, flash drive, magnetic data storage medium, any optical or physical data storage medium, memory chip, or the like.

For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise a bus of I/O subsystem <NUM>. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infrared data communications.

Various forms of media may be involved in carrying at least one sequence of at least one instruction to processor <NUM> for execution. The remote computer can load the instructions into its dynamic memory and send the instructions over a communication link such as a fiber optic or coaxial cable or telephone line using a modem. A modem or router local to computer system <NUM> can receive the data on the communication link and convert the data to be read by computer system <NUM>. For instance, a receiver such as a radio frequency antenna or an infrared detector can receive the data carried in a wireless or optical signal and appropriate circuitry can provide the data to I/O subsystem <NUM> such as place the data on a bus. I/O subsystem <NUM> carries the data to memory <NUM>, from which processor <NUM> retrieves and executes the instructions. The instructions received by memory <NUM> may optionally be stored on storage <NUM> either before or after execution by processor <NUM>.

Communication interface <NUM> provides a two-way data communication coupling to network link(s) <NUM> that are directly or indirectly connected to at least one communication networks, such as a network <NUM> or a public or private cloud on the Internet. For example, communication interface <NUM> may be an Ethernet networking interface, integrated-services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of communications line, for example an Ethernet cable or a metal cable of any kind or a fiber-optic line or a telephone line. Network <NUM> broadly represents a local area network (LAN), wide-area network (WAN), campus network, internetwork or any combination thereof. Communication interface <NUM> may comprise a LAN card to provide a data communication connection to a compatible LAN, or a cellular radiotelephone interface that is wired to send or receive cellular data according to cellular radiotelephone wireless networking standards, or a satellite radio interface that is wired to send or receive digital data according to satellite wireless networking standards. In any such implementation, communication interface <NUM> sends and receives electrical, electromagnetic or optical signals over signal paths that carry digital data streams representing various types of information.

Network link <NUM> typically provides electrical, electromagnetic, or optical data communication directly or through at least one network to other data devices, using, for example, satellite, cellular, Wi-Fi, or BLUETOOTH technology. For example, network link <NUM> may provide a connection through a network <NUM> to a host computer <NUM>.

Furthermore, network link <NUM> may provide a connection through network <NUM> or to other computing devices via internetworking devices and/or computers that are operated by an Internet Service Provider (ISP) <NUM>. ISP <NUM> provides data communication services through a world-wide packet data communication network represented as internet <NUM>. A server computer <NUM> may be coupled to internet <NUM>. Server <NUM> broadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or computer executing a containerized program system such as DOCKER or KUBERNETES. Server <NUM> may represent an electronic digital service that is implemented using more than one computer or instance and that is accessed and used by transmitting web services requests, uniform resource locator (URL) strings with parameters in HTTP payloads, API calls, app services calls, or other service calls. Computer system <NUM> and server <NUM> may form elements of a distributed computing system that includes other computers, a processing cluster, server farm or other organization of computers that cooperate to perform tasks or execute applications or services. Server <NUM> may comprise one or more sets of instructions that are organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps. The instructions may comprise an operating system and/or system software; one or more libraries to support multimedia, programming or other functions; data protocol instructions or stacks to implement TCP/IP, HTTP or other communication protocols; file format processing instructions to interpret or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. Server <NUM> may comprise a web application server that hosts a presentation layer, application layer and data storage layer such as a relational database system using structured query language (SQL) or NoSQL, an object store, a graph database, a flat file system or other data storage.

Computer system <NUM> can send messages and receive data and instructions, including program code, through the network(s), network link <NUM> and communication interface <NUM>. The received code may be executed by processor <NUM> as it is received, and/or stored in storage <NUM>, or other non-volatile storage for later execution.

The execution of instructions as described in this section may implement a process in the form of an instance of a computer program that is being executed, and consisting of program code and its current activity. Depending on the operating system (OS), a process may be made up of multiple threads of execution that execute instructions concurrently. In this context, a computer program is a passive collection of instructions, while a process may be the actual execution of those instructions. Several processes may be associated with the same program; for example, opening up several instances of the same program often means more than one process is being executed. Multitasking may be implemented to allow multiple processes to share processor <NUM>. While each processor <NUM> or core of the processor executes a single task at a time, computer system <NUM> may be programmed to implement multitasking to allow each processor to switch between tasks that are being executed without having to wait for each task to finish. In an embodiment, switches may be performed when tasks perform input/output operations, when a task indicates that it can be switched, or on hardware interrupts. Time-sharing may be implemented to allow fast response for interactive user applications by rapidly performing context switches to provide the appearance of concurrent execution of multiple processes simultaneously. In an embodiment, for security and reliability, an operating system may prevent direct communication between independent processes, providing strictly mediated and controlled inter-process communication functionality.

Claim 1:
A computer-implemented method of separating audio signals from different speech sources, comprising:
receiving (<NUM>), by a processor, a mixture audio signal comprising audio signals from a plurality of speech sources;
transforming (<NUM>), by the processor, the mixture audio signal into an encoded representation by an encoder convolutional neural network (CNN) with multiple convolutional layers and nonlinear activation, wherein transforming the mixture audio signal into an encoded representation by an encoder convolutional neural network comprises:
transforming, by the processor, the mixture audio signal into an intermediate encoded representation using at least a first convolutional layer, the intermediate encoded representation being represented by N dimensions;
transforming, by the processor, the intermediate encoded representation to the encoded representation using at least three subsequent convolutional layers, each layer implementing one-dimensional convolutional operation with N learnable kernels and outputting an output representation with N dimensions, wherein the output representation of the final layer of said at least three convolutional layers is the encoded representation, wherein said at least three subsequent convolutional layers have nonlinear activation;
separating (<NUM>) the encoded representation into a plurality of individual representations corresponding to the plurality of speech sources;
transforming (<NUM>) the plurality of individual representations into a plurality of audio signals corresponding to the plurality of speech sources by a decoder CNN, the structure of the decoder CNN corresponding to the structure of the encoder CNN, wherein the decoder CNN comprises at least three subsequent decoder convolutional layers and a final decoder convolutional layer, each of said at least three subsequent decoder convolutional layers implementing one-dimensional convolutional operation and nonlinear activation.