Patent Description:
Further aspects and examples are provided for facilitating the understanding of the invention.

One or more of the implementations of the subject matter described herein can be implemented so as to realize one or more of the following advantages. As one example, implementations provide a new class of enhancement that conditions a generative network to focus on particular features, the salient features, in any input. Such conditioning makes the generative networks so conditioned robust to noise. As a result, the generative networks conditioned using the salient features produce more natural errors than errors resulting from generative networks not similarly conditioned. In addition, the generative networks conditioned using the salient features handle more distorted input signals without producing errors than other networks not similarly conditioned. For example, as noise levels increase in an input signal, a generative speech network conditioned using the salient features disclosed herein produces fewer errors and, if any errors are produced, produces natural sounding errors that stay in the speech manifold. In contrast, artifacts generated by other generative networks stray from the speech manifold and sound more unnatural, and therefore more noticeable, to listeners as noise level in the input increases.

As another example, salient features are compact. For example, implementations may extract a small number (e.g., ten or fewer, twelve or fewer, fifteen or fewer, twenty or fewer) of features from an input, but because the features represent salient, and therefore perceptually important, information and are independent from each other, a decoder can use the salient features to produce realistic output from the features. Put another way, salient features ignore features that are perceptually irrelevant but use significant memory allocation. This contrasts with most conventional encoders, such as variational autoencoders or VAEs, which fail to account for perceptual importance. Salient features can be used for the storage or transmission of signals, or for the manipulation of attributes of signals. For example, salient features may be used for robust coding of speech, encoding specific classes of images (e.g., human faces, handwriting, etc.), changing the identity of a speaker, resynthesizing a speech signal without noise, etc. While some previous methods have attempted to identify salient information, such previous methods require either explicit knowledge of the nuisance variables to be discarded or require pairs of equivalent signals for training; such methods do not scale as well as disclosed implementations.

As another example, disclosed implementations do not interfere with the generative nature of generative networks because implementations do not attempt to reconstruct the ground-truth, e.g., clean speech. Traditional enhancement metrics for evaluation use a reference and do not account for other solutions, which work well for networks like feed forward networks and recurrent neural networks, which try to find a good approximation of the clean speech waveform based on noisy observations and available prior knowledge. In contrast, generative networks (e.g., generative convolutional neural networks, generative deep neural networks, adversarial networks, etc.) are restricted to a manifold, or topological space. For example, generative speech networks are restricted to producing natural speech, and generative image networks are restricted to producing images. Generative networks can use a stochastic process to generate complex details that are perceptually irrelevant instead of trying to reproduce the input signal exactly. For example, in reproducing an image of a tree leaves should look correct (e.g., color, shape) but the leaves can vary in number or location from the original. In this example, the number and location are perceptually irrelevant. In other words, generative networks can provide a solution that differs considerably from the ground truth but that is equivalent to it. Current systems for enhancing generative networks are optimized at least in part to reconstruct the ground-truth input, which restricts the generative aspect of the generative network. In other words, traditional enhancement metrics do not account for these other solutions. Because disclosed implementations do not rely on reconstruction of the ground truth, disclosed implementations tend not to restrict the generative aspects and, therefore, tend to result in more natural or realistic output.

Implementations provide an enhancement to generative networks by learning to extract salient features from an input signal. Salient features are features that are shared by signals that are defined as being equivalent by a system designer. The system designer provides qualitative knowledge to the feature extraction. The qualitative knowledge enables the encoders to ignore features that are perceptually irrelevant (e.g., noise, pauses, distortions, etc., not affecting the meaning or content), extracting only those features that impact the content or meaning of the input. In other words, perceptually relevant features are features that affect the ability of humans to grasp the substance of the input and other features. Features that are perceptible but do not affect the substance are perceptually irrelevant. Thus, salient features can be described as perceptually important to humans. Such features may be small in number per input, but result in better, more realistic reconstruction, as perceived by a user.

<FIG> is a block diagram of a salient feature extraction system in accordance with an example implementation. The system <NUM> may be used to train an encoder to extract salient features from an input. The salient features capture perceptually relevant features from the input signal in a scalable fashion. The salient features are used to store or transport an encoded signal. The salient features are used to condition a generative network. The salient feature extraction system <NUM> jointly trains a set of cloned encoders <NUM>. Each encoder, e.g., <NUM>(<NUM>), <NUM>(<NUM>),. , <NUM>(N), receives as input a different input signal, e.g., <NUM>(<NUM>), <NUM>(<NUM>),. , <NUM>(N), from a set of equivalent signals <NUM>. The objective function used by the cloned encoders <NUM> encourages the encoders (<NUM>(<NUM>) to <NUM>(N)) to map their respective input to a set of unit-variance features that is identical across the cloned encoders <NUM>. Training can be supervised or unsupervised. Conventionally, supervised training uses labeled input during training. As used herein, supervised training does not refer to this conventional technique. Instead, as used herein, supervised training refers to using a reconstruction target term as an additional optimization term during training. Thus, in supervised training, the system <NUM> includes cloned decoders <NUM> that maps the salient features to a shared target signal. For ease of description, the depiction of system <NUM> in <FIG> is sometimes described as processing speech input (e.g., mel-frequency spectra), but implementations are not so limited. For example, the system <NUM> of <FIG> can process image input, video input, music input, etc..

The salient feature extraction system <NUM> may be a computing device or devices that take the form of a number of different devices, for example, a standard server, a group of such servers, or a rack server system, etc. In addition, system <NUM> may be implemented in a personal computer, for example, a laptop computer. The system <NUM> may be an example of computer device <NUM>, as depicted in <FIG> or computer device <NUM>, as depicted in <FIG>.

Although not shown in <FIG>, the system <NUM> includes one or more processors formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processors can be semiconductor-based - that is, the processors can include semiconductor material that can perform digital logic. The processors can be specialty processors, such as graphics processing units (GPUs). The system <NUM> can also include an operating system and one or more computer memories, for example a main memory, configured to store one or more pieces of data, either temporarily, permanently, semi-permanently, or a combination thereof. The memory may include any type of storage device that stores information in a format that can be read and/or executed by the one or more processors. The memory may include volatile memory, non-volatile memory, or a combination thereof, and store modules that, when executed by the one or more processors, perform certain operations. In some implementations, the modules may be stored in an external storage device and loaded into the memory of system <NUM>.

The cloned encoders <NUM> represents a plurality of machine-learned computational models or encoders. In machine learning, a computational model is organized as connected nodes, the nodes being organized into layers. The nodes perform a mapping function on provided input to produce some output. A first layer of nodes takes the input provided to the model, i.e., input from an outside source. The output of the first layer of nodes is provided as input to a second layer of nodes. The nodes in the second layer provide input to a subsequent layer, etc., until a final layer is reached. The final, or output, layer of nodes provides the output of the model. In the case of system <NUM>, the output of the encoders is a feature vector. A vector is generally an array of numbers, with each position in the array representing a different attribute. The number of array positions is referred to as the dimension of the vector. The values in each array position can be a whole number or a decimal number. In some implementations, the value can represent a percentage, probability, likelihood, etc., of the attribute being present. In some implementations, the value can represent an actual value for the attribute. The layers can be fully connected or partially connected. In a fully connected model, each node in a layer sends its output to each node in the next layer. In a partially connected network, each node in a layer sends its output to some of the nodes in the next layer.

The function performed by the nodes on the input values maps the input to the output. The function uses parameters to perform the mapping. The mapping may be a surjective mapping. The model requires training to determine the parameters, which may start as random values. The parameters are also referred to as weights. For the purposes of this application, the weights may be expressed as ψ. The training process determines the optimal parameters using an objective function. The objective function identifies the goals of the mapping and helps the model modify the parameters through iterative training rounds until arriving at an optimal set of parameters. Once the optimal parameters are identified, the model is said to be trained and can be used in an inference mode. In an inference mode, the model uses the parameters to provide or predict an output from a given input. Each machine learned model is trained for a specific task, e.g., prediction, classification, encoding, etc. The task performed by the computational model is determined by the inputs provided, the mapping function, and the desired output.

In the example of <FIG>, the cloned encoders <NUM> includes a plurality of encoders. Each encoder has its own layers and receives a separate input, but each encoder shares the same set of weights ψ with the other encoders. Thus, during training, the weights ψ are adjusted the same for all encoders in the cloned encoders <NUM>. Because the encoders share weights, they may be referred to as clones. In other words, each encoder of cloned encoders effectively represent the same encoder and would, if given the same input produce the same features. However, during training each encoder of the cloned encoders <NUM> is given a different input, but each input is considered equivalent in substance to a system designer. The encoders of the cloned encoders <NUM> use the same objective function.

The inputs provided to the cloned encoders <NUM> represent equivalent signals. In some implementations, equivalent signals are signals deemed equivalent by a system designer. Thus, a system designer may select the types of modifications made to a clean input. In this sense, a system designer may oversee the generation of a set of equivalent input for processing. A clean input is any input that is modified to generate a set of equivalent signals. In general, the clean input represents an original file, desired output, etc. The set of equivalent signals <NUM> represents different modifications made to a clean signal. As used herein, modifications can include any modification to or information added to a clean signal that a designer deems acceptable. A modified signal can include noise or artifacts added to a clean signal. A modified signal can include distortions of a clean signal, such as all-pass filtering (i.e., relative delay of different frequency bands, phase shifts, etc.). A modified signal can include information outside a target manifold added to the clean signal. Put another way, a modification may be any modification made to an input not regarded as salient to a human designer. For example, if clean input <NUM>(<NUM>) is a speech sample, modified input <NUM>(<NUM>) may be the same speech sample with traffic noise added, modified input <NUM>(<NUM>) may be a relative delay of a frequency band of the speech signal, modified input <NUM>(<NUM>) may be the same speech sample with restaurant noise added, modified input <NUM>(<NUM>) may be the same speech sample with reverberation added, input <NUM>(<NUM>) may be the same speech sample with microphone distortions added, etc..

In some implementations, the system designer decides what kinds of modifications to make and then uses a modification engine <NUM> to apply the modifications to the clean signal and generate the set of equivalent signals <NUM>. The modification engine <NUM> may use training data <NUM> as the source of clean data. The modification engine <NUM> may use the training data <NUM> as a source of one or more of the modifications, e.g., to generate one or more of the modified inputs. For example, the training data can be a dataset of clean signals, e.g., a dataset of images, a dataset of speech data uttered by native speakers, professionally recorded music compositions, etc. In some implementations, the modification engine <NUM> may be configured to provide inputs within a signal to noise ratio (SNR) range. For example, if signals provided to the cloned encoders <NUM> are too noisy/modified too much, an objective function that minimizes reconstruction error may encourage removal of attributes that are salient (perceived by humans and relevant to comprehension and/or quality). On the other hand, if the noise is insufficient, the salient features may be more sensitive to noise. The number N of signals in the set of equivalent signals <NUM> is implementation dependent. Generally, the number depends on the type of modifications applicable to the input as well as the processing capabilities of the hardware of system <NUM>, and training time. In some implementations the selection of N is a tradeoff between performance and quality. In some implementations the set of equivalent signals <NUM> may include <NUM> inputs (e.g., N=<NUM>).

Each of the encoders <NUM>(<NUM>)-<NUM>(N) in the cloned encoders <NUM> receives a different one of the set of equivalent signals <NUM>. The cloned encoders <NUM> includes one encoder for each different equivalent signal. For example, encoder <NUM>(<NUM>) may receive a clean input <NUM>(<NUM>), while encoder <NUM>(<NUM>) receives a first modified input <NUM>(<NUM>), etc..

The encoders in the cloned encoders <NUM> each use an objective function to learn the parameters (weights) that enable the encoders to extract a feature vector that is similar across all encoders despite the different inputs and includes information needed to reconstruct a representation of the clean input. The resulting extracted feature vectors, the salient features, generally lack information about the modifications and are therefore robust to the modifications. In an unsupervised learning implementation the objective function includes two terms. The first term encourages similarity across the salient features and the second term encourages independence and unit-variance. The second term encourages sparsity. In some implementations, the objective function may include a third term that encourages the cloned encoders <NUM> to find salient features that map to a shared target signal via a decoder, e.g., minimizing decoder loss. The decoder is one of a set of cloned decoders. In some implementations, the second and optional third terms may be weighted. In some implementations, the objective function may be expressed as Dglobal = DE + λMMDDMMD + λDDD where Dglobal is the global loss, which training attempts to minimize, DE is the first term that maximizes similarity between the salient features output by the cloned encoders, DMMD is the second term that maximizes independence, unit-variance, and optionally sparsity, λMMD is a weighting factor applied to the second term, DD is the third term that encourages reconstruction of a target signal, and λD is a weighting factor applied to the third term. The goal of training is to determine weights that get Dglobal as close to zero as feasible. In some implementations, the system <NUM> defines the global loss as an expectation over the data distribution. In some implementations, the system <NUM> defines the global loss as an average over an observed batch of m data. In other words, the training occurs over a batch of m clean inputs and their respective sets of equivalent signals <NUM>. The system <NUM> may use a stochastic gradient descent over the batch of m data points to optimize over the empirical distribution associated with the training data <NUM>. The training data <NUM> may be any data consistent with the input that will be used in inference mode. In a speech domain, the input data may be a block or frame of speech. For example, the training data may be <NUM> of speech. The training data <NUM> may be a block (e.g., <NUM>, <NUM>) of speech converted to a spectral representation on a mel-frequency scale. In an image domain, the input data may be a block of pixel data. For example, the training data may be a 32x32 block of pixels. The training data <NUM> may be <NUM> of music, etc..

The first term of the objective function maximizes similarity among the salient features produced by the encoders of the cloned encoders <NUM>. Similarity between the salient features extracted by the different encoders of the cloned encoders <NUM> can be maximized by minimizing the L2 norm (or square root of the squares), between the salient features generated by a first clone and the salient features generated by remaining clones (for reduced computational effort) or between the salient features generated by each clone and those generated by the remaining clones. For example, in some implementations the first term may be expressed as <MAT>, where m is the number of inputs (sets of equivalent signals <NUM>), N is the number of encoders in the cloned encoders <NUM>, ∥ · ∥ is the L2 norm, and z represents the salient features extracted by an encoder, and i labels a particular feature. In other words, as an example, encoder <NUM>(<NUM>) may extract features z(<NUM>), encoder <NUM>(<NUM>) may extract features z(<NUM>), etc. for one of the m sets of equivalent signals <NUM>. The salient features z(<NUM>) from the expression above are referred to as the reference features and the encoder that extracts z(<NUM>) from its respective input is the reference encoder. It is understood that any of the encoders in the cloned encoders <NUM> can be selected as the reference encoder. Accordingly, implementations are not limited to comparing the salient features from the clean signal (i.e., salient features <NUM>(<NUM>)) to the other salient features (i.e., salient features <NUM>(<NUM>) to <NUM>(N)). Instead, any of salient features <NUM>(<NUM>) to <NUM>(N) can be selected as the reference features. Some implementations only compare the reference features to the remaining salient features to reduce computational effort.

In some implementations, the system <NUM> may compare all salient features for a set of equivalent signals <NUM> to all other salient features for the set of equivalent signals <NUM>. In such implementations, each encoder becomes a reference encoder. In some implementations, a subset, e.g., two, three, five, of the encoders may be selected as reference encoders. In some implementations the system may minimize a <NUM>-norm rather than an L2 norm. The L2 norm reduces the larger differences, which is a benefit in determining salient features, but implementations can use <NUM>-norm. The first term of the objective function may be computed by an equivalency optimizer <NUM>. The equivalency optimizer <NUM> may be configured to select features describing information components that are shared between the signals in the set of equivalent signals (e.g., input <NUM>(<NUM>) to <NUM>(N)) at relatively high fidelity. In some implementations, the equivalency optimizer <NUM> may calculate the L2 norm as outlined above. In some implementations, the equivalency optimizer <NUM> may calculate the <NUM>-norm. The equivalency optimizer <NUM> may determine which weights ψ to adjust to minimize the differences in the salient features produced for sets of equivalent signals.

The second term of the objective function, which maximizes independence and variance for the salient features, may be determined using any approach that encourages/forces the salient features to have a specified distribution. In other words, the system may force a specified distribution on the features. The specified distribution encourages independence and a given variance. Examples of such approaches include the chi-square test, the earth-moving distance reformulated via the Kantorovich-Rubinstein duality, and maximum mean discrepancy (MMD). MMD measures the distance between two distributions and requires a desired (specified) distribution for comparison. Example distributions include Gaussian, uniform, normal, Laplacian, etc. The choice of distribution determines sparsity. Gaussian and normal distributions do not encourage sparsity. A Laplacian distribution encourages sparsity. In a sparse vector, most of the dimensions have low values (e.g., a value close to zero) and only a few have large values. In some implementations, the system <NUM> may encourage the salient features to be sparse, e.g., with ten, twelve, fifteen, etc. large values among the dimensions. The dimension of a vector is the number of different attributes it represents.

The second term of the objective function, when using the MMD measure, may be expressed as <MAT>, where m is the number of batches (e.g., number of different sets of equivalent signals <NUM>), yi is drawn from the selected distribution, zi is a set of salient features generated by a reference encoder, and k(·,·) is a kernel with the desired scale and shape. In some implementations, the kernel is a multi quadratic kernel. In some implementations, when using the MMD measure, the second term may be expressed as <MAT>. For the MMD to perform correctly, m must be sufficiently large and depends on the desired precision. For example, m may be in the thousands. The reference encoder for the second term of the objective function can be the same encoder as the reference encoder used for the first term of the objective function. The reference encoder for the second term can be different than the reference encoder used for the first term. In some implementations, the system <NUM> may use more than one reference encoder. For example, the system may compare the salient features extracted by two, three, five, etc., different encoders to the distribution. The second term of the objective function may be computed by an independence optimizer <NUM> configured to measure the independence and variance as outlined above and to determine which weights ψ to adjust, for example, to minimize the difference between the distribution and a selected distribution. In some implementations, the second term may be weighted.

The third term of the objective function is an optional term that encourages the mapping of the different salient features to a shared target signal. In other words, the third term relates to the reconstruction of a target signal. The target signal may be derived from a clean signal (e.g., clean input <NUM>(<NUM>)). The target signal may not attempt to approximate the clean signal directly. In some implementations, the target signal may characterize a short-term spectral characteristic of the clean signal. Such an implementation lowers the use of computational resources. In some implementations, the target signal may be one of the equivalent modified inputs. In some implementations, the target signal may be a representation of the clean signal with an appropriate criterion. For example, in a speech domain, a mel-spectrum representation of a clean speech signal with an L2 norm (i.e., squared error) can be used as the target signal. As another example, in a music domain, a mel-spectrum representation of a clean music signal with an L2 norm can be used as the target signal. As another example, in an image domain, a wavelet representation of a selected resolution may be used as the target signal. In some implementations, the target signal may be the clean signal.

The system <NUM> may use cloned decoders <NUM> to reproduce the target signal from the salient features. In some implementations, the cloned decoders <NUM> may have the same number of decoders as encoders in the cloned encoders <NUM>. In such implementations, each decoder in the cloned decoders <NUM> receives a different one of the salient feature vectors, e.g., decoder <NUM>(<NUM>) receives as input salient features <NUM>(<NUM>), decoder <NUM>(<NUM>) receives as input salient features <NUM>(<NUM>), etc. In some implementations (not illustrated in <FIG>), the cloned decoders <NUM> may include a single decoder. In some implementations, the cloned decoders <NUM> may have fewer cloned decoders than cloned encoders. The cloned decoders take as input salient features from an encoder and map the salient features to an output. The output represents a reconstruction of the input provided to the encoders. In some implementations, the configuration of the decoders may mirror the configuration of the encoders. For example, the encoders may each use two layers of Long Short-Term Memory (LSTM) nodes and one fully connected layer of nodes, each layer having <NUM> nodes and each of the cloned decoders may use one fully connected layer followed by two layers of LSTM, each with <NUM> nodes. As another example, the encoders may use four layers of LSTMs and two layers of fully connected nodes with the decoders mirroring this configuration. In some implementations, the decoders may not mirror the configuration of the encoders. The encoders are not limited to these exact configurations and can include feed forward layers, convolutional neural networks, ReLU activation, etc..

The cloned decoders <NUM> can include any suitable decoders. The cloned decoders <NUM> may be optimized by a loss function that employs an L2 norm. In some implementations, the third term may be expressed as <MAT>, where m is the number of sets of equivalent signals, N is the number of cloned encoders, <MAT> is a salient feature vector generated by one of the cloned encoders, <MAT> is a suitable signal representation for the clean input (i.e., the target signal), which has dimensionality P, <MAT> is the cloned decoders network <NUM> with learned parameters ϕ that map a vector with dimensionality Q (e.g., <MAT>) to a vector of dimensionality P (e.g., vi) and where summation is over the m sets of equivalent inputs. In some implementations, the number of cloned encoders (N) used to calculate the third term may be less than the number of cloned encoders <NUM>. In some implementations, the third term may be weighted. In some implementations, the weight of the third term may be higher than the weight of the second term. For example, the weight of the third term may be <NUM> when the weight of the second term is <NUM>. In some implementations, the weight of the third term may be zero. In such implementations, the training is unsupervised. The third term of the objective function may be computed by a decoder loss optimizer <NUM> configured to measure the similarity of the decoded output against a target signal as outlined above and to determine which weights ψ to adjust, for example, to minimize the difference between the reconstructed signals and the target signal. The parameters ϕ are also learned during training.

The system <NUM> may include or be in communication with other computing devices (not shown). For example, the other devices may provide the training data <NUM>, the modification engine <NUM>, and/or the sets of equivalent signals <NUM>. In addition, the system <NUM> may be implemented in a plurality of computing devices in communication with each other. Thus, salient feature extraction system <NUM> represents one example configuration and other configurations are possible. In addition, components of system <NUM> may be combined or distributed in a manner differently than illustrated.

<FIG> illustrates an example system <NUM> used for inference, in accordance with the disclosed subject matter. System <NUM> is one example of how salient features may be used. In the example of system <NUM>, the salient features are used to condition a generative network <NUM>. The system <NUM> is a computing device or devices that that take the form of a number of different devices, for example a standard server, a group of such servers, a rack server system, two computers in communication with each other, etc. In addition, system <NUM> may be implemented in a personal computer, for example a desktop or laptop computer. The system <NUM> may be an example of computer device <NUM>, as depicted in <FIG> or computer device <NUM>, as depicted in <FIG>.

Although not shown in <FIG>, the system <NUM> can include one or more processors formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processors can be semiconductor-based - that is, the processors can include semiconductor material that can perform digital logic. The processors can be specialty processors, such as graphics processing units (GPUs). The system <NUM> can also include an operating system and one or more computer memories, for example a main memory, configured to store one or more pieces of data, either temporarily, permanently, semi-permanently, or a combination thereof. The memory may include any type of storage device that stores information in a format that can be read and/or executed by the one or more processors. The memory may include volatile memory, non-volatile memory, or a combination thereof, and store modules that, when executed by the one or more processors, perform certain operations. In some implementations, the modules may be stored in an external storage device and loaded into the memory of system <NUM>.

The system <NUM> includes an encoder <NUM>. The encoder <NUM> represents one of the encoders of the cloned encoders <NUM> of <FIG>. In other words, the encoder <NUM> is a trained encoder, with weights ψ that have been optimized to produce salient features <NUM> from a given input <NUM>. As the encoders of the cloned encoders <NUM> share the same weights ψ, they are identical encoders and any encoder (<NUM>(<NUM>) to <NUM>(N)) can be used as encoder <NUM> in inference mode. Put another way, as an encoder uses the weights to map an input to an output, and only one set of weights is used in the cloned encoders <NUM>, the weights ψ represent the encoder. Thus, weights ψ, determined by system <NUM>, enable the encoder <NUM> to map the input <NUM> to salient features <NUM>. The input <NUM> is a signal of the same format as the signals used to train the cloned encoders <NUM> of <FIG>. For example, the input <NUM> may be a mel-frequency block of <NUM> of speech. The encoder <NUM> maps the input <NUM> to salient features <NUM>. The system <NUM> provides the salient features <NUM> to the generative network <NUM>. In some implementations, the system <NUM> may compress and/or store the salient features <NUM> and transmit the features <NUM> to the generative network <NUM>. The generative network <NUM> then uses the salient features <NUM> and the input <NUM> for conditioning. Conditioning is a method of providing features to the network <NUM> to produce specific characteristics. The salient features <NUM> provide better features for conditioning. In the example of system <NUM>, the generative network <NUM> is conditioned to focus on the salient features of the input <NUM>, which makes the network <NUM> robust to modifications, such as noise and distortions. Although not shown in <FIG>, the system <NUM> in inference mode processes many different inputs as input <NUM>. For example, system <NUM> may break a longer audio recording into frames, e.g., <NUM> or <NUM>, and process each frame in the recording as a separate input <NUM>, providing respective salient features <NUM> for each input <NUM> for conditioning.

The system <NUM> is one example use of the salient features <NUM>, but salient features may be used in other ways. For example, the salient features <NUM> may be used to store or transmit data in a compressed format. When salient features <NUM> are sparse, they may be compressed to a much smaller size and transmitted with less bandwidth than the original signal. A decoder, such as a decoder used in training the encoder, can regenerate the compressed salient features. Moreover, while <FIG> and <FIG> have been discussed in general with respect to speech, implementations are not so limited. For example, implementations can be adapted for use with input from image, music, video, etc., files.

<FIG> is a flowchart of an example process for identifying and using salient features, in accordance with disclosed subject matter. Process <NUM> may be performed by a salient feature system, such as system <NUM> of <FIG> and system <NUM> of <FIG>. Process <NUM> begins by obtaining a set of inputs for a batch of clean inputs (<NUM>). The clean inputs may be from a database, e.g., a database of hundreds of hours of speech, an image library, etc. The clean inputs need not be clean in the conventional sense, just a signal for which equivalent signals are generated. In this sense a clean input represents one data point in a batch of data points. For each clean input, the system also obtains a number of equivalent inputs. A system designer may select the types of modifications made to the clean input to obtain an equivalent input. The system thus obtains a set of equivalent inputs. In some implementations, the set may include the clean input and the number of equivalent inputs. In some implementations, the set may not include the clean input, but is still referred to as associated with the clean input. The number of equivalent inputs is implementation dependent and is a trade off between training time, computational resources, and accuracy. Generally, less than <NUM> equivalent inputs are used. In some implementations, less than <NUM> equivalent inputs may be used. In some implementations, less than <NUM> equivalent inputs may be used. Each of the inputs in the set of equivalent inputs are based on the clean input. For example, different artifacts may be added to the clean signal. Different distortions may be made to the clean input. Different noise may be added to the clean input. In general, modified inputs are any modifications made to the clean input, but still deemed equivalent with the clean input, e.g., in terms of content and understanding. The cloned encoders learn to ignore this extra information. The system may train a set of cloned encoders, i.e., a plurality of encoders sharing weights, to extract salient features from a set of equivalent inputs (<NUM>). This process is described in more detail with regard to <FIG>. Once the cloned encoders are trained, i.e., a set of optimized weights are determined, the weights represent a trained encoder, also referred to as a salient feature encoder. The system uses the salient feature encoder (i.e., using the optimized weights) to extract salient features for an input (<NUM>). The input is of a type similar to the training inputs used in step <NUM>. The input may also be referred to as a conditioning input signal. In some implementations, the input signal may be parsed into several inputs, e.g., a plurality of time sequences of an audio file, a plurality of pixel blocks of a video file, etc. Each of the parsed components, e.g., each time sequence, may be used by the system as a separate input. The system may use the salient features and the conditioning input to condition a generative network (<NUM>) or for compression (<NUM>). In some implementations, the system may compress and store the salient features (<NUM>) before transmitting the features to the generative network for conditioning (<NUM>). Although shown in <FIG> as used for one input in step <NUM>, the system may repeat step <NUM> and either of steps <NUM> or <NUM> any number of times. For example, the system may perform step <NUM> repeatedly for frames in a video or audio file or for blocks of pixels in an image file. Thus, it is understood that process <NUM> includes repeating step <NUM> and either of steps <NUM> or <NUM> with different input, as needed. Thus, this portion of process <NUM> can be used over a time sequence, over a large image, etc..

<FIG> is a flowchart of an example process <NUM> for training an encoder to identify salient features, in accordance with disclosed subject matter. Process <NUM> may be performed by a salient feature system, such as system <NUM> of <FIG>. Process <NUM> may begin with one of the clean inputs (<NUM>). The clean input represents a training data point. The training data may be a frame of an audio or video file, may be a specified time (e.g., <NUM>, <NUM>, <NUM>) of an audio or video file, may be a pixel block of a specified size from an image, etc. The clean input is associated with a set of equivalent inputs. The set of equivalent inputs includes the clean input and one or more modified inputs, as discussed with regard to step <NUM> of <FIG>. The system provides each encoder of a set (plurality) of cloned encoders with a respective input from the set of equivalent inputs (<NUM>). Thus, each encoder in the set receives a different input from the set of equivalent inputs. The cloned encoders share weights. Each encoder provides an output based on the shared weights (<NUM>). The output of an encoder represents salient features for the respective input. The system may repeat the process of generating salient features for different clean input (<NUM>, yes) until a batch of clean input has been processed (<NUM>, no). The batch may have a size (e.g., m) sufficient to make a distribution measurement perform correctly. The system may then adjust the shared weights to minimize a global loss function that maximizes equivalence, independence, variance and optionally sparsity and/or signal reconstruction (<NUM>). The global loss function has a first term that maximizes similarity of the salient features extracted by each of the encoders for a set of equivalent inputs. Thus, each encoder is encouraged to extract the same features as the other encoders for a given set of equivalent inputs. The global loss function has a second term that maximizes independence and variance. The second term forces the salient features to have a particular distribution. The distribution can be a sparse distribution, e.g., encouraging sparseness in the salient features. The second term favors features that are disentangled. Some implementations may include a third term in the global loss function that ensures the salient features can be mapped to a target input. The target input may be derived from the clean input for the set of equivalent inputs. The target input may be the clean input. The target input may be any of the inputs in the set of equivalent inputs. The objective function is described in more detail with regard to <FIG>. The system repeats process <NUM>, with the newly adjusted weights, until convergence, e.g., the weights result in a mapping that minimizes the objective function to an acceptable degree, or until a predetermined number of training iterations has been performed. When process <NUM> completes, the optimal weights for the encoder has been determined and the weights can be used in inference mode.

<FIG> demonstrate benefits provided by disclosed implementations. <FIG> is a graph illustrating the listening test results for various implementations compared with conventional systems. In the example of <FIG>, the training database includes <NUM> hours of speech and <NUM> speakers and a mixed corpora of noise. The mixed corpora of noise includes stationary and non-stationary noise from approximately <NUM>,<NUM> recordings captured in a variety of environments, including busy streets, cafes, and pools. The input to the encoders of the cloned encoders is a set of equivalent inputs that included <NUM> different versions of a signal that contains an utterance. The set of equivalent inputs include the clean utterance and versions with noise additions from <NUM> to <NUM> dB signal to noise ratio (SNR).

In the example of <FIG>, the signals are preprocessed into an oversampled log mel spectrogram representation. In some implementations a single window (sw) approach is used. The single window (sw) approach uses <NUM> with a time shift of <NUM> and a resolution of the representation of <NUM> coefficients for each time shift. In some implementations, a dual window (dw) approach is used. In the dual window (dw) approach, each <NUM> shift is associated with one window of <NUM> and two windows of <NUM>, located at <NUM>-<NUM> and <NUM>-<NUM> of the <NUM> window. The <NUM> windows are described with <NUM> log mel spectrogram coefficients, for a total of <NUM> coefficients (dimensions) for each <NUM> shift. Implementations labeled SalientS use a decoder (supervised) during training. Implementations labeled SalientU use unsupervised training. When a decoder is used (supervised training), the mel spectrum of the clean signal is used as the target signal. Each implementation (e.g., supervised/unsupervised, sw/dw) are used to condition a different WaveNet. The conditioned WaveNet is provided with clean (-clean) and noisy (-noisy) inputs and the output is evaluated using a MUSHRA-like listening test.

The conventional systems illustrated in <FIG> as a reference use feature sets based on a principal component analysis (PCA) that extracts <NUM> features from the <NUM>-dimensional vector of the dual window (dw) data. The PCA is computed for the signals that were used as input to the cloned encoders during training. A PCA that extracts four features is also illustrated.

<FIG> illustrates that the WaveNet conditioned using disclosed implementations are more robust to noise, significantly outperforming the reference system. More specifically, unsupervised learning with a single window (SalientU-sw) provides natural speech quality with good speaker identity but fairly frequent errors for phonemes of short duration. The number of errors is lower for the clean (SalientU-sw-clean) than for noisy (SalientU-sw-noisy) input signals. Supervised learning reduces the errors for noisy inputs for the clean input. For the noisy input the errors are further reduce by using the dual window, reaching almost the quality obtained with a clean input.

<FIG> are graphs illustrating a comparison of disclosed implementations with other reference systems. <FIG> illustrates a comparison between various implementations and SEAGAN. <FIG> illustrates a comparison between various implementations and Denoising WaveNet. In the examples of <FIG> and5C, two sizes of models are used. A first implementation (SalientS and SalientP) has two layers of LSTM cells and one fully connected layer, each with <NUM> nodes. A second implementation (SalientL) has <NUM> layers of LSTMs and <NUM> layers of fully connected nodes, also with <NUM> nodes per layer. All implementations illustrated in <FIG> use supervised training, with the decoders mirroring the encoder configuration. In the example implementations of <FIG>, training processes a sequence from <NUM> input of six <NUM> dual-window mel-frequency frames that overlap by <NUM> percent. Each of the dual window frames include <NUM> mel frequency bins from one window of <NUM> and two windows of <NUM> (located at <NUM>-<NUM> and <NUM>-<NUM> of the <NUM> window) for a total of <NUM> mel frequency bins per frame. The cloned encoders output, as the salient features, <NUM> values per frame with linear activation (e.g., <NUM> salient features per frame). The salient features are inferred on full utterances from the training set to create the conditioning training data for WaveNet, using the clean speech for teacher forcing and negative log-likelihood loss. The SalientL and SalientS examples are trained on the VoiceBank-DEMAND speech enhancement dataset (provided by Valentini et al. In addition, a SalientP example is pre-trained on the WSJ0 dataset and switched to the VoiceBank-DEMAND mid-training.

<FIG> illustrates that in a listening test (MUSHRA-like) implementations match or exceed the performance for SEGAN. <FIG> illustrates that implementations outperform the denoised WaveNet at all SNR ranges. SEGAN and denoised WaveNet are examples of conventional systems that exploit a generative network but are at least in part optimized to reconstruct the ground-truth waveform, which restricts the generative aspect of the generative network. Thus, <FIG> demonstrated that implementations, which do not try to reconstruct the ground truth, outperform such methods.

<FIG> shows an example of a generic computer device <NUM>, which may be system <NUM> of <FIG> or system <NUM> of <FIG>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various example forms of computing devices, such as laptops, desktops, workstations, personal digital assistants, cellular telephones, smart phones, tablets, servers, and other computing devices, including wearable devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

Computing device <NUM> includes a processor <NUM>, memory <NUM>, a storage device <NUM>, and expansion ports <NUM> connected via an interface <NUM>. In some implementations, computing device <NUM> may include transceiver <NUM>, communication interface <NUM>, and a GPS (Global Positioning System) receiver module <NUM>, among other components, connected via interface <NUM>. Each of the components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> may be mounted on a common motherboard or in other manners as appropriate.

The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a GUI on an external input/output device, such as display <NUM>. Display <NUM> may be a monitor or a flat touchscreen display. In some implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory.

In some implementations, the memory <NUM> may include expansion memory provided through an expansion interface.

In one implementation, the storage device <NUM> may be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in such a computer-readable medium. The computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The computer- or machine-readable medium is a storage device such as the memory <NUM>, the storage device <NUM>, or memory on processor <NUM>.

The interface <NUM> may be a high speed controller that manages bandwidth-intensive operations for the computing device <NUM> or a low speed controller that manages lower bandwidth-intensive operations, or a combination of such controllers. An external interface <NUM> may be provided so as to enable near area communication of device <NUM> with other devices. In some implementations, controller <NUM> may be coupled to storage device <NUM> and expansion port <NUM>. The expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

It may also be implemented as part of a rack server system. In addition, it may be implemented in a personal computer such as a laptop computer <NUM>, or smart phone <NUM>. An entire system may be made up of multiple computing devices <NUM> communicating with each other. Other configurations are possible.

<FIG> shows an example of a generic computer device <NUM>, which may be system <NUM> of <FIG> or system <NUM> of <FIG>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various example forms of large-scale data processing devices, such as servers, blade servers, datacenters, mainframes, and other large-scale computing devices. Computing device <NUM> may be a distributed system having multiple processors, possibly including network attached storage nodes, that are interconnected by one or more communication networks. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

Distributed computing system <NUM> may include any number of computing devices <NUM>. Computing devices <NUM> may include a server or rack servers, mainframes, etc. communicating over a local or wide-area network, dedicated optical links, modems, bridges, routers, switches, wired or wireless networks, etc..

In some implementations, each computing device may include multiple racks. For example, computing device 780a includes multiple racks 758a - 758n. Each rack may include one or more processors, such as processors 752a-752n and 762a-762n. The processors may include data processors, network attached storage devices, and other computer controlled devices. In some implementations, one processor may operate as a master processor and control the scheduling and data distribution tasks. Processors may be interconnected through one or more rack switches <NUM>, and one or more racks may be connected through switch <NUM>. Switch <NUM> may handle communications between multiple connected computing devices <NUM>.

Each rack may include memory, such as memory <NUM> and memory <NUM>, and storage, such as <NUM> and <NUM>. Storage <NUM> and <NUM> may provide mass storage and may include volatile or non-volatile storage, such as network-attached disks, floppy disks, hard disks, optical disks, tapes, flash memory or other similar solid state memory devices, or an array of devices, including devices in a storage area network or other configurations. Storage <NUM> or <NUM> may be shared between multiple processors, multiple racks, or multiple computing devices and may include a computer-readable medium storing instructions executable by one or more of the processors. Memory <NUM> and <NUM> may include, e.g., volatile memory unit or units, a non-volatile memory unit or units, and/or other forms of computer-readable media, such as a magnetic or optical disks, flash memory, cache, Random Access Memory (RAM), Read Only Memory (ROM), and combinations thereof. Memory, such as memory <NUM> may also be shared between processors 752a-752n. Data structures, such as an index, may be stored, for example, across storage <NUM> and memory <NUM>. Computing device <NUM> may include other components not shown, such as controllers, buses, input/output devices, communications modules, etc..

An entire system, such as system <NUM>, may be made up of multiple computing devices <NUM> communicating with each other. For example, device 780a may communicate with devices 780b, 780c, and 780d, and these may collectively be known as system <NUM>. As another example, system <NUM> of <FIG> may include one or more computing devices <NUM>. Some of the computing devices may be located geographically close to each other, and others may be located geographically distant. The layout of system <NUM> is an example only and the system may take on other layouts or configurations.

Various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

As used herein, the terms "machine-readable medium" "computer-readable medium" refers to any non-transitory computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory (including Read Access Memory), Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor.

Claim 1:
A computer-implemented method comprising:
receiving an input signal;
extracting salient features (<NUM>) for the input signal by providing the input signal to an encoder (<NUM>) which has been trained to extract salient features (<NUM>) by:
(a) obtaining (<NUM>) a set of inputs (<NUM>) for each first input (<NUM>(<NUM>)) in a batch of inputs, the set of inputs (<NUM>) comprising at least two inputs and including at least one modified input, each modified input being a different modified version of the first input (<NUM>(<NUM>));
for each set of inputs (<NUM>) in the batch of inputs:
(b) providing (<NUM>) the set of inputs (<NUM>) to two or more cloned encoders (<NUM>), each cloned encoder (<NUM>) sharing the weights, and each of the two or more cloned encoders (<NUM>) receiving a different respective input of the set of inputs (<NUM>), and
(c) modifying (<NUM>) the shared weights to minimize a global loss function, the global loss function having a first term that maximizes similarity between features generated by the cloned encoders (<NUM>) for the set of inputs (<NUM>) and a second term that maximizes independence, unit-variance, and sparsity within the features generated by the encoder (<NUM>), the encoder (<NUM>) being one of the two or more cloned encoders (<NUM>);
the salient features (<NUM>) being independent and having a sparse distribution; and either
(i) conditioning a generative network (<NUM>) using the salient features (<NUM>),
wherein the inputs are one of speech inputs, image inputs, video inputs, or music inputs,
and the conditioned generative network (<NUM>) is configured to generate a corresponding speech output, image output, video output, or music output, or
(ii) using (<NUM>) an encoder (<NUM>) of the trained cloned encoders (<NUM>) to extract salient features (<NUM>) for a new input;
compressing (<NUM>) the salient features (<NUM>) extracted for the new input; and
storing the compressed salient features (<NUM>) extracted for the new input as a compressed input,
wherein the inputs are one of speech inputs, image inputs, video inputs, or music inputs,
the method further comprising:
using a decoder to regenerate the salient features from the compressed salient features.