Patent Publication Number: US-10763893-B2

Title: Method for data compression

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     The present application relates to and claims priority of U.S. provisional patent application (“Provisional Application”), Ser. No. 62/364,773, entitled “Method for Data Compression,” filed on Jul. 20, 2016. The Provisional Patent Application is hereby incorporated by reference in its entirety. 
    
    
     BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to data compression using machine learning techniques. In particular, the present invention relates to data compression for improved communication and storage efficiencies, such as desirable in applications involving speech, audio data and video data. 
     2. Discussion of the Related Art 
     In machine learning, an optimizing predictive model is a computational model that learns a function that receives certain input values. One type of optimizing predictive model applies a stochastic gradient descent optimization technique over a loss function. One example of a loss function may be, for example, the difference (“error”) between the output of the predictive model and the desired output. Typically, the optimization procedure involves iteratively executing the model using training data, and then differentiating the model to adapt the values of the model parameters to minimize the loss function. The goal is, over the course of the training, the optimizing predictive model is adapted to perform the function to be learned, which can then be applied to data outside of the training data. 
     An optimizing predictive model may be implemented, for example, in a neural network model. A neural network model is usually based on a graph consisting of nodes (referred to as “neurons”) and a set of directed, weighted edges that connect the neurons. The goal of the training is to achieve a directed graph that represents the function to be learned. In a typical implementation, each neuron is assigned a simple computational task (e.g., a linear transformation followed by a non-linear function, such as a logistic function) and the loss function is computed over the entire neural network model. The parameters of the neural network model are typically determined (“learned”) using a method that minimizes the loss function. Stochastic gradient descent is a method that is often used to achieve the minimization. In stochastic gradient descent, optimization is achieved iteratively by (a) finding analytical gradients for the loss functions and (b) perturbing or moving the test values by a small amount in the direction of the gradient, until the loss function is minimized. 
     In multimedia applications (e.g., an audio or video recording), it is often necessary to compress the data to achieve data communication or storage efficiencies. In addition to eliminating information redundancy, data compression often takes advantage of knowledge of the human senses of perception. For example, in speech encoding, data compression has long been achieved using a linear prediction technique (“LPC”) technique, in which speech signals are analyzed for a set of recognized resonant patterns of the human vocal tract (“formants”), which are then encoded separately from the remainder of the speech signal. The formants may be represented by significantly less bits than the raw speech signal, thereby achieving data compression. Likewise, video encoding often takes advantage of knowledge of the human psycho-visual perception of light intensities and sensitivities to specific colors. However, such approaches result in complex compression and decompression algorithms which are difficult to optimize. 
     SUMMARY 
     According to one embodiment of the present invention, a data compression system includes: (a) a data compression module that receives a sequence of input vectors and that provides a sequence of compressed vectors, the data compression module implementing a computational model characterized by a first set of parameters; (b) a data decompression module that receives the compressed vectors to provide a sequence of output vectors, the data decompression module implementing a computational model characterized by a second set of parameters; and (c) a parameter update module that receives the sequence of input vectors and the sequence of output vectors, and which updates values of the first set of parameters and the second set of parameters based on a loss function of the input vectors, the output vectors, the first set of parameters and the second set of parameters. Each input vector and a corresponding output vector may represent digitized time-domain signals (e.g., speech, audio or video signals) over a predetermined time period. The loss function may be evaluated for each predetermined time period. 
     According to one embodiment of the present invention, the parameter update module updates the first and second sets of parameters based on a stochastic gradient descent method. 
     According to one embodiment of the present invention, the data compression module and the data decompression module are each implemented by a neural network computational model. 
     In one implementation, the data compressor and decompressor pair in a system may be personalized to the voice of a specific individual to allow reproduction (i.e., decompression) at a high fidelity, even with a small decompressor and at a high compression ratio. Such a system is particularly useful in telephone and teleconferencing applications. 
     The present invention is better understood upon consideration of the detailed description below. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram of compressor-decompressor system  100 , which includes data compressor  101  and data decompressor  102  that are trained together, in accordance with one embodiment of the present invention. 
         FIG. 2  is a block diagram of compressor-decompressor system  200 , which includes data compressor  101  and data decompressor  102  that are trained together and that process data in the frequency domain, in accordance with a second embodiment of the present invention. 
     
    
    
     To simplify cross-reference among like elements across the figures, like elements in the figures are assigned like reference numerals. 
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       FIG. 1  is a block diagram compressor-decompressor system  100 , which includes data compressor  101  and data decompressor  102  that are trained together, in accordance with one embodiment of the present invention. To facilitate discussion herein, compressor-decompressor system  100  of  FIG. 1  is described as representing an exemplary system for compression and decompression of audio signals. However, the systems and methods of the present invention are not so limited. The present invention is applicable to any data or signal compression including text, speech, audio or video signal compression and signals used in other multimedia applications. As shown in  FIG. 1 , analog input signals, such as audio signals from one or more source channels, are sampled and filtered in signal preprocessing subsystem  103 . Therefore, signal processing subsystem  103  may provide, for example, a digital data stream of 16-bit samples at 16 KHz, such that 16,000 16-bit samples are available every second for compression. In one embodiment, compression may be performed for data collected over fixed-length time periods (e.g., each second). Compressor  101  takes the input samples for each compression period (e.g., 1 second) and compresses it into, for example, a 8000-bit output compressed vector, to achieve a 32:1 compression. (A rule of thumb has been developed over experience in audio processing that 16-bit samples can be compressed to 1-bit per sample on the average, without significant loss of fidelity to the human listener.) From each 8000-bit value, decompressor  102  provides as output 16,000 16-bit samples. The 16000 16-bit samples may then be digital-to-analog converted and filtered in output module  104  to reproduce the input audio signal over the decompression period. Of course, the sampling frequency, the number of bits per sample and the size of the compressed vector are selected herein merely for illustrative purpose. In actual practice, these quantities may be varied for performance and fidelity as desired. 
     During the time period when compressor-decompressor system  100  is trained, both the input vectors to compressor  101  of each compression period and the corresponding output vectors at decompressor  102  are provided to parameter update unit  105 . A loss function based on these input and output vectors is evaluated multiple times and processed to provide updates to the parameter values so as to adaptively determine the functions implemented in compressor  101  and decompressor  102 . The loss function may be evaluated at regular time intervals, e.g., when an input vector and its corresponding output vector are available. For example, in one embodiment, the input vectors and the output vectors may each be a 16,000-dimensional vector. According to one embodiment of the present invention, the output vector is a function of the input vector and both the model parameters of compressor  101  and decompressor  102 . In one embodiment, to evaluate the loss function, a 2-norm value between the two vectors for each compression period (i.e., the square of the Euclidean distance between the input and output vectors) is calculated. (The loss function may be a function of the 2-norm values of multiple compression periods.) Initially, i.e., at the beginning of the training process, the model parameters of compressor  101  and decompressor  102  may be each initialized to any suitable value (e.g., 0). 
     The training process is expected to move the value of each model parameter in a way that minimizes the loss function. According to one embodiment of the present invention, in which the method of stochastic gradient descent is used, a gradient of the loss function is the calculated to drive the optimization process. (The gradient may be, for example, the derivative of the loss function with respect to each model parameter). In some embodiments, a “straight-through estimator” approach may be used, regardless of any non-linearity that may exist in the underlying data processing structures. Based on the gradient, the model parameters in both compressor  101  and decompressor  102  are updated so that the resulting evaluations of the computational models in compressor  101  and decompressor  102  would result in a reduced value in the loss function. In one embodiment, each model parameter is updated by multiplying the gradient with a small negative value, projecting the product on the model parameter, and adjusting the model parameter accordingly to achieve the reduced loss function). Reinforcement learning techniques may also be applied. In addition, any other suitable technique to improve performance may also be applied. For example, the techniques disclosed by Georges Harik in U.S. patent application Ser. No. 14/165,431, entitled “METHOD FOR AN OPTIMIZING PREDICTIVE MODEL USING GRADIENT DESCENT AND CONJUGATE RESIDUALS,” filed on Jan. 27, 2014, may also be used. 
     In one embodiment, compressor-decompressor system  100  is trained using audiobooks (i.e, audio recordings in which text is read aloud). As compressor-decompressor system  100  may be implemented as relatively small programs, even for a high compression ratio, the decompression module can be stored or transmitted with the compressed data for customized decompression. In fact, the compressor-decompressor systems of the present invention may be used in mobile applications (e.g., smartphones). For example, in one application, trained decompressors can be exchanged at the beginning of a communication session among communicating participants prior to communicating the compressed data. The inventor discovered that the compressor-decompressor system  100  may be easily trained to become personalized to a specific speaker (e.g., trained to become particular efficient for compressing speech of a particular speaker). Such personalized compressor-decompressor systems are particularly desirable for telephone or teleconference applications. 
     The methods of the present invention may be implemented, for example, in a neural network model. In one embodiment of the present invention, compressor  101  and decompressor  102  may each be implemented as a neural network in a computational environment that includes a number of parallel processors. In one implementation, in which audio signals sampled at 16 KHz are used, with data compressed to 8,000 bits per second, compressor  101  and decompressor  102  may each be implemented by a neural network of 2.5 million nodes. Each parallel processor, which may be provided by a graphics processor, may implement multiple nodes to take advantage of computational structures optimized for arithmetic operations, as is typical in such graphics processors. Alternatively, each processor may also be implemented by a custom circuit optimized for implementing neural network model elements. A host computer system using conventional programming techniques may configure compressor-decompressor system  100 . Of course, each neuron in the neural network model may also be implemented by a customized circuit. 
     A compression-decompression system can also process data in the frequency domain.  FIG. 2  is a block diagram of compressor-decompressor system  200 , which includes data compressor  101  and data decompressor  102  that are trained together and that process data in the frequency domain, in accordance with a second embodiment of the present invention. In  FIG. 2 , a fast fourier transform (FFT) is performed on the sampled audio data in FFT module  201  prior to being provided to compressor  101 . Correspondingly, an inverse fast fourier transform (iFFT) is performed on the decompressed data from decompressor  102  in iFFT module  202  to provide time-domain digital output data to output module  104 . In this embodiment, parameter updates may be performed using time domain data, as shown in  FIG. 2 . 
     The above detailed description is provided to illustrate the specific embodiments of the present invention and is not intended to be limiting. Various modification and variations within the scope of the present invention are possible. The present invention is set forth in the following claims.