Patent Publication Number: US-2019197363-A1

Title: Computer-implemented methods and systems for optimal linear classification systems

Description:
CROSS-REFERENCES TO RELATED APPLICATIONS 
     This application claims the benefit of U.S. provisional application No. 62/556,185 filed Sep. 8, 2017. 
    
    
     FIELD OF THE INVENTION 
     This invention relates generally to learning machines. More particularly, it relates to methods and systems for statistical pattern recognition and statistical classification. This invention is described in an article by applicant, “Design of Data-Driven Mathematical Laws for Optimal Statistical Classification Systems,” arXiv:1612.03902v8: submitted on 22 Sep. 2017. 
     BACKGROUND OF THE INVENTION 
     Statistical pattern recognition and classification methods and systems enable computers to describe, recognize, classify, and group patterns, e.g., digital signals and digital images, such as fingerprint images, human faces, spectral signatures, speech signals, seismic and acoustic waveforms, radar imaaes, multispectral images, and hyperspectral images. Given a pattern, its automatic or computer-implemented recognition or classification may consist of one of the following two tasks; (a) supervised classification (e.g., discriminant analysis) in which the input pattern is identified as a member of a predefined class, (b) unsupervised classification (e.g., clustering) in which the pattern is assigned to a hitherto unknown class. 
     Automatic or computer-implemented recognition, description, classification and grouping of patterns are important problems that have important applications in a variety of enaineering and scientific fields such as biology, psychology, medicine, computer vision, artificial intelligence, and remote sensing. Computer-implemented classification methods and systems enable the best possible utilization of available sensors, processors, and domain knowledge to make decisions automatically: based on automated processes such as optical character recognition, geometric object recognition, speech recognition, spoken language identification, handwriting recognition, waveform recognition, face recognition, system identification, spectrum identification, fingerprint identification, and DNA sequencing. 
     The design of statistical pattern recognition systems involves two fundamental problems. The first problem involves identifying measurements or numerical features of the objects being classified and using these measurements to form pattern or feature vectors for each pattern class. For M classes of patterns, a pattern or feature space is composed of M regions, where each region contains the pattern vectors of a class. The second problem involves generating decision boundaries that divide a pattern or feature space into M regions. 
     A suitable criterion is necessary to determine the best possible partitioning for a given feature space. Bayes&#39; criterion divides a feature space in a manner that minimizes the probability of classification error so that the average risk of the total probability of making a decision error is minimized. Bayes&#39; classifiers are difficult to design because the class-conditional density functions are usually not known. Instead, a collection of training data is used to estimate either decision boundaries or class-conditional density functions. 
     Machine learning algorithms enable computers to learn either decision boundaries or class-conditional density functions from training data. The estimation error between a learning machine and its target function depends on the training data in a twofold manner: large numbers of parameter estimates raise the variance, whereas incorrect statistical models increase the bias. For this reason, model-free architectures based on insufficient data samples are unreliable and have slow convergence speeds. However, model-based architectures based on incorrect statistical models are also unreliable. Model-based architectures based on accurate statistical models are reliable and have reasonable convergence speeds, but proper statistical models for model-based architectures are difficult to identify. The design of accurate statistical models for learning machines involves the difficult problem of identifying correct forms of equations for statistical models of learning machine architectures. 
     The design and development of learning machine architectures has primarily been based on curve and surface fitting methods of interpolation or regression, alongside statistical methods of reducing data to minimum numbers of relevant parameters. The generalization performance of any given learning machine depends on a variety of factors, including the quality and quantity of the training data, the complexity of the underlying problem, the learning machine architecture, and the learning algorithm used to train the network. 
     Machine learning algorithms introduce four sources of error into a classification system: (1) Bayes&#39; error (also known as Bayes&#39; risk), (2) model error or bias, (3) estimation error or variance, and (4) computational errors, e.g., errors in software code. Bayes&#39; error is a result of overlap among statistical distributions and is an inherent source of error in a classification system. As a result, the generalization error of any learning machine whose target function is a classification system includes Bayes&#39; error, modeling error, estimation error, and computational error. The probability of error is the key parameter of all statistical pattern recognition and classification systems. The amount of overlap between data distributions determines the Bayes&#39; error rate which is the lowest error rate and highest accuracy that can be achieved by any statistical classifier. In general, Bayes&#39; error rate is difficult to evaluate. 
     The generalization error of any learning machine whose target function is a classification system determines the error rate and the accuracy of the classification system. What would be desirable therefore is computer-implemented classification methods and systems for which the generalization error of any given classification system is Bayes&#39; error for M classes of pattern or feature vectors. Further, it would be advantageous to have computer-implemented methods and systems that enable the fusing of classification systems for different data sources. It would also be advantageous to have computer-implemented methods and systems that provide a practical statistical gauge for measuring data distribution overlap and Bayes&#39; error rate for given sets of feature or pattern vectors. 
     SUMMARY OF THE INVENTION 
     The present invention addresses the above needs by providing computer-implemented methods and systems for statistical pattern recognition and classification applications for which the generalization error of any given linear classification system is Bayes&#39; error for M classes of pattern or feature vectors that are drawn from statistical distributions that have similar covariance functions and further computer-implemented methods and systems for fusing feature vectors from different data sources and measuring data distribution overlap and Bayes&#39; error rate for given sets of feature or pattern vectors. 
     One aspect provides linear classification systems that have the highest accuracy and achieve Bayes&#39; error rate for two sets of feature vectors drawn from statistical distributions that have similar covariance functions. Another aspect provides multiclass linear classification systems that have the highest accuracy and achieve Bayes&#39; error rate for feature vectors drawn from similar or different data sources. Additional aspects will become apparent in view of the following descriptions. 
     In accordance with an aspect of the invention, a method for computer-implemented, linear classification involves transforming two sets of pattern or feature vectors into a data-driven, likelihood ratio test that is based on a dual locus of likelihoods and principal eigenaxis components formed by a locus of weighted extreme points, all of which determine a point of statistical equilibrium where the opposing forces and influences of a linear classification system are balanced with each other, and the eigenenergy and the Bayes&#39; risk of the classification system are minimized, where each weight specifies a class membership statistic and a conditional density for an extreme point, and each weight determines the magnitude and the total allowed eigenenergy of an extreme vector. A dual locus of likelihoods and principal eigenaxis components contains Bayes&#39; likelihood ratio and delineates the coordinate system of a linear decision boundary. Thereby, a dual locus of likelihoods and principal eigenaxis components is the basis of an optimal linear classification system that implements Bayes&#39; likelihood ratio test: the gold standard of linear classification tasks. A dual locus of likelihoods and principal eigenaxis components is generated by a system of fundamental, data-driven, vector-based locus equations of binary classification for a linear classification system in statistical equilibrium, where the opposing forces and influences of a system are balanced with each other, and the eigenenergy and the corresponding Bayes&#39; risk of a linear classification system are minimized. The method generates the best linear decision boundary for two given sets of feature vectors drawn from statistical distributions that have similar covariance functions and constant or unchanging statistics. 
     In accordance with yet another aspect of the invention, a method for computer-implemented, multiclass linear classification involves transforming multiple sets of pattern or feature vectors into linear combinations of data-driven, likelihood ratio tests, each of which is based on a dual locus of likelihoods and principal eigenaxis components formed by a locus of weighted extreme points that contains Bayes&#39; likelihood ratio and generates the best linear decision boundary for feature vectors drawn from statistical distributions that have similar covariance functions. Thereby, linear combinations of data-driven, likelihood ratio tests provide M-class linear classification systems for which the eigenenergy and the Bayes&#39; risk of each classification system are minimized, and each classification system is in statistical equilibrium. Moreover, any given M-class linear classification system exhibits the highest accuracy and achieves Bayes&#39; error rate for feature vectors drawn from statistical distributions that have similar covariance functions or feature vectors that have large numbers of components. 
     Further, feature vectors that have been extracted from different data sources can be fused with each other by transforming multiple sets of feature vectors from different data sources into linear combinations of data-driven, likelihood ratio tests that achieve Bayes&#39; error rate and generate the best linear decision boundary. 
     In accordance with yet another aspect of the invention, a method for measuring data distribution overlap and Bayes&#39; error rate for two given sets of feature vectors drawn from statistical distributions that have similar covariance functions and constant or unchanging statistics involves transforming the feature vectors into a data-driven, likelihood ratio test that is the basis of an optimal linear classification system for which the eigenenergy and the Bayes&#39; risk of the classification system are minimized, and the classification system is in statistical equilibrium. The data-driven likelihood ratio test provides a practical statistical gauge for measuring data distribution overlap and Bayes&#39; error rate for the two given sets of feature or pattern vectors. The data-driven, likelihood ratio test can also be used to identify homogeneous data distributions. 
     Additional aspects, applications, and advantages will become apparent in view of the following description and associated figures. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a diagram illustrating how overlapping data distributions determine decision regions according to the invention. 
         FIG. 2  is a diagram illustrating how non-overlapping data distributions determine decision regions according to the invention. 
         FIG. 3  is a diagram illustrating that the decision space of a binary classification involves risks and counter risks in each of the decision regions for overlapping data distributions according to the invention. 
         FIG. 4  is a diagram illustrating congruent decision regions that are symmetrically partitioned by a linear decision boundary according to the invention. 
         FIG. 5  is a diagram illustrating congruent decision regions that are symmetrically partitioned by a linear decision boundary according to the invention. 
         FIG. 6  is a flowchart of one embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The present invention involves new criteria that have been devised for the binary classification problem and new geometric locus methods that have been devised and formulated within a statistical framework. Before describing the innovative concept, a new theorem for binary classification is presented along with new geometric locus methods. Geometric locus methods involve equations of curves or surfaces, where the coordinates of any given point on a curve or surface satisfy an equation, and all of the points on any given curve or surface possess a uniform characteristic or property. Geometric locus methods have important and advantageous features: locus methods enable the design of locus equations that determines curves or surfaces for which the coordinates of all of the points on a curve or surface satisfy a locus equation, and all of the points on a curve or surface possess a uniform property. 
     The new theorem for binary classification establishes the existence of a system of fundamental, vector-based locus equations of binary classification for a classification system in statistical equilibrium that must be satisfied by Bayes&#39; likelihood ratio and decision boundary. Further, the new theorem provides the result that classification systems seek a point of statistical equilibrium where the opposing forces and influences of a classification system are balanced with each other, and the eigenenergy and the Bayes&#39; risk of a classification system are minimized. The theorem and new geometric locus methods enable the design of a system of fundamental, data-driven, vector-based locus equations of binary classification for a classification system in statistical equilibrium that are satisfied by Bayes&#39; likelihood ratio and decision boundary. 
     It will be appreciated by those ordinarily skilled in the art that Bayes&#39; decision rule is the gold standard for statistical classification problems. Bayes&#39; decision rules, which are also known as Bayes&#39; likelihood ratio tests, divide two-class feature spaces into decision regions that have minimal conditional probabilities of classification error. Results from the prior art are outlined next. 
     The general form of Bayes&#39; decision rule for a binary classification system is given by the likelihood ratio test: 
     
       
         
           
             
                 
             
              
             
               
                 
                   Λ 
                    
                   
                     ( 
                     x 
                     ) 
                   
                 
                  
                 
                   = 
                   Δ 
                 
                  
                 
                   
                     ? 
                   
                    
                   
                     
                       
                         
                           &gt; 
                         
                       
                       
                         
                           &lt; 
                         
                       
                     
                     
                       ω 
                       2 
                     
                   
                    
                   
                     ? 
                   
                    
                   
                     
                       ( 
                       
                         
                           C 
                           12 
                         
                         - 
                         
                           C 
                           22 
                         
                       
                       ) 
                     
                     
                       ( 
                       
                         
                           C 
                           21 
                         
                         - 
                         
                           C 
                           11 
                         
                       
                       ) 
                     
                   
                 
               
               , 
               
                 
 
               
                
               
                 
                   ? 
                 
                  
                 
                   indicates text missing or illegible when filed 
                 
               
             
           
         
       
     
     where ω 1  or ω 2  is the true data category, p(x|ω 1 ) and p(x|ω 2 ) are class-conditional probability density functions, P(ω 1 ) and P(ω 1 ) are prior probabilities of the pattern classes ω 1  and ω 2 , and C 11 , C 21 , C 22 , and C 12  denote costs for four possible outcomes, where the first subscript indicates the chosen class and the second subscript indicates the true class, 
     Bayes&#39; decision rule computes the likelihood ratio for a feature vector x 
     
       
         
           
             
                 
             
              
             
               
                 Λ 
                  
                 
                   ( 
                   x 
                   ) 
                 
               
                
               
                 = 
                 Δ 
               
                
               
                 ? 
               
             
           
         
       
       
         
           
             
               ? 
             
              
             
               indicates text missing or illegible when filed 
             
           
         
       
     
     and makes a decision by comparing the ratio Λ(x) to the threshold ρ 
     
       
         
           
             η 
             = 
             
               
                 
                   
                     P 
                      
                     
                       ( 
                       
                         ω 
                         2 
                       
                       ) 
                     
                   
                    
                   
                     ( 
                     
                       
                         C 
                         12 
                       
                       - 
                       
                         C 
                         22 
                       
                     
                     ) 
                   
                 
                 
                   
                     P 
                      
                     
                       ( 
                       
                         ω 
                         1 
                       
                       ) 
                     
                   
                    
                   
                     ( 
                     
                       
                         C 
                         21 
                       
                       - 
                       
                         C 
                         11 
                       
                     
                     ) 
                   
                 
               
               . 
             
           
         
       
     
     Costs and prior probabilities are usually based on educated guesses. Therefore, it is common practice to determine a likelihood ratio Λ(x) that is independent of costs and prior probabilities and let η be a variable threshold that accommodates changes in estimates of cost assignments and prior probabilities. Bayes&#39; classifiers are difficult to design because the class-conditional density functions are usually not known. Instead, a collection of training data is used to estimate either decision boundaries or class-conditional density functions. 
     If C 11 =C 22 =0 and C 21 =C 12 =1, then the average risk  (Z) is given by the expression 
     
       
         
           
             
                
               
                 ( 
                 Z 
                 ) 
               
             
             = 
             
               
                 
                   
                     P 
                      
                     
                       ( 
                       
                         ω 
                         2 
                       
                       ) 
                     
                   
                    
                   
                     
                       ∫ 
                       
                         - 
                         ∞ 
                       
                       η 
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             x 
                             | 
                             
                               ω 
                               2 
                             
                           
                           ) 
                         
                       
                        
                       dx 
                     
                   
                 
                 + 
                 
                   
                     P 
                      
                     
                       ( 
                       
                         ω 
                         1 
                       
                       ) 
                     
                   
                    
                   
                     
                       ∫ 
                       η 
                       ∞ 
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             x 
                             | 
                             
                               ω 
                               1 
                             
                           
                           ) 
                         
                       
                        
                       dx 
                     
                   
                 
               
               = 
               
                 
                   
                     P 
                      
                     
                       ( 
                       
                         ω 
                         2 
                       
                       ) 
                     
                   
                    
                   
                     
                       ∫ 
                       
                         z 
                         1 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             x 
                             | 
                             
                               ω 
                               2 
                             
                           
                           ) 
                         
                       
                        
                       dx 
                     
                   
                 
                 + 
                 
                   
                     P 
                      
                     
                       ( 
                       
                         ω 
                         1 
                       
                       ) 
                     
                   
                    
                   
                     
                       ∫ 
                       
                         z 
                         2 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             x 
                             | 
                             
                               ω 
                               1 
                             
                           
                           ) 
                         
                       
                        
                       dx 
                     
                   
                 
               
             
           
         
       
     
     which is the total probability of making an error, where the integral ∫ Z     2    p(x|ω 1 )dx is a conditional probability given the density p(x|ω 1 ) and the decision region Z 2 , and the integral ∫ Z     2    p(x|ω 1 )dx is a conditional probability given the density p(x|ω 2 ) and the decision region Z 1 . Accordingly, the Z 1  and Z 2  decision regions are defined to consist of values of x for which the likelihood ratio Λ(x) is, respectively, less than or greater than a threshold n, where any given set of Z 1  and Z 2  decision regions spans an entire feature space over the interval of (−∞, ∞) 
     The general forms of Bayes&#39; decision rule can be written as: 
     
       
         
           
             
               Λ 
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 
                   ln 
                    
                   
                       
                   
                    
                   
                     p 
                      
                     
                       ( 
                       
                         x 
                         | 
                         
                           ω 
                           1 
                         
                       
                       ) 
                     
                   
                 
                 - 
                 
                   ln 
                    
                   
                       
                   
                    
                   
                     p 
                      
                     
                       ( 
                       
                         x 
                         | 
                         
                           ω 
                           2 
                         
                       
                       ) 
                     
                   
                    
                   
                     
                       
                         
                           &gt; 
                         
                       
                       
                         
                           &lt; 
                         
                       
                     
                     
                       ω 
                       2 
                     
                   
                 
               
               = 
               
                 
                   p 
                    
                   
                     ( 
                     
                       
                         Λ 
                          
                         
                           ( 
                           x 
                           ) 
                         
                       
                       | 
                       
                         ω 
                         1 
                       
                     
                     ) 
                   
                 
                 - 
                 
                   ln 
                    
                   
                       
                   
                    
                   
                     p 
                      
                     
                       ( 
                       
                         
                           Λ 
                            
                           
                             ( 
                             x 
                             ) 
                           
                         
                         | 
                         
                           ω 
                           2 
                         
                       
                       ) 
                     
                   
                    
                   
                     
                       
                         
                           &gt; 
                         
                       
                       
                         
                           &lt; 
                         
                       
                     
                     
                       ω 
                       2 
                     
                   
                 
               
             
           
         
       
     
     where P(ω 1 )=P(ω 2 ), C 11 =C 22 =0 and C 21 =C 12 =1. For Gaussian data, Bayes&#39; decision rule and boundary are completely defined by the likelihood ratio test: 
     
       
         
           
             
               
                 Λ 
                  
                 
                   ( 
                   x 
                   ) 
                 
               
               = 
               
                 
                   
                     
                       
                          
                         
                           ∑ 
                           2 
                         
                          
                       
                       
                         1 
                         / 
                         2 
                       
                     
                      
                     exp 
                      
                     
                       { 
                       
                         
                           - 
                           
                             1 
                             2 
                           
                         
                          
                         
                           
                             ( 
                             
                               x 
                               - 
                               
                                 μ 
                                 1 
                               
                             
                             ) 
                           
                           T 
                         
                          
                         
                           
                             ∑ 
                             1 
                             
                               - 
                               1 
                             
                           
                            
                           
                               
                           
                            
                           
                             ( 
                             
                               x 
                               - 
                               
                                 μ 
                                 1 
                               
                             
                             ) 
                           
                         
                       
                       } 
                     
                   
                   
                     
                       
                          
                         
                           ∑ 
                           1 
                         
                          
                       
                       
                         1 
                         / 
                         2 
                       
                     
                      
                     exp 
                      
                     
                       { 
                       
                         
                           - 
                           
                             1 
                             2 
                           
                         
                          
                         
                           
                             ( 
                             
                               x 
                               - 
                               
                                 μ 
                                 2 
                               
                             
                             ) 
                           
                           T 
                         
                          
                         
                           
                             ∑ 
                             2 
                             
                               - 
                               1 
                             
                           
                            
                           
                               
                           
                            
                           
                             ( 
                             
                               x 
                               - 
                               
                                 μ 
                                 2 
                               
                             
                             ) 
                           
                         
                       
                       } 
                     
                   
                 
                  
                 
                   
                     
                       
                         &gt; 
                       
                     
                     
                       
                         &lt; 
                       
                     
                   
                   
                     ω 
                     2 
                   
                 
                  
                 
                   ? 
                 
                  
                 
                   
                     ( 
                     
                       
                         C 
                         12 
                       
                       - 
                       
                         C 
                         22 
                       
                     
                     ) 
                   
                   
                     ( 
                     
                       
                         C 
                         21 
                       
                       - 
                       
                         C 
                         11 
                       
                     
                     ) 
                   
                 
               
             
             , 
             
               
 
             
              
             
               
                 ? 
               
                
               
                 indicates text missing or illegible when filed 
               
             
           
         
       
     
     where μ 1  and μ 2  are d-component mean vectors, Σ 1  and Σ 2  are d-by-d covariance matrices, Σ −1  and |Σ| denote the inverse and determinant of a covariance matrix, and ω 1  or ω 2  is the true data category. 
     A new theorem for binary classification is motivated next. 
     An important and advantageous feature of the new theorem is that decision regions are redefined in terms of regions that are associated with decision errors or lack thereof. Accordingly, regions associated with decision errors involve regions associated with overlapping data distributions and regions associated with no decision errors involve regions associated with non-overlapping data distributions. 
     For overlapping data distributions, decision regions are defined to be those regions that span regions of data distribution overlap. Accordingly, the Z 1  decision region, which is associated with class ω 1 , spans a region between the region of distribution overlap between p(x|ω 1 ) and p(x|ω 2 ) and the decision threshold n, whereas the Z 2  decision region, which is associated with class ω 2 , spans a region between the decision threshold n and the region of distribution overlap between p(xx|ω 2 ) and p(xx|ω 1 ).  FIG. 1  illustrates how overlapping data distributions determine decision regions. 
     For non-overlapping data distributions, the Z 1  decision region, which is associated with class ω 1 , spans a region between the tail region of p(x|ω 1 ) and the decision threshold η, whereas the Z 2  decision region, which is associated with class ω 2 , spans a region between the decision threshold η and the tail region of p(x|ω 2 ).  FIG. 2  illustrates how non-overlapping data distributions determine decision regions, 
     Take any given decision boundary D(x):Λ(x)=0 that is determined by the vector equation: 
         D ( x ): x   T Σ 1   −1 μ 1 −½ x   T Σ 1   −1   x −½μ 1   T Σ 1   −1 μ 1 −½In(|Σ 1 | 1/2 )− x   T Σ 2   −1 μ 2 −½ x   T Σ 2   −1   x −½μ 2   T Σ 2   −1 μ 1 −½In(|Σ 2 | 1/2 )=0  (1.1)
 
     and is generated according to the transform of the likelihood ratio test In(Λ)(x))In(η) for 
     Gaussian data, where C 11 =C 22 =0, C 12 =C 21 =1, and P(ω 1 )=P(ω 2 )=1/2: 
     
       
         
           
             
               
                 
                   
                     Λ 
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         x 
                         T 
                       
                        
                       
                         
                           ∑ 
                           1 
                           
                             - 
                             1 
                           
                         
                          
                         
                             
                         
                          
                         
                           μ 
                           1 
                         
                       
                     
                     - 
                     
                       
                         1 
                         2 
                       
                        
                       
                         x 
                         T 
                       
                        
                       
                         
                           ∑ 
                           1 
                           
                             - 
                             1 
                           
                         
                          
                         
                             
                         
                          
                         x 
                       
                     
                     - 
                     
                       
                         1 
                         2 
                       
                        
                       
                         μ 
                         1 
                         T 
                       
                        
                       
                         
                           ∑ 
                           1 
                           
                             - 
                             1 
                           
                         
                          
                         
                             
                         
                          
                         
                           μ 
                           1 
                         
                       
                     
                     - 
                     
                       
                         1 
                         2 
                       
                        
                       
                         ln 
                          
                         
                           ( 
                           
                             
                                
                               
                                 ∑ 
                                 1 
                               
                                
                             
                             
                               1 
                               / 
                               2 
                             
                           
                           ) 
                         
                       
                     
                     - 
                     
                       
                         x 
                         T 
                       
                        
                       
                         
                           ∑ 
                           2 
                           
                             - 
                             1 
                           
                         
                          
                         
                             
                         
                          
                         
                           μ 
                           2 
                         
                       
                     
                     - 
                     
                       
                         1 
                         2 
                       
                        
                       
                         x 
                         T 
                       
                        
                       
                         
                           ∑ 
                           2 
                           
                             - 
                             1 
                           
                         
                          
                         
                             
                         
                          
                         x 
                       
                     
                     - 
                     
                       
                         1 
                         2 
                       
                        
                       
                         μ 
                         2 
                         T 
                       
                        
                       
                         
                           ∑ 
                           2 
                           
                             - 
                             1 
                           
                         
                          
                         
                             
                         
                          
                         
                           μ 
                           2 
                         
                       
                     
                     - 
                     
                       
                         1 
                         2 
                       
                        
                       
                         ln 
                          
                         
                           ( 
                           
                             
                                
                               
                                 ∑ 
                                 2 
                               
                                
                             
                             
                               1 
                               / 
                               2 
                             
                           
                           ) 
                         
                       
                        
                       
                         
                           
                             
                               &gt; 
                             
                           
                           
                             
                               &lt; 
                             
                           
                         
                         
                           ω 
                           2 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1.2 
                   ) 
                 
               
             
           
         
       
     
     where the decision space Z and the corresponding decision regions Z 1  and Z 2  of the classification system: 
     
       
         
           
             
               p 
                
               
                 ( 
                 
                   
                     Λ 
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   | 
                   
                     ω 
                     1 
                   
                 
                 ) 
               
             
             - 
             
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                     | 
                     
                       ω 
                       2 
                     
                   
                   ) 
                 
               
                
               
                 
                   
                     
                       &gt; 
                     
                   
                   
                     
                       &lt; 
                     
                   
                 
                 
                   ω 
                   2 
                 
               
             
           
         
       
     
     are determined by either overlapping or non-overlapping data distributions, and decision boundaries D(x):Λ(x)=0 are characterized by the class of hyperquadric decision surfaces which include hyperplanes, pairs of hyperplanes, hyperspheres, hyperellipsoids, hyperparaboloids, and hyperhyperboloids. 
     The general idea of a curve or surface which at any point of it exhibits some uniform property is expressed in geometry by the term locus. Generally speaking, a geometric locus is a curve or surface formed by points, all of which possess some uniform property. Any given geometric locus is determined by either an algebraic or a vector equation, where the locus of an algebraic or a vector equation is the location of all those points whose coordinates are solutions of the equation. 
     Using the general idea of a geometric locus, it follows that any given decision boundary in Eq. (1.1) that is determined by the likelihood ratio test 
     
       
         
           
             
               Λ 
                
               
                 ( 
                 x 
                 ) 
               
             
              
             
               
                 
                   
                     &gt; 
                   
                 
                 
                   
                     &lt; 
                   
                 
               
               
                 ω 
                 2 
               
             
           
         
       
     
     in Eq. (1.2), where me likelihood ratio Λ(x)=p(Λ(x)|ω 1 )−p(Λ(x)|ω 2 ) and the decision boundary D)(x):Λ(x)=0 satisfy the vector equation: 
         p (Λ( x )|ω 1 )− p (Λ( x )|ω 2 )=0,  (1.3)
 
     the statistical equilibrium equation: 
         p (Λ( x )|ω 1 )= p (Λ( x )|ω 2 ),  (1.4)
 
     the corresponding integral equation: 
       ∫ Z   p (Λ( x )|ω 1 ) dΛ=∫   Z   p (Λ( x )|ω 2 ) dΛ   (1.5)
 
     the fundamental integral equation of binary classification: 
     
       
         
           
             
               
                 
                   
                     
                       f 
                        
                       
                         ( 
                         
                           Λ 
                            
                           
                             ( 
                             x 
                             ) 
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         
                           
                             ∫ 
                             
                               Z 
                               1 
                             
                             
                                 
                             
                           
                            
                           
                             
                               p 
                                
                               
                                 ( 
                                 
                                   
                                     Λ 
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                   | 
                                   
                                     ω 
                                     1 
                                   
                                 
                                 ) 
                               
                             
                              
                             d 
                              
                             
                                 
                             
                              
                             Λ 
                           
                         
                         + 
                         
                           
                             ∫ 
                             
                               Z 
                               2 
                             
                             
                                 
                             
                           
                            
                           
                             
                               p 
                                
                               
                                 ( 
                                 
                                   
                                     Λ 
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                   | 
                                   
                                     ω 
                                     1 
                                   
                                 
                                 ) 
                               
                             
                              
                             d 
                              
                             
                                 
                             
                              
                             Λ 
                           
                         
                       
                       = 
                       
                         
                           
                             ∫ 
                             
                               Z 
                               1 
                             
                             
                                 
                             
                           
                            
                           
                             
                               p 
                                
                               
                                 ( 
                                 
                                   
                                     Λ 
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                   | 
                                   
                                     ω 
                                     2 
                                   
                                 
                                 ) 
                               
                             
                              
                             d 
                              
                             
                                 
                             
                              
                             Λ 
                           
                         
                         + 
                         
                           
                             ∫ 
                             
                               Z 
                               2 
                             
                             
                                 
                             
                           
                            
                           
                             
                               p 
                                
                               
                                 ( 
                                 
                                   
                                     Λ 
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                   | 
                                   
                                     ω 
                                     2 
                                   
                                 
                                 ) 
                               
                             
                              
                             d 
                              
                             
                                 
                             
                              
                             Λ 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   1.6 
                   ) 
                 
               
             
           
         
       
     
     and the corresponding integral equation for a classification system in statistical equilibrium: 
         f (Λ( x )):∫ Z     1     p (Λ( x )|ω 1 ) dΛ=∫   Z     2     p (Λ( x )|ω 2 ) dΛ∫   Z     1     p (Λ( x )|ω 1 ) dΛ   (1.7)
 
     is a locus formed by all of the endpoints of pattern vectors x whose coordinates are solutions of the vector equation: 
         D ( x ): x   T Σ 1   −1 μ 1 −½ x   T Σ 1   −1   x −½μ 1   T Σ 1   −1 μ 1 −½In(|Σ 1 | 1/2 )− x   T Σ 2   −1 μ 2 −½ x   T Σ 2   −1   x −½μ 2   T Σ 2   −1 μ 1 −½In(|Σ 2 | 1/2 )=0,
 
     where the endpoints of the pattern vectors x on the locus are located in regions that are either (1) associated with overlapping data distributions or (2) associated with non-overlapping data distributions, 
     Therefore, the equilibrium point p(Λ(x)|ω 1 )−p(Λ(x)|ω 2 )=0 of a classification system involves a locus of points x that jointly satisfy the likelihood ratio test in Eq. (1.2), the decision boundary in Eq. (1.1), and the system of fundamental, vector-based locus equations of binary classification for a classification system in statistical equilibrium in Eqs (1.3)-(1.7). 
     Further, Eqs (6) and (7)indicate that Bayes&#39; risk  (Z|Λ(x)) in the decision space Z involves counter risks  (Z 1 |p(Λ(x)|ω 1 )) and  (Z 2 |p(Λ(x)|ω 2 )) associated with class ω 1  and class ω 2  in the Z 1  and Z 2  decisions regions that are opposing forces for risks  (Z 1 |p(Λ(x)|ω 2 )) and  (Z 2 |p(Λ(x)|ω 1 )) associated with class ω 2  and class ω 1  in the Z 1  and Z 2  decisions regions.  FIG. 3  illustrates that the decision space of a binary classification involves risks and counter risks in each of the decision regions for overlapping data distributions. Thereby, for non-overlapping data distributions, any given decision space is determined by counter risks. 
     Vector-based locus equations have been devised for all of the conic sections and quadratic surfaces: lines, planes, and hyperplanes; d-dimensional parabolas, hyperbolas, and ellipses; and circles and d-dimensional spheres. The form of each vector-based locus equation hinges on both the geometric property and the frame of reference (the coordinate system) of the locus. Moreover, the locus of a point is defined in terms of the locus of a vector. A position vector x is defined to be the locus of a directed, straight line segment formed by two points P 0  and P x  which are at a distance of 
       ∥ x ∥=( x   1   2   +x   2   2 + . . .
 
     from each other, where ∥x∥ denotes the length of a position vector x, such that each point coordinate or vector component x i  is at a signed distance of ∥x∥cos α ij  from the origin P 0 , along the direction of an orthonormal coordinate axis , where cos α ij  is the direction cosine between the vector component x i  and the orthonormal coordinate axis e j . Accordingly, a point is the endpoint on the locus of a position vector. Points and vectors are both denoted by x, and the locus of any given vector x is based on the above definition. 
     The vector-based locus equations have been used to identify important and advantageous features of conic sections and quadratic surfaces: 
     The uniform properties exhibited by all of the points x on any given linear locus are specified by the locus of its principal eigenaxis v, where each point x on the linear locus and the principal eigenaxis v of the linear locus satisfy the linear locus in terms of the eigenenergy ∥v∥ 2  exhibited by its principal eigenaxis v. Accordingly, the vector components of a principal eigenaxis specify all forms of lines, planes, and hyperplanes, and all of the points x on any given linear curve or surface explicitly and exclusively reference the principal eigenaxis v of the linear locus. Therefore, the important generalizations and properties for a linear locus are specified by the eigenenergy exhibited by the locus of its principal eigenaxis, and the principal eigenaxis of a linear locus provides an elegant, general eigen-coordinate system for a linear locus of points. 
     The uniform properties exhibited by all of the points x on any given quadratic locus are specified by the locus of its principal eigenaxis v, where each point x on the quadratic locus and the principal eigenaxis v of the quadratic locus satisfy the quadratic locus in terms of the eigenenergy ∥v∥ 2  exhibited by its principal eigenaxis v. Accordingly, the vector components of a principal eigenaxis specify all forms of quadratic curves and surfaces, and all of the points x on any given quadratic curve or surface explicitly and exclusively reference the principal eigenaxis v of the quadratic locus. Therefore, the important generalizations and properties for a quadratic locus are specified by the eigenenergy exhibited by the locus of its principal eigenaxis, and the principal eigenaxis of a quadratic locus provides an elegant, general eigen-coordinate system for a quadratic locus of points. 
     In summary, the vector-based locus equations of conic sections and quadratic surfaces determine an elegant, general eigen-coordinate system for each class of conic sections and quadratic surfaces and a uniform property that is exhibited by all of the points on any given conic section or quadratic surface. Moreover, the vector-based locus equations establish that the locus of points x that satisfies the decision boundary D(x):Λ(x)=0 in Eq. (1.1) must involve a locus of principal eigenaxis components that satisfies the decision boundary D(x) in terms of a total allowed eigenenergy. 
     Take the system of fundamental locus equations of binary classification for a classification system in statistical equilibrium that must be satisfied by Bayes&#39; likelihood ratio: 
       Λ( x )= p (Λ( x )|ω 1 )− p (Λ( x )|ω 2 )
 
     and decision boundary: 
         D ( x ): p (Λ( x )|ω 1 )− p (Λ( x )|ω 2 )=0,
 
     where the decision boundary D(x):Λ(x)=0 and the likelihood ratio Λ(x) satisfy Eqs (1.3)-(1.7). 
     Given that the locus of a conic section or a quadratic surface is determined by the locus of its principal eigenaxis, it follows that the vector-based locus equation in Eq. (1.3) 
         p (Λ( x )|ω 1 )− p (Λ( x )|ω 2 )=0,
 
     that is satisfied by the likelihood ratio Λ(x) and the decision boundary 1)(x):Λ(x)=0 must involve a parameter vector of likelihoods and a corresponding locus of principal eigenaxis components that delineates a decision boundary. Furthermore, the locus of principal eigenaxis components must satisfy the decision boundary D(x):Λ(x)=0 in terms of a total allowed eigenenergy, where the total allowed eigenenergy of a classification system is the eigenenergy associated with the position or location of the likelihood ratio Λ(x)=p(Λ(x)|ω 1 )−p(Λ(x)|ω 2 ) and the locus of a corresponding decision boundary D(x):pΛ(x)=p(Λ(x)|ω 1 )−p(Λ(x)|ω 2 )=0. 
     The new theorem for binary classification can be stated as follows. 
     Let 
     
       
         
           
             
               p 
                
               
                 ( 
                 
                   
                     Λ 
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   | 
                   
                     ω 
                     1 
                   
                 
                 ) 
               
             
             - 
             
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                     | 
                     
                       ω 
                       2 
                     
                   
                   ) 
                 
               
                
               
                 
                   
                     
                       &gt; 
                     
                   
                   
                     
                       &lt; 
                     
                   
                 
                 
                   ω 
                   2 
                 
               
             
           
         
       
     
     denote the likelihood ratio test for a binary classification system, where ω 1  or ω 2  is the true data category, and d-component random vectors x from class ω 1  and class ω 2  are generated according to probability density functions p(x|ω 1 ) and p(x|ω 2 ) related to statistical distributions of random vectors x that have constant or unchanging statistics. 
     The discriminant function 
       Λ( x )= p (Λ( x )|ω 1 )− p (Λ( x )|ω 2 )
 
     is the solution to the integral equation 
     
       
         
           
             
               
                 f 
                  
                 
                   ( 
                   
                     Λ 
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   ) 
                 
               
               = 
               
                 
                   
                     
                       ∫ 
                       
                         Z 
                         1 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             
                               Λ 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                              
                             
                               ω 
                               1 
                             
                           
                           ) 
                         
                       
                        
                       d 
                        
                       
                           
                       
                        
                       Λ 
                     
                   
                   + 
                   
                     
                       ∫ 
                       
                         Z 
                         2 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             
                               Λ 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                              
                             
                               ω 
                               1 
                             
                           
                           ) 
                         
                       
                        
                       d 
                        
                       
                           
                       
                        
                       Λ 
                     
                   
                 
                 = 
                 
                   
                     
                       ∫ 
                       
                         Z 
                         1 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             
                               Λ 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                              
                             
                               ω 
                               2 
                             
                           
                           ) 
                         
                       
                        
                       d 
                        
                       
                           
                       
                        
                       Λ 
                     
                   
                   + 
                   
                     
                       ∫ 
                       
                         Z 
                         2 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             
                               Λ 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                              
                             
                               ω 
                               2 
                             
                           
                           ) 
                         
                       
                        
                       d 
                        
                       
                           
                       
                        
                       Λ 
                     
                   
                 
               
             
             , 
           
         
       
     
     over the decision space Z=Z 1 +Z 2 , such that the Bayes&#39; risk  (Z|Λ(x)) and the corresponding eigenenergy E min (Z|Λ(x) of the classification system 
     
       
         
           
             
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       1 
                     
                   
                   ) 
                 
               
               - 
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       2 
                     
                   
                   ) 
                 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     are governed by the equilibrium point) 
         p (Λ( x )|ω 1 )− p (Λ( x )|ω 2 )=0
 
     of the integral equation f(Λ(x)). 
     Therefore, the forces associated with Bayes&#39; counter risk  (Z 1 |p(Λ(x)|ω 1 )) and Bayes&#39; risk  (Z 2 |p(Λ(x)|ω 1 )) in the Z 1  and Z 2  decision regions, which are related to positions and potential locations of random vectors x that are generated according to p(Λ(x)|ω 1 ), are equal to the forces associated with Bayes&#39; risk  (Z 1 |p(Λ(x)|ω 2 )) and Bayes&#39; counter risk  (Z 2 |p(Λ(x)|ω 2 )) in the Z 1  and Z 2  decision regions, which are related to positions and potential locations of random vectors x that are generated according to p(x|ω 2 ). 
     Furthermore, the eigenenergy E min (Z|p(Λ(x)|ω 1 )) associated with the position or location of the likelihood ratio p(Λ(x)|ω 1 ) given class ω 1  is equal to the eigenenergy) E min (Z|p(Λ(x)|ω 2 )) associated with the position or location of the likelihood ratio p(Λ(x)|ω 2 ) given class ω 2 : 
         E   min ( Z|p (Λ( x )|ω 1 ))= E   min ( Z|p (Λ( x )|ω 2 )).
 
     Thus, the total eigenenergy E min (Z|Λ(x)) of the binary classification system 
     
       
         
           
             
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       1 
                     
                   
                   ) 
                 
               
               - 
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       2 
                     
                   
                   ) 
                 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     is equal to the eigenenergies associated with the position or location of the likelihood ratio p(Λ(x)|ω 1 )−p(Λ(x)|ω 2 ) and the locus of a corresponding decision boundary D(x): p(Λ(x)|ω 1 )−p(Λ(x)|ω 2 )=0: 
         E   min ( Z|p (Λ( x ))= E   min ( Z|p (Λ( x )|ω 1 )+i E min ( Z|p (Λ( x )|ω 2 )).
 
     It follows that the binary classification system 
     
       
         
           
             
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       1 
                     
                   
                   ) 
                 
               
               - 
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       2 
                     
                   
                   ) 
                 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     is in statistical equilibrium: 
         f (Λ( x )):∫ Z     1     p (Λ( x )|ω 1 ) dΛ−∫   Z     1     p (Λ( x )|ω 2 ) dΛ=∫   Z     1     p (Λ( x )|ω 2 ) dΛ−∫   Z     2     p (Λ( x )|ω 1 ) dΛ,  
 
     where the forces associated with Bayes&#39; counter risk  (Z 1 |p(Λ(x)|ω 1 )) for class ω 1  and Bayes&#39; risk  (Z 1 |p(Λ(x)|ω 2 )) for class ω 2  in the Z 1  decision region are balanced with the forces associated with Bayes&#39; counter risk  (Z 2 |p(Λ(x)|ω 2 )) for class ω 2 2  and Bayes&#39; risk  (Z 2 |p(Λ(x)|ω 1 )) for class ω 1  in the Z 2  decision region such that the Bayes&#39; risk  (Z|Λ(x)) of the classification system is minimized, and the eigenenergies associated with Bayes&#39; counter risk  (Z 1 |p(Λ(x)|ω 1 )) for class ω 1  and Bayes&#39; risk  (Z 1 |p(Λ(x)|ω 2 )) for class ω 2  in the Z 1  decision region are balanced with the eigenenergies associated with Bayes&#39; counter risk  (Z 2 |p(Λ(x)|ω 2 )) for class ω 2  and Bayes&#39; risk  (Z 2 |p(Λ(x)|ω 1 )) for class ω 1  in the Z 2  decision region such that the eigenenergy E min (Z|Λ(x)) of the classification system is minimized. Thus, any given binary classification system 
     
       
         
           
             
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       1 
                     
                   
                   ) 
                 
               
               - 
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       2 
                     
                   
                   ) 
                 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     exhibits an error rate that is consistent with the Bayes&#39;risk  (Z|Λ(x)) and the corresponding eigenenergy E min (Z|Λ(x)) of the classification system: for all random vectors x that are generated according to p(x|ω 1 ) and p(x|ω 2 ), where p(x|ω 1 ) and p(x|ω 2 ) are related to statistical distributions of random vectors x that have constant or unchanging statistics. 
     Therefore, the Bayes&#39; risk  (Z|Λ(x) and the corresponding eigenenergy E min (Z|Λ(x)) of the classification system 
     
       
         
           
             
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       1 
                     
                   
                   ) 
                 
               
               - 
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       2 
                     
                   
                   ) 
                 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     are governed by the equilibrium point 
         p (Λ( x )|ω 1 )− p (Λ( x )|ω 2 )=0
 
     of the integral equation 
     
       
         
           
             
               
                 f 
                  
                 
                   ( 
                   
                     Λ 
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   ) 
                 
               
               = 
               
                 
                   
                     
                       ∫ 
                       
                         Z 
                         1 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             
                               Λ 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                              
                             
                               ω 
                               1 
                             
                           
                           ) 
                         
                       
                        
                       d 
                        
                       
                           
                       
                        
                       Λ 
                     
                   
                   + 
                   
                     
                       ∫ 
                       
                         Z 
                         2 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             
                               Λ 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                              
                             
                               ω 
                               1 
                             
                           
                           ) 
                         
                       
                        
                       d 
                        
                       
                           
                       
                        
                       Λ 
                     
                   
                 
                 = 
                 
                   
                     
                       ∫ 
                       
                         Z 
                         1 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             
                               Λ 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                              
                             
                               ω 
                               2 
                             
                           
                           ) 
                         
                       
                        
                       d 
                        
                       
                           
                       
                        
                       Λ 
                     
                   
                   + 
                   
                     
                       ∫ 
                       
                         Z 
                         2 
                       
                       
                           
                       
                     
                      
                     
                       
                         p 
                          
                         
                           ( 
                           
                             
                               Λ 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                              
                             
                               ω 
                               2 
                             
                           
                           ) 
                         
                       
                        
                       d 
                        
                       
                           
                       
                        
                       Λ 
                     
                   
                 
               
             
             , 
           
         
       
     
     over the decision space Z=Z 1 +Z 2 , where the opposing forces and influences of the classification system are balanced with each other, such that the eigenenergy and the Bayes&#39; risk of the classification system are minimized, and the classification system is in statistical equilibrium, 
     Moreover, the eigenenergy E min (Z|Λ(x)) is the state of a binary classification system 
     
       
         
           
             
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       1 
                     
                   
                   ) 
                 
               
               - 
               
                 p 
                  
                 
                   ( 
                   
                     
                       Λ 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       ω 
                       2 
                     
                   
                   ) 
                 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     that is associated with the position or location of a likelihood ratio in statistical equilibrium: p(Λ(x)|ω 1 )−p(Λ(x)|ω 2 )=0 and the locus of a corresponding decision boundary: D(x): p(Λ(x)|ω 1 )−p(Λ(x)|ω 2 )=0. 
     The binary classification theorem that is outlined above has unique, important, and advantageous features. The theorem establishes new and essential criteria for the binary classification problem: which is the fundamental technical problem that underlies all automated decision making and statistical pattern recognition applications. The theorem establishes the existence of a system of fundamental, vector-based, locus equations of binary classification for a classification system in statistical equilibrium that must be satisfied by Bayes&#39; likelihood ratio and decision boundary. Further, the theorem provides the result that any given binary classification system seeks a point of statistical equilibrium where the opposing forces and influences of the classification system are balanced with each other, so that the eigenenergy and the Bayes&#39; risk of the classification system are minimized, and the classification system is in statistical equilibrium, 
     Moreover, given that new geometric locus methods establish that the vector components of a principal eigenaxis specify all forms of conic curves and quadratic surfaces, such that all of the points on any given conic curve or quadratic surface explicitly and exclusively reference the principal eigenaxis of the conic section or quadratic surface, and that the principal eigenaxis and all of the points of any given conic section or quadratic surface satisfy the eigenenergy exhibited by its principal eigenaxis, it follows that the locus of points x that satisfies the decision boundary D(x): Λ(x)=0 in Eq. (1.1) must involve a locus of principal eigenaxis components that satisfies the decision boundary D(x): Λ(x)=0 in terms of a critical minimum or total allowed eigenenergy. 
     Therefore, the system of fundamental locus equations of binary classification for a classification system in statistical equilibrium that must be satisfied by Bayes&#39; likelihood ratio Λ(x) and decision boundary D(x):Λ(x)=0 involves a dual locus of likelihoods and principal eigenaxis components: i.e., a parameter vector of likelihoods that satisfies a decision boundary in terms of a minimum amount of risk and a corresponding locus of principal eigenaxis components that satisfies a decision boundary in terms of a critical minimum eigenenergy. Moreover, because the decision space Z=Z 1 +Z 2  of a binary classification system is determined by decision regions Z 1  and Z 2  that are associated with either overlapping regions or tail regions between two data distributions, the dual locus of likelihoods and principal eigenaxis components must be formed by feature vectors that lie in either overlapping regions or tail regions between two data distributions. Feature vectors that lie in either overlapping regions or tail regions between two data distributions are called extreme points. Extreme points are fundamental components of computer-implemented binary classification systems. Properties of extreme points are outlined next. 
     Take a collection of feature vectors for any two pattern classes drawn from any two statistical distributions, where the data distributions are either overlapping or non-overlapping with each other. An extreme point is defined to be a data point which exhibits a high variability of geometric location, that is, possesses a large covariance, such that it is located (1) relatively far from its distribution mean, (2) relatively close to the mean of the other distribution, and (3) relatively close to other extreme points. Therefore, an extreme point is located somewhere within either an overlapping region or a tail region between the two data distributions. Given the geometric and statistical properties exhibited by the locus of an extreme point, it follows that a set of extreme vectors determine principal directions of large covariance for a given collection of training data. Thus, extreme vectors are discrete principal components that specify directions for which a given collection of training data is most variable or spread out. Any given extreme point is characterized by an expected value (a central location) and a covariance (a spread). Thereby, distributions of extreme points determine decision regions for binary classification systems, where the forces associated with Bayes&#39; risks and Bayes&#39; counter risks are related to positions and potential locations of extreme data points. 
     The innovative concept is outlined next. The innovative concept involves a system of fundamental, data-driven, vector-based locus equations of binary classification for a linear classification system in statistical equilibrium that generates a data-driven likelihood ratio test that contains Bayes&#39; likelihood ratio and automatically generates the best linear decision boundary. The data-driven likelihood ratio test, which is based on a dual locus of likelihoods and principal eigenaxis components formed by a locus of weighted extreme points, satisfies fundamental statistical laws for a binary classification system in statistical equilibrium and is the basis of a linear classification system for which the eigenenergy and the Bayes&#39; risk are minimized, so that the opposing forces and influences of the classification system are balanced with each other, and the classification system achieves Bayes&#39; error rate: for feature vectors drawn from statistical distributions that have similar covariance functions, 
     A dual locus of likelihoods and principal eigenaxis components formed by a locus of weighted extreme points is a locus of principal eigenaxis components that delineates the best linear decision boundary as well as a parameter vector of likelihoods of extreme points that satisfies the linear decision boundary in terms of a minimum amount of risk. Any given dual locus of likelihoods and principal eigenaxis components has the following unique and advantageous features. 
     The dual locus of likelihoods and principal eigenaxis components satisfies a data-driven version of the system of fundamental locus equations of binary classification for a classification system in statistical equilibrium in Eqs (1.3)-(1.7). 
     The dual locus of principal eigenaxis components provides an elegant, statistical eigen-coordinate system for a linear classification system. Further, the dual locus of principal eigenaxis components satisfies a critical minimum, i.e., a total allowed, eigenenergy constraint, so that the locus of principal eigenaxis components satisfies a linear decision boundary in terms of its critical minimum or total allowed eigenenergies. 
     The dual locus of likelihoods and principal eigenaxis components is formed by extreme points, i.e., extreme feature vectors or extreme pattern vectors, that lie in either overlapping regions or tail regions between two data distributions, thereby determining decision regions based on forces associated with Bayes&#39; risks and Bayes&#39; counter risks: which are related to positions and potential locations of the extreme points, where an unknown portion of the extreme points are the source of Bayes&#39; decision error. 
     The dual locus of likelihoods and principal eigenaxis components is the basis of a linear classification system for which the eigenenergy and the Bayes&#39; risk are minimized, so that the opposing forces and influences of the classification system are balanced with each other, and the classification system achieves Bayes&#39; error rate: for feature vectors drawn from statistical distributions that have similar covariance functions. 
     The dual locus of likelihoods and principal eigenaxis components is a parameter vector of likelihoods of extreme points that contains Bayes&#39; likelihood ratio. 
     The inventor has named a dual locus of likelihoods and principal eigenaxis components formed by weighted extreme points a “linear eigenlocus.” The inventor has named the related parameter vector that provides an estimate of class-conditional densities for extreme points a “locus of likelihoods,” The inventor has named the system of fundamental, data-driven, vector-based locus equations of binary classification for a classification system in statistical equilibrium that generates a linear eigenlocus a “linear eigenlocus transform.” 
     Linear eigenlocus transforms generate a locus of weighted extreme points that is a dual locus of likelihoods and principal eigenaxis components, where each weight specifies a class membership statistic and conditional density for an extreme point and each weight determines the magnitude and the total allowed eigenenergy of an extreme vector, 
     Linear eigenlocus transforms generate a set of weights that satisfy the following criteria: 
     Criterion 1: Each conditional density of an extreme point describes the central location (expected value) and the spread (covariance) of the extreme point. 
     Criterion 2: Distributions of the extreme points are distributed over the locus of likelihoods in a symmetrically balanced and well-proportioned manner. 
     Criterion 3: The total allowed eigenenergy possessed by each weighted extreme vector specifies the probability of observing the extreme point within a localized region. 
     Criterion 4: The total allowed eigenenergies of the weighted extreme vectors are symmetrically balanced with each other about the center of total allowed eigenenergy. 
     Criterion 5: The forces associated with Bayes&#39; risks and Bayes&#39; counter risks related to the weighted extreme points are symmetrically balanced with each other about the center of Bayes&#39; risk. 
     Criterion 6: The locus of principal eigenaxis components formed by weighted extreme vectors partitions any given feature space into congruent decision regions which are symmetrically partitioned by a linear decision boundary. 
     Criterion 7: The locus of principal eigenaxis components is the focus of a linear decision boundary. 
     Criterion 8: The locus of principal eigenaxis components formed by weighted extreme vectors satisfies the linear decision boundary in terms of a critical minimum eigenenergy. 
     Criterion 9: The locus of likelihoods formed by weighted extreme points satisfies the linear decision boundary in terms of a minimum probability of decision error. 
     Criterion 10: For data distributions that have dissimilar covariance matrices, the forces associated with Bayes&#39; counter risks and Bayes&#39; risks, within each of the congruent decision regions, are balanced with each other. For data distributions that have similar covariance matrices, the forces associated with Bayes&#39; counter risks within each of the congruent decision regions are equal to each other, and the forces associated with Bayes&#39; risks within each of the congruent decision regions are equal to each other. 
     Criterion 11: For data distributions that have dissimilar covariance matrices, the eigenenergies associated with Bayes&#39; counter risks and the eigenenergies associated with Bayes&#39; risks, within each of the congruent decision regions, are balanced with other. For data distributions that have similar covariance matrices, the eigenenergies associated with Bayes&#39; counter risks within each of the congruent decision regions are equal to each other, and the eigenenergies associated with Bayes&#39; risks within each of the congruent decision regions are equal to each other. 
     The system of data-driven, locus equations that generates likelihood ratios and decision boundaries satisfies all of the above criteria. The set of criteria involve a unique and advantageous statistical property that the inventor has named “symmetrical balance.” This unique and advantageous feature ensures that learning machines generated by linear eigenlocus transforms exhibit optimal generalization performance and have the highest possible accuracy: for feature vectors drawn from statistical distributions that have similar covariance functions. 
     Symmetrical balance can be described as having an even distribution of “weight” or a similar “load” on equal sides of a centrally placed fulcrum. As a practical example, consider the general machinery of a fulcrum and a lever, where a lever is any rigid object capable of turning about some fixed point called a fulcrum. If a fulcrum is placed under directly under a lever&#39;s center of gravity, the lever will remain balanced. Accordingly, the center of gravity is the point at which the entire weight of a lever is considered to be concentrated, so that if a fulcrum is placed at this point, the lever will remain in equilibrium. If a lever is of uniform dimensions and density, then the center of gravity is at the geometric center of the lever. For example, consider the playground device known as a seesaw or teeter-totter. The center of gravity is at the geometric center of a teeter-totter, which is where the fulcrum of a seesaw is located. Accordingly, the physical property of symmetrical balance involves a physical system in equilibrium, whereby the opposing forces or influences of the system are balanced with each other. 
     The statistical property of symmetrical balance involves a data-driven, binary classification system in statistical equilibrium, whereby the opposing forces or influences of the classification system are balanced with each other, and the eigenenergy and Bayes&#39; risk of the classification system are minimized. Linear eigenlocus transforms generate a data-driven likelihood ration test that is based on a dual locus of principal eigenaxis components and likelihoods, formed by a locus of weighted extreme points, all of which exhibit the statistical property of symmetrical balance. The dual locus provides an estimate of a principal eigenaxis that has symmetrically balanced distributions of eigenenergies on equal sides of a centrally placed fulcrum, which is located at its center of total allowed eigenenergy. The dual locus also provides an estimate of a parameter vector of likelihoods that has symmetrically balanced distributions of forces associated with Bayes&#39; risks and Bayes&#39; counter risks on equal sides of a centrally placed fulcrum, which is located at the center of Bayes&#39; risk. Thereby, a dual locus of principal eigenaxis components and likelihoods is in statistical equilibrium. 
     Linear eigenlocus transforms involve solving an inequality constrained optimization problem. 
     Take any given collection of training data for a binary classification problem of the form: 
     
       
         
           
             
               
                   
               
                
               
                 
                   ( 
                   
                     
                       x 
                       1 
                     
                     , 
                     
                       y 
                       1 
                     
                   
                   ) 
                 
                 , 
                 
                   
                     ? 
                   
                    
                   
                       
                   
                    
                   … 
                    
                   
                       
                   
                    
                 
                 , 
               
               } 
             
             , 
             
               
 
             
              
             
               
                 ? 
               
                
               
                 indicates text missing or illegible when filed 
               
             
           
         
       
     
     where feature vectors x from class ω 1  and class ω 2  are drawn from unknown, class-conditional probability density functions p(x|ω 1 ) and p(x|ω 2 ) and are identically distributed. Feature vectors x can be extracted from any given source of digital data: i.e., digital images, digital videos, digital signals, or digital waveforms, and are labeled. 
     A linear eigenlocus τ is estimated by solving an inequality constrained optimization problem: 
       minΨ(τ)=∥τ∥ 2 /2+ C/ 2Σ i=1   N ξ i   2   , s.t. y   i ( x   i   T τ+τ 0 )≥1−ξ i , ξ i ≥0, i= 1 ,   (1.8)
 
     where τ is a d×1 constrained, primal linear eigenlocus which is a dual locus of likelihoods and principal eigenaxis components, ∥τ∥ 2  is the total allowed eigenenergy exhibited by τ, τT 0  is a functional of τ, C and ξ i  are regularization parameters, and y i  are class membership statistics: if x i ϵω 1 , assign y i =+1; if x i ϵω 2 , assign y i =−1. 
     Equation (1.8) is the primal problem of a linear eigenlocus, where the system of N inequalities must be satisfied: 
         y   i ( x   i   T τ+τ 0 )≥1−ϵ i , ϵ i ≥0, = 1 ,  
 
     such that τ satisfies a critical minimum eigenenergy constraint: 
         y (τ)=∥τ∥ min     c     2 ,  (1.9)
 
     where the total allowed eigenenergy ∥τ∥ min     c     2  exhibited by τ determines the Bayes&#39; risk  (Z|τ) of a linear classification system. 
     Solving the inequality constrained optimization problem in Eq. (1.8) involves solving a dual optimization problem that determines the fundamental unknowns of Eq. (1.8). Denote a Wolfe dual linear eigenlocus by Ψ, and the Lagrangian dual problem of Ψ by max Ξ(Ψ). Let Ψ be a Wolfe dual of κ such that proper and effective strong duality relationships exist between the algebraic systems of min Ψ(τ) and max Ξ(Ψ). Thereby, let Ψ be related with τ in a symmetrical manner that specifies the locations of the principal eigenaxis components on τ. 
     For the problem of linear eigenlocus transforms, the Lagrange multipliers method introduces a Wolfe dual linear eigenlocus Ψ of principal eigenaxis components, for which the Lagrange multipliers {Ψ i } i=1   N  are the magnitudes or lengths of a set of Wolfe dual principal eigenaxis components 
     
       
         
           
             
               
                 { 
                 
                   
                     ψ 
                     i 
                   
                    
                   
                     
                       e 
                       → 
                     
                     i 
                   
                 
                 } 
               
               
                 i 
                 = 
                 1 
               
             
             , 
           
         
       
     
     where 
     
       
         
           
             
               { 
               
                 
                   e 
                   → 
                 
                 i 
               
               } 
             
             
               i 
               = 
               1 
             
           
         
       
     
     are non-orthogonal unit vectors and finds extrema for the restriction of τ to a Wolfe dual eigenspace. The fundamental unknowns associated with Eq. (1.8) are the magnitudes or lengths of the Wolfe dual principal eigenaxis components on Ψ: scale factors of the principal eigenaxis components on Ψ. Each scale factor specifies a conditional density for a weighted extreme point on a locus of likelihoods, and each scale factor determines the magnitude and the eigenenergy of a weighted extreme vector on a locus of principal eigenaxis components. 
     The inequality constrained optimization problem in Eq. (1.8) is solved by using Lagrange multipliers Ψ i ≥0 and the Lagrangian: 
         L   Ψ(τ) (τ, τ 0 , ξ, Ψ)=∥τ∥ 2 /2+ C/ 2Σ i=1   N ξ i   2 −Σ i=1   N Ψ i   {y   i ( x   i   T τ+τ 0 )−1+ξ i }  (1.10)
 
     which is minimized with respect to the primal variables τ and τ 0  and is maximized with respect to the dual variables Ψ i . 
     The Karush-Kuhn-Tucker (KKT) conditions on the Lagrangian L Ψ(τ) : 
       τ−Σ i=1   N Ψ i   y   i   x   i =0,  i= 1,  (1.11)
 
       Σ i=1   N Ψ i   y   i =0,  i= 1,  (1.12)
 
         CΣ   i=1   N ξ i −Σ i=1   N Ψ i =0,  i= 1,  (1.13)
 
       Ψ i ≥0, i=1,  (1.14)
 
       Ψ i   [y   i ( x   i   T τ+τ 0 )−1+ξ i ]≥0,  i= 1  (1.15)
 
     determine a system of fundamental, data-driven locus equations of binary classification for a linear classification system in statistical equilibrium that are jointly satisfied by τ and Ψ. The system of locus equations is a data-driven version of Eqs (1.3)-(1.7). 
     The resulting expressions for τ in Eq. (1.11) and Ψ in Eq. (1.12) are substituted into the Lagrangian functional L Ψ(κ)  of Eq. (1.10) and simplified. This produces the Lagrangian dual problem: 
     
       
         
           
             
               
                 
                   
                     max 
                      
                     
                         
                     
                      
                     Ξ 
                      
                     
                         
                     
                      
                     
                       ( 
                       Ψ 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         N 
                       
                        
                       
                         ψ 
                         i 
                       
                     
                     - 
                     
                       
                         ∑ 
                         
                           i 
                           , 
                           
                             j 
                             = 
                             1 
                           
                         
                         N 
                       
                        
                       
                         
                           ψ 
                           i 
                         
                          
                         
                           ψ 
                           j 
                         
                          
                         
                           y 
                           i 
                         
                          
                         
                           y 
                           j 
                         
                          
                         
                           
                             
                               
                                 x 
                                 i 
                                 T 
                               
                                
                               
                                 x 
                                 j 
                               
                             
                             + 
                             
                               
                                 δ 
                                 ij 
                               
                                
                               
                                 / 
                               
                                
                               C 
                             
                           
                           2 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1.16 
                   ) 
                 
               
             
           
         
       
     
     which is subject to the algebraic constraints Σ i=1   N Ψ i y i =0, and Ψ i ≥0, where δ ij  is the Kronecker δ defined as unity for i=j and 0 otherwise. 
     Equation (1.16) is a quadratic programming problem that can be written in vector notation by letting 
     
       
         
           
             
                 
             
              
             
               
                 Q 
                  
                 
                   = 
                   Δ 
                 
                  
                 
                   
                     
                       ? 
                     
                      
                     
                         
                     
                      
                     and 
                      
                     
                         
                     
                      
                     X 
                   
                    
                   
                     = 
                     Δ 
                   
                    
                   
                     ? 
                   
                 
               
               , 
               
                 
 
               
                
               
                 
                   ? 
                 
                  
                 
                   indicates text missing or illegible when filed 
                 
               
             
           
         
       
     
     where D is an N×N diagonal matrix of training labels (class membership statistics) y, and the N×d data matrix is 
     
       
         
           
             
                 
             
              
             
               X 
               = 
               
                 ( 
                 
                   
                     x 
                     1 
                   
                   , 
                   
                     x 
                     2 
                   
                   , 
                   
                     … 
                      
                     
                       ? 
                     
                      
                     
                       
 
                     
                      
                     
                       ? 
                     
                      
                     
                       indicates text missing or illegible when filed 
                     
                   
                 
               
             
           
         
       
     
     This produces the matrix version of the Lagrangian dual problem: 
     
       
         
           
             
               
                 
                   
                     max 
                      
                     
                         
                     
                      
                     Ξ 
                      
                     
                         
                     
                      
                     
                       ( 
                       Ψ 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         1 
                         T 
                       
                        
                       Ψ 
                     
                     - 
                     
                       
                         
                           Ψ 
                           T 
                         
                          
                         Q 
                          
                         
                             
                         
                          
                         Ψ 
                       
                       2 
                     
                   
                 
               
               
                 
                   ( 
                   1.17 
                   ) 
                 
               
             
           
         
       
     
     which is subject to the constraints Ψ T y=0 and Ψ i ≥0. Given the theorem for convex duality, it follows that Ψ is a dual locus of likelihoods and principal eigenaxis components, where Ψ exhibits a total allowed eigenenergy ∥Ψ∥ min     c     2  that is symmetrically related to the total allowed eigenenergy ∥τ∥ min     c     2  of τ: ∥Ψ∥ min     c     2 =τ 
     Using the KKT conditions in Eqs (1.11) and (1.14), it follows that τ satisfies the following locus equation: 
       τ=Σ i=1   N   y   i Ψ i   x   i ,  (1.18)
 
     where the y i  terms are class membership statistics (if x i  is a member of class ω 1 , assign y i =+1; otherwise, assign y i =−1) and the magnitude Ψ i  of each principal eigenaxis component Ψ i {right arrow over (e)} i  on Ψ is greater than or equal to zero: Ψ i ≥0. Data points x i  correlated with Wolfe dual principal eigenaxis components Ψ i {right arrow over (e)} i  that have non-zero magnitudes Ψ i &gt;0 are termed extreme vectors. 
     All of the principal eigenaxis components on ti are labeled extreme points in   Denote the labeled, scaled extreme vectors that belong to class ω 1  and ω 2  by Ψ 1i* x 1i*  and −Ψ 2i* x 2i* , with scale factors: Ψ 1i*  and Ψ 2i* , extreme vectors: x 1i*  and x 2i* , and labels: y i =+1 and y i =−1 respectively. Let there be l i  labeled, scaled extreme vectors {Ψ 1i* x 1i* } i−1   l     1    and l 2  labeled, scaled extreme vectors {Ψ 2i* x 2i* } i−1   l     2   . 
     Given Eq. (1.18) and the assumptions outlined above, it follows that τ is based on the vector difference between a pair of components: 
     
       
         
           
             
               
                 
                   
                     
                       
                         τ 
                         = 
                           
                          
                         
                           
                             τ 
                             1 
                           
                           - 
                           
                             τ 
                             2 
                           
                         
                       
                     
                   
                   
                     
                       
                         
                           = 
                             
                            
                           
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 
                                   l 
                                   i 
                                 
                               
                                
                               
                                 
                                   ψ 
                                   
                                     1 
                                      
                                     
                                       i 
                                       * 
                                     
                                   
                                 
                                  
                                 
                                   x 
                                   
                                     1 
                                      
                                     
                                       i 
                                       * 
                                     
                                   
                                 
                               
                             
                             - 
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 
                                   l 
                                   2 
                                 
                               
                                
                               
                                 
                                   ψ 
                                   
                                     2 
                                      
                                     
                                       i 
                                       * 
                                     
                                   
                                 
                                  
                                 
                                   x 
                                   
                                     2 
                                      
                                     
                                       i 
                                       * 
                                     
                                   
                                 
                               
                             
                           
                         
                         , 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1.19 
                   ) 
                 
               
             
           
         
       
     
     where the components τ 1 =Σ i=1   l     1   Ψ 1i* x 1i*  and τ 2 =Σ i=1   l     2   Ψ 2i* x 2i*  are denoted by are denoted by τ 1  and τ 2  respectively. The scaled extreme points on τ 1  and τ 2  determine the loci of τ 1  and τ 2  and therefore determine the dual locus of τ=τ 1 −τ 2 . 
     The number and the locations of the principal eigenaxis components on Ψ and τ are considerably affected by the rank and eigenspectrum of the Gram matrix Q. Low rank Gram matrices Q generate “weak dual” linear eigenlocus transforms that produce irregular, linear partitions of decision spaces. These problems are solved by the regularization method that is described next. 
     For any collection of N training vectors of dimension d, where d&lt;N, the Gram matrix Q has low rank. The regularized form of Q, for which ϵ&lt;&lt; and 
     
       
         
           
             
                 
             
              
             
               
                 Q 
                  
                 
                   = 
                   Δ 
                 
                  
                 
                   ? 
                 
               
               , 
               
                 
 
               
                
               
                 
                   ? 
                 
                  
                 
                   indicates text missing or illegible when filed 
                 
               
             
           
         
       
     
     ensures that Q has full rank and a complete eigenvector set: so that Q has a complete eigenspectrum. The regularization constant C is related to the regularization parameter ϵ by (1C). 
     For N training vectors of dimension d, where d&lt;N, all of the regularization parameters {ξ i } i=1   N  in Eq. (1.8) and all of its derivatives are set equal to a very small value: ξ i =ξ&lt;&lt;. The regularization constant C is set equal to 
     
       
         
           
             
               
                 1 
                 ξ 
               
               : 
               C 
             
             = 
             
               
                 1 
                 ξ 
               
               . 
             
           
         
       
     
     For N training vectors of dimension d, where N&lt;d, all of the regularization parameters {ξ i } i=1   N  in Eq. (1.8) and all of its derivatives are set equal to zero: ξ i =ξ=0. The regularization constant C is set equal to infinity: C=∞. 
     A primal linear eigenlocus τ=τ 1 −τ 2  is the primary basis of a linear discriminant function D(x)=τ T x+τ 0 . A constrained, linear discriminant function D(x)=τ T x+τ 0 , where D(x)=0 , D(x)+1, and D(x)=1, determines a linear classification system 
     
       
         
           
             
               
                 
                   
                     τ 
                     T 
                   
                    
                   x 
                 
                 + 
                 
                   τ 
                   0 
                 
               
                
               
                 ≷ 
                 
                   ω 
                   2 
                 
               
             
             , 
           
         
       
     
     where τ=τ 1 −τ 2  is the likelihood ratio of the classification system. 
     A linear eigenlocus test 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     generates a decision boundary that divides a feature space Z into congruent decision regions Z 1  and Z 2 . The manner in which a linear eigenlocus test partitions a feature space is specified by the KKT condition in Eq. (1.15) and the KKT condition of complementary slackness, The KKT condition of complementary slackness requires that for all constraints that are not active in Eq. (1.15), where locus equations are ill-defined: 
         y   i ( x   i   T τ+τ 0 )−1+ξ i &gt;0
 
     because they are not satisfied as equalities, the corresponding magnitudes Ψ i  of the Wolf dual principal eigenaxis components Ψ i {right arrow over (e)} i  on Ψ must be zero: Ψ i =0. Accordingly, if an inequality is “slack” (not strict), the other inequality cannot be slack. 
     Therefore, let there be l active constraints, where l=l 1 +l 2 , and let ξ i =ξ=0 or ξ i =ξ&lt;&lt;. The theorem of Karush, Kuhn, and Tucker provides the guarantee that a Wolf dual linear eigenlocus Ψ exists such that the following constraints are satisfied: 
       {ω i* &gt;0} i=1   l  
 
     and the following locus equations are satisfied: 
       Ψ i*   [y   i ( x   i*   T τ+τ 0 )−1+ξ i ]=0,  i= 1,
 
     where l Wolfe dual principal eigenaxis components Ψ i* {right arrow over (e)} i  have non-zero magnitudes {Ψ i {right arrow over (e)} i |Ψ i*   υ} i=1   l . 
     Accordingly, let there be l 1  locus equations: 
         x   1*   T τ+τ 0 +ξ i =1,  i= 1,
 
     where y i =+1, and l 2  locus equations: 
         x   1*   T τ+τ 0 −ξ i =−1,  i= 1,
 
     where y i =−1.
 
It follows that the linear discriminant function
 
         D ( x )=τ T   x+τ   0   (1.20)
 
     satisfies the set of constraints. 
         D   0 ( x )=0,  D   +1 ( x )=+1, and  D   −1 ( x )=−1
 
     where D 0 (x)=0 denotes a linear decision boundary, D +1 (x) denotes a linear decision border for the Z 1  decision region, and D —1  (x) denotes a linear decision border for the Z 2  decision region. 
     Given the assumption that D(x)=0 , the linear discriminant function in Eq. (1.20) can be rewritten as: 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           x 
                           T 
                         
                          
                         τ 
                       
                       
                          
                         τ 
                          
                       
                     
                     = 
                     
                       - 
                       
                         
                           τ 
                           0 
                         
                         
                            
                           τ 
                            
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   1.21 
                   ) 
                 
               
             
           
         
       
     
     where 
     
       
         
           
             
                
               
                 τ 
                 0 
               
                
             
             
                
               τ 
                
             
           
         
       
     
     is the distance of a linear decision boundary to the origin. Any point x that satisfies Eq. (1.21) is on the linear decision boundary D 0 (x), and all of the points x on the linear decision boundary D 0 (x) exclusively reference τ. Thereby, the constrained, linear discriminant function τ T x+τ 0  satisfies the boundary value of a linear decision boundary D 0 (x):τ T   x+τ   0 =0. 
     Given the assumption that D(x)=1, the linear discriminant function in Eq. (1.20) can be rewritten as: 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           x 
                           T 
                         
                          
                         τ 
                       
                       
                          
                         τ 
                          
                       
                     
                     = 
                     
                       
                         
                           τ 
                           0 
                         
                         
                            
                           τ 
                            
                         
                       
                       + 
                       
                         1 
                         
                            
                           τ 
                            
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   1.22 
                   ) 
                 
               
             
           
         
       
     
     where 
     
       
         
           
             
                
               
                 1 
                 - 
                 
                   τ 
                   0 
                 
               
                
             
             
                
               τ 
                
             
           
         
       
     
     is the distance of the linear decision border to the origin. Any point x that satisfies Eq. (1.22) is on the linear decision border D +1 (x), and all of the points x on the linear decision border D +1 (x)) exclusively reference τ. Thereby, the constrained, linear discriminant function τ T   x+τ   0  satisfies the boundary value of a linear decision border D +1 (x): τ T   x+τ   0 =1. 
     Given the assumption that D(x)=−1 the linear discriminant function in Eq. (1.20) can be rewritten as: 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           x 
                           T 
                         
                          
                         τ 
                       
                       
                          
                         τ 
                          
                       
                     
                     = 
                     
                       
                         
                           τ 
                           0 
                         
                         
                            
                           τ 
                            
                         
                       
                       + 
                       
                         1 
                         
                            
                           τ 
                            
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   1.23 
                   ) 
                 
               
             
           
         
       
     
     where 
     
       
         
           
             
                
               
                 
                   - 
                   1 
                 
                 - 
                 
                   τ 
                   0 
                 
               
                
             
             
                
               τ 
                
             
           
         
       
     
     is the distance of the linear decision border to the origin. Any point x that satisfies Eq. (1.23) is on the linear decision border D —1 (x), and all of the points x on the linear decision border D −1 (x) exclusively reference τ. Thereby, the constrained, linear discriminant function τ T   x+τ   0  satisfies the boundary value of a linear decision border D −1 (x): τ T   x+τ   0 −1. 
     The linear decision borders D +1 (x) and D −1 (x) in Eqs (1.22) and (1.23) satisfy the symmetrically balanced constraints 
     
       
         
           
             
               - 
               
                 
                   τ 
                   0 
                 
                 
                    
                   τ 
                    
                 
               
             
             + 
             
               
                 1 
                 
                    
                   τ 
                    
                 
               
                
               
                   
               
                
               and 
             
              
             
                 
             
             - 
             
               
                 τ 
                 0 
               
               
                  
                 τ 
                  
               
             
             - 
             
               1 
               
                  
                 τ 
                  
               
             
           
         
       
     
     with respect to the constraint 
     
       
         
           
             - 
             
               
                 τ 
                 0 
               
               
                  
                 τ 
                  
               
             
           
         
       
     
     satisfied by the linear decision boundary D 0 (x) so that a constrained, linear discriminant function delineates congruent decision regions 
     
       
         
           
             
               Z 
               1 
             
             ≃ 
           
         
       
     
     that are symmetrically partitioned by the linear decision boundary in Eq. (1.21). 
     Thereby, τ is an eigenaxis of symmetry which delineates congruent decision regions 
     
       
         
           
             
               Z 
               1 
             
             ≃ 
           
         
       
     
     that are symmetrically partitioned by a linear decision boundary, where the span of both decision regions is regulated by the constraints in Eqs (1.21), (1.22), and (1.23).  FIG. 4  illustrates congruent decision regions 
     
       
         
           
             
               Z 
               1 
             
             ≃ 
           
         
       
     
     that are symmetrically partitioned by a linear decision boundary for homogeneous data distributions, where τ is an eigenaxis of symmetry.  FIG. 5  illustrates congruent decision regions 
     
       
         
           
             
               Z 
               1 
             
             ≃ 
           
         
       
     
     that are symmetrically partitioned by a linear decision boundary for which τ is an eigenaxis of symmetry. 
     Using the KKT condition in Eq. (1.15) and the KKT condition of complementary slackness, the following set of locus equations must be satisfied: 
         y   i ( x   i*   T τ+τ 0 )−+ξ i =0,  i= 1,
 
     such that τ 0  satisfies the locus equation: 
       τ 0 =Σ i=1   l   y   i (1−ξ i )−Σ i=1   l   x   i*   T τ,  (1.24)
 
     Substitution of the equation for τ in Eq. (1.15) and the statistic for τ 0  in Eq. (1.24) into the expression for the linear discriminant function in Eq. (1.20) provides the linear eigenlocus test for classifying an unknown pattern vector x: 
     
       
         
           
             
               
                 
                   
                     
                       Λ 
                       τ 
                     
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         
                           ( 
                           
                             x 
                             - 
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 l 
                               
                                
                               
                                   
                               
                                
                               
                                 x 
                                 
                                   i 
                                   * 
                                 
                               
                             
                           
                           ) 
                         
                          
                         
                           τ 
                           1 
                         
                       
                       - 
                       
                         
                           ( 
                           
                             x 
                             - 
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 l 
                               
                                
                               
                                   
                               
                                
                               
                                 x 
                                 
                                   i 
                                   * 
                                 
                               
                             
                           
                           ) 
                         
                          
                         
                           τ 
                           2 
                         
                       
                       + 
                       
                         
                           ∑ 
                           
                             i 
                             = 
                             1 
                           
                           l 
                         
                          
                         
                             
                         
                          
                         
                           
                             y 
                             i 
                           
                            
                           
                             ( 
                             
                               1 
                               - 
                               
                                 ξ 
                                 i 
                               
                             
                             ) 
                           
                         
                       
                     
                      
                     
                       ≷ 
                       
                         ω 
                         2 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1.25 
                   ) 
                 
               
             
           
         
       
     
     where the statistic Σ i=1   l x i*  the locus of an aggregate or cluster of a set of I extreme points, and the statistic Σ i=1   l y i (1−ξ i ) accounts for the class membership of the primal principal eigenaxis components on τ 1  and τ 2 . The cluster Σ i=1   l x i*  of a set of extreme points represents the aggregated risk   for a decision space Z. Accordingly, the vector transform x−Σ i=1   l x i*  accounts for the distance between the unknown vector x and the locus of aggregated risk  . 
     Let there bel principal eigenaxis components {Ψ i* {right arrow over (e)} i |Ψ i*   υ} i=1   l  on Ψ within the Wolfe dual eigenspace. 
     
       
         
           
             
               
                 max 
                  
                 
                     
                 
                  
                 
                   Ξ 
                    
                   
                     ( 
                     Ψ 
                     ) 
                   
                 
               
               = 
               
                 
                   
                     1 
                     T 
                   
                    
                   Ψ 
                 
                 - 
                 
                   
                     
                       Ψ 
                       T 
                     
                      
                     Q 
                      
                     
                         
                     
                      
                     Ψ 
                   
                   2 
                 
               
             
             , 
           
         
       
     
     where w satisfies the constraints Ψ T y=0 and Ψ i ≥0. 
     The theorem for convex duality guarantees an equivalence and corresponding symmetry between τ and Ψ. Moreover, Raleigh&#39;s principle and the theorem for convex duality indicate that Eq. (1.17) provides an estimate of the largest eigenvector Ψ of a Gram matrix, where Ψ satisfies the constraints Ψ T y=0 and Ψ i ≥0, such that Ψ is a principal eigenaxis of three, symmetrical hyperplane partitioning surfaces associated with the constrained quadratic Ψ T QΨ. 
     Equation (1.9) and the theorem for convex duality also indicate that Ψ satisfies an eigenenergy constraint that is symmetrically related to the eigenenergy constraint on τ within its Wolfe dual eigenspace: 
       ∥Ψ∥ min     c     2 ≈∥τ∥ min     c     2  
 
     Therefore, Ψ satisfies an eigenenergy constraint 
       maxΨ T   QΨ=λ   maxΨ ∥Ψ min     c     2  
 
     for which the functional  1   T Ψ−Ψ T QΨ/2 in Eq. (1.17) is maximized by the largest eigenvector Ψ of Q, such that the constrained quadratic form Ψ T QΨ/2, where Ψ T y=0 and Ψ i ≥0, reaches its smallest possible value. This indicates that principal eigenaxis components on Ψ satisfy minimum length constraints. Principal eigenaxis components on Ψ also satisfy an equilibrium constraint. 
     The KKT condition in Eq. (1.12) requires that the magnitudes of the Wolfe dual principal eigenaxis components on Ψ satisfy the equation: 
       ( y   i =1)Σ i=1   l     1   Ψ 1i* +( y   i =−1)Σ i=1   l     2   Ψ 2i* =0
 
     so that 
       Σ i=1   l     1   Ψ 1i* −Σ i=1   l     2   Ψ 2i* +0.  (1.26)
 
     It follows that the integrated lengths of the Wolfe dual principal eigenaxis components correlated with each pattern category must balance each other: 
       Σ i=1   l     1   Ψ 1i* ≡Σ i=1   l     2   Ψ 2i* .  (1.27)
 
     Accordingly, let l 1 +l 2 =l and express Ψ in terms of l non-orthogonal unit vectors 
     
       
         
           
             { 
             
               
                 
                   ψ 
                   
                     1 
                      
                     
                       i 
                       * 
                     
                   
                 
                  
                 
                   
                     e 
                     → 
                   
                   i 
                 
               
               , 
               
                 
                   … 
                    
                   
                       
                   
                 
                 
                   i 
                   = 
                   1 
                 
               
             
              
             
                 
             
           
         
       
     
     
       
         
           
             
               
                 
                   
                     
                       
                         Ψ 
                         = 
                           
                          
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             l 
                           
                            
                           
                               
                           
                            
                           
                             
                               ψ 
                               
                                 1 
                                  
                                 
                                     
                                 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                              
                             
                               
                                 e 
                                 → 
                               
                               
                                 i 
                                 * 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                          
                         
                           
                             
                               ∑ 
                               
                                 i 
                                 = 
                                 1 
                               
                               
                                 l 
                                 1 
                               
                             
                              
                             
                                 
                             
                              
                             
                               
                                 ψ 
                                 
                                   1 
                                    
                                   
                                       
                                   
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                                
                               
                                 
                                   e 
                                   → 
                                 
                                 
                                   1 
                                    
                                   
                                       
                                   
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                             
                           
                           + 
                           
                             
                               ∑ 
                               
                                 i 
                                 = 
                                 1 
                               
                               l 
                             
                              
                             
                                 
                             
                              
                             
                               
                                 ψ 
                                 
                                   2 
                                    
                                   
                                       
                                   
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                                
                               
                                 
                                   e 
                                   → 
                                 
                                 
                                   2 
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                           
                          
                         
                           
                             
                               Ψ 
                               1 
                             
                             + 
                             
                               Ψ 
                               2 
                             
                           
                           , 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1.28 
                   ) 
                 
               
             
           
         
       
     
     where each scaled, non-orthogonal unit vector 
     
       
         
           
             
               ψ 
               
                 1 
                  
                 
                   i 
                   * 
                 
               
             
              
             
               
                 e 
                 → 
               
               
                 1 
                  
                 
                   i 
                   * 
                 
               
             
              
             
                 
             
              
             or 
              
             
               
                   
               
                
               
                   
               
             
              
             
               ψ 
               
                 2 
                  
                 
                   i 
                   * 
                 
               
             
              
             
               
                 e 
                 → 
               
               
                 2 
                  
                 
                   i 
                   * 
                 
               
             
           
         
       
     
     is correlated with an extreme vector x 1i*  or x 2i*  respectively, Ψ 1  denotes the Wolfe dual eigenlocus component 
     
       
         
           
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 
                   l 
                   1 
                 
               
                
               
                 
                   ψ 
                   
                     1 
                      
                     
                       i 
                       * 
                     
                   
                 
                  
                 
                   
                     e 
                     → 
                   
                   
                     1 
                      
                     
                       i 
                       * 
                     
                   
                 
               
             
             , 
           
         
       
     
     and ω 2  denotes the Wolfe dual eigenlocus component 
     
       
         
           
             
               ∑ 
               
                 i 
                 = 
                 1 
               
               
                 l 
                 2 
               
             
              
             
               
                 ψ 
                 
                   2 
                    
                   
                     i 
                     * 
                   
                 
               
                
               
                 
                   
                     e 
                     → 
                   
                   
                     2 
                      
                     
                       i 
                       * 
                     
                   
                 
                 . 
               
             
           
         
       
     
     Given Eq. (1.27) and data distributions that have dissimilar covariance matrices, it follows that the forces associated with Bayes&#39; counter risks and Bayes&#39; risks, within each of the congruent decision regions, are balanced with each other. Given Eq. (1.27) and data distributions that have similar covariance matrices, it follows that the forces associated with Bayes&#39; counter risks within each of the symmetrical decision regions are equal to each other, and the forces associated with Bayes&#39; risks within each of the symmetrical decision regions are equal to each other. 
     Given Eqs (1.27) and (1.28), the axis of Ψ can be regarded as a lever that is formed by sets of principal eigenaxis components which are evenly or equally distributed over either side of the axis of Ψ, where a fulcrum is placed directly under the center of the axis of Ψ. Thereby, the axis of Ψ is in statistical equilibrium, where all of the principal eigenaxis components on Ψ are equal or in correct proportions, relative to the center of Ψ, such that the opposing forces associated with Bayes&#39; risks and Bayes&#39; counter risks of a linear classification system are balanced with each other. 
     Using Eq. (1.27), it follows that the length ∥Ψ 1 ∥ of Ψ 1  is balanced with the length ∥Ψ 2 ∥ of Ψ 2 : 
       ∥Ψ 1 ∥≡∥Ψ 2 ∥  (1.29)
 
     and that the total allowed eigenenergies exhibited by Ψ 1  and Ψ 2  are balanced with each other: 
       ∥Ψ 1 ∥ min     c     2 ≡∥Ψ 2 ∥ min     c     2 .  (1.30)
 
     Therefore, the equilibrium constraint on Ψ in Eq. (1.27) ensures that the total allowed eigenenergies exhibited by the Wolfe dual principal eigenaxis components on Ψ 1  and Ψ 2  are symmetrically balanced with each other: 
     
       
         
           
             
               
                  
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     
                       l 
                       1 
                     
                   
                    
                   
                     
                       ψ 
                       
                         1 
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                     
                       
                         e 
                         → 
                       
                       
                         1 
                          
                         
                           i 
                           * 
                         
                       
                     
                   
                 
                  
               
               
                 min 
                 c 
               
             
             = 
             
               
                  
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     
                       l 
                       2 
                     
                   
                    
                   
                     
                       ψ 
                       
                         2 
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                     
                       
                         e 
                         → 
                       
                       
                         2 
                          
                         
                           i 
                           * 
                         
                       
                     
                   
                 
                  
               
               
                 min 
                 c 
               
             
           
         
       
     
     about the center of total allowed eigenenergy Ψ 2 ∥ min     c     2 : which is located at the geometric center of Ψ because ∥Ψ 1 ∥≡∥Ψ 2 ∥. This indicates that the total allowed eigenenergies of Ψ are distributed over its axis in a symmetrically balanced and well-proportioned manner. 
     Given Eqs (1.29) and (1.30), the axis of Ψ can be regarded as a lever that has equal weight on equal sides of a centrally placed fulcrum. Thereby, the axis of is a lever that has an equal distribution of eigenenergies on equal sides of a centrally placed fulcrum. 
     The eigenspectrum of the Gram matrix Q determines the shapes of the quadratic surfaces associated with Ψ that are specified by the constrained quadratic form in Eq. (1.17). Furthermore, the eigenvalues of any given Gram or kernel matrix Q are essentially determined by its inner product elements φ(x i , x j ), so that the geometric shapes of the three, symmetrical hyperplane partitioning surfaces determined by Eqs (1.16) and (1.17) are an inherent function of inner product statistics. Thereby, the form of the inner product statistics φ(x i , x j ) contained within the Gram matrix Q essentially determines the geometric shapes of the linear decision boundary D 0 (s)=0 in Eq. (1.21) and the linear decision borders D +1 (s)=+1 and D −1 (s)=−1 in Eqs (1.22) and (1.23). 
     The inner product relationship x γ y=∥x∥∥y∥cosφ between two vectors x and y can be derived by using the law of cosines: 
       ∥ x−y∥   2   =∥x∥   2   +∥y∥   2 −2∥ x∥∥y ∥ cos φ
 
     which reduces to 
       ∥ x∥∥y ∥ cos φ= x   1   y   1   +x   2   y   2 +
 
     so that 
         x   T   y=∥x∥∥y ∥ cos φ= x−y∥.  
 
     Thereby, inner product statistics determine a rich system of geometric and topological relationships between vectors. 
     A Wolfe dual quadratic eigenlocus Ψ can be written as: 
     
       
         
           
             
               
                 
                   
                     Ψ 
                     = 
                     
                       
                         
                           
                             ψ 
                             1 
                           
                           
                             λ 
                             
                               max 
                                
                               
                                   
                               
                                
                               Ψ 
                             
                           
                         
                          
                         
                           ( 
                           
                             
                               
                                 
                                   
                                      
                                     
                                       x 
                                       1 
                                     
                                      
                                   
                                    
                                   
                                      
                                     
                                       x 
                                       1 
                                     
                                      
                                   
                                    
                                   cos 
                                    
                                   
                                       
                                   
                                    
                                   
                                     θ 
                                     
                                       
                                         x 
                                         1 
                                       
                                        
                                       
                                         x 
                                         1 
                                       
                                     
                                   
                                 
                               
                             
                             
                               
                                 
                                   
                                      
                                     
                                       x 
                                       2 
                                     
                                      
                                   
                                    
                                   
                                      
                                     
                                       x 
                                       2 
                                     
                                      
                                   
                                    
                                   cos 
                                    
                                   
                                       
                                   
                                    
                                   
                                     θ 
                                     
                                       
                                         x 
                                         2 
                                       
                                        
                                       
                                         x 
                                         1 
                                       
                                     
                                   
                                 
                               
                             
                             
                               
                                 ⋮ 
                               
                             
                             
                               
                                 
                                   
                                     - 
                                     
                                        
                                       
                                         x 
                                         N 
                                       
                                        
                                     
                                   
                                    
                                   
                                      
                                     
                                       x 
                                       1 
                                     
                                      
                                   
                                    
                                   cos 
                                    
                                   
                                       
                                   
                                    
                                   
                                     θ 
                                     
                                       
                                         x 
                                         N 
                                       
                                        
                                       
                                         x 
                                         1 
                                       
                                     
                                   
                                 
                               
                             
                           
                           ) 
                         
                       
                       + 
                       
                         … 
                          
                         
                             
                         
                          
                         
                           
                             λ 
                             
                               max 
                                
                               
                                   
                               
                                
                               Ψ 
                             
                           
                            
                           
                             ( 
                             
                               
                                 
                                   
                                     
                                       - 
                                       
                                          
                                         
                                           x 
                                           1 
                                         
                                          
                                       
                                     
                                      
                                     
                                        
                                       
                                         x 
                                         N 
                                       
                                        
                                     
                                      
                                     cos 
                                      
                                     
                                         
                                     
                                      
                                     
                                       θ 
                                       
                                         
                                           x 
                                           1 
                                         
                                          
                                         
                                           x 
                                           N 
                                         
                                       
                                     
                                   
                                 
                               
                               
                                 
                                   
                                     
                                       - 
                                       
                                          
                                         
                                           x 
                                           2 
                                         
                                          
                                       
                                     
                                      
                                     
                                        
                                       
                                         x 
                                         N 
                                       
                                        
                                     
                                      
                                     cos 
                                      
                                     
                                         
                                     
                                      
                                     
                                       θ 
                                       
                                         
                                           x 
                                           2 
                                         
                                          
                                         
                                           x 
                                           N 
                                         
                                       
                                     
                                   
                                 
                               
                               
                                 
                                   ⋮ 
                                 
                               
                               
                                 
                                   
                                     
                                        
                                       
                                         x 
                                         N 
                                       
                                        
                                     
                                      
                                     
                                        
                                       
                                         x 
                                         N 
                                       
                                        
                                     
                                      
                                     cos 
                                      
                                     
                                         
                                     
                                      
                                     
                                       θ 
                                       
                                         
                                           x 
                                           N 
                                         
                                          
                                         
                                           x 
                                           N 
                                         
                                       
                                     
                                   
                                 
                               
                             
                             ) 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   1.31 
                   ) 
                 
               
             
           
         
       
     
     which illustrates that Ψ 1  is correlated with scalar projections ∥x 1 ∥cosθ x     i     x     j    of the vector x j  onto labeled vectors x i . Further, it has been demonstrated that Ψ j  is correlated with a first and second-order statistical moment about the locus of x j , where a first and second-order statistical moment involves a pointwise covariance statistic cov up (x i ): 
       cov up ( x   i )=∥ x   i ∥Σ j=1   N   ∥x   j ∥ cos θ x     i     x     j    
 
     that provides a unidirectional estimate of the joint variations between the random variables of each training vector x j  in a training data collection and the random variables of a fixed vector x i  and a unidirectional estimate of the joint variations between the random variables of the mean vector Σ j=1   N x j  and the fixed vector x i , along the axis of the fixed vector x i . 
     Each extreme vector x 1i*  and x 2i*  exhibits a critical first and second-order statistical moment cov up (x 1i* ) and cov up (x 2i* ) that exceeds some threshold ϑ, for which each corresponding scale factor Ψ 1i*  and Ψ 2i*  exhibits a critical value that exceeds zero: Ψ 1i* &gt;0 and Ψ 2i* &gt;0. 
     Let the extreme vectors x 1i*  and x 2i*  that belong to class ω 1  and ω 2  have labels y i =1 and y i =−1 respectively. Let there be l 1  extreme vectors from class ω 1  and l 2  extreme vectors from class ω 2 . 
     Let i=1:l 1 , where each extreme vector x 1i*  is correlated with a Wolfe principal eigenaxis component 
     
       
         
           
             
               ψ 
               
                 1 
                  
                 
                   i 
                   * 
                 
               
             
              
             
               
                 
                   e 
                   → 
                 
                 
                   1 
                    
                   
                     i 
                     * 
                   
                 
               
               . 
             
           
         
       
     
     The Wolfe dual eigensystem in Eq. (1.31) can be used to show that the locus of 
     
       
         
           
             
               ψ 
               
                 1 
                  
                 
                   i 
                   * 
                 
               
             
              
             
               
                 e 
                 → 
               
               
                 1 
                  
                 
                   i 
                   * 
                 
               
             
           
         
       
     
     is a function of the expression: 
       Ψ 1i* =λ maxΨ   −1   ∥x   1i* ∥Σ j=1   l     1   Ψ 1j*   ∥x   1j* ∥cos θ x     1i*     x     1j*   −λ maxΨ   −1   ∥x   1i* ∥Σ j=1   l     2   Ψ 2j*   ∥x   2j* ∥cos θ x     1i*     x     2j*     (1.32)
 
     where Ψ 1i*  provides a scale factor for 
     
       
         
           
             
               
                 x 
                 
                   1 
                    
                   
                       
                   
                    
                   
                     i 
                     * 
                   
                 
               
               
                  
                 
                   x 
                   
                     1 
                      
                     
                         
                     
                      
                     
                       i 
                       * 
                     
                   
                 
                  
               
             
             . 
           
         
       
     
     Let i=1:l 2 , where each extreme vector x 2i*  is correlated with a Wolfe principal eigenaxis component 
     
       
         
           
             
               ψ 
               
                 2 
                  
                 
                   i 
                   * 
                 
               
             
              
             
               
                 
                   e 
                   → 
                 
                 
                   2 
                    
                   
                     i 
                     * 
                   
                 
               
               . 
             
           
         
       
     
     The Wolfe dual eigensystem in Eq. (1.31) can be used to show that the locus of 
     
       
         
           
             
               ψ 
               
                 2 
                  
                 
                   i 
                   * 
                 
               
             
              
             
               
                 e 
                 → 
               
               
                 2 
                  
                 
                   i 
                   * 
                 
               
             
           
         
       
     
     is a function of the expression. 
       Ψ 2i* =λ maxΨ   −1   ∥x   2i* ∥Σ j=1   l     1   Ψ 2j*   ∥x   2j* ∥cos θ x     2i*     x     2j*   −λ maxΨ   −1   ∥x   2i* ∥Σ j=1   l     1   Ψ 1j*   ∥x   1j* ∥cos θ x     1i*     x     1j*     (1.32)
 
     where Ψ 2i*  provides a scale factor for 
     
       
         
           
             
               
                 x 
                 
                   2 
                    
                   
                       
                   
                    
                   
                     i 
                     * 
                   
                 
               
               
                  
                 
                   x 
                   
                     2 
                      
                     
                         
                     
                      
                     
                       i 
                       * 
                     
                   
                 
                  
               
             
             . 
           
         
       
     
     Equations (1.32) and (1.33) have been used to demonstrate that any given Wolfe dual principal eigenaxis component Ψ 1i* {right arrow over (e)} 1i*  correlated an x 1i*  extreme point and any given Wolfe dual principal eigenaxis component Ψ 2i* {right arrow over (e)} 2i*  correlated with an X 2i*  extreme point provides an estimate for how the components of I scaled extreme vectors {Ψ j* x j* } j−   l  are symmetrically distributed along the axis of a correlated extreme vector x 1i*  or X 2i* , where components of scaled extreme vectors Ψ j* x j*  are symmetrically distributed according to class labels ±1, signed magnitudes ∥x j* ∥cosθ x     1i*     x     j*    or ∥x j* ∥cosθ x     2i*     x     j*    and symmetrically balanced distributions of scaled extreme vectors {Ψ k* x k* } k−   l  specified by scale factors Ψ j* . 
     Accordingly, Ψ is formed by a locus of scaled, normalized extreme vectors 
     
       
         
           
             Ψ 
             = 
             
               
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     
                       l 
                       1 
                     
                   
                    
                   
                     
                       ψ 
                       
                         1 
                          
                         
                             
                         
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                     
                       
                         x 
                         
                           1 
                            
                           
                               
                           
                            
                           
                             i 
                             * 
                           
                         
                       
                       
                          
                         
                           x 
                           
                             1 
                              
                             
                                 
                             
                              
                             
                               i 
                               * 
                             
                           
                         
                          
                       
                     
                   
                 
                 + 
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     
                       l 
                       2 
                     
                   
                    
                   
                     
                       ψ 
                       
                         2 
                          
                         
                             
                         
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                     
                       
                         x 
                         
                           2 
                            
                           
                               
                           
                            
                           
                             i 
                             * 
                           
                         
                       
                       
                          
                         
                           x 
                           
                             2 
                              
                             
                                 
                             
                              
                             
                               i 
                               * 
                             
                           
                         
                          
                       
                     
                   
                 
               
               = 
               
                 
                   Ψ 
                   1 
                 
                 + 
                 
                   Ψ 
                   2 
                 
               
             
           
         
       
     
     where each scale factor Ψ 1i*  or Ψ 2i*  provides a unit measure, i.e., estimate, of density and likelihood for a respective extreme point x 1i*  or x 2i* . 
     Therefore, conditional densities Ψ 1i* x 1i*  for the x 1i*  extreme points are distributed over the principal eigenaxis components of τ 1    
       τ 1 =Σ i=1   l     1   Ψ 1i*   x   1i*   (1.34)
 
     so that τ 1  is a parameter vector for a class-conditional probability density p(x 1i* |τ 1 ) for a given set {x 1i* } i=1   l     1    of x 1i*  extreme points: 
       τ 1   =p ( x   1i* |τ 1 ),
 
     where the area under Ψ 1i* x 1i*  is a conditional probability that an extreme point x 1i*  will be observed in either region Z 1  or region Z 2 . 
     Likewise, conditional densities Ψ 2i* x 2i*  for the x 2i*  extreme points are distributed over the principal eigenaxis components of τ 2    
       τ 2 =Σ i=1   l     2   Ψ 2i*   x   2i*   (1.35)
 
     so that τ 2  is a parameter vector for a class-conditional probability density p(x 2i* |τ 2 ) for a given set {x 2i* } i=1   l     2    of x 2i*  extreme points: 
       τ 2   =p ( x   2i* |τ 2 ),
 
     where the area under Ψ 2i* x 2i*  is a conditional probability that an extreme point x 2i*  will be observed in either region Z 1  or region Z 2 . 
     The area P(x 1i* |τ 1 ) under the class-conditional density function p(x 1i* |τ 1 ) in Eq. (1.34) 
     
       
         
           
             
               
                 
                   
                     P 
                      
                     
                       ( 
                       
                         
                           x 
                           
                             1 
                              
                             
                                 
                             
                              
                             
                               i 
                               * 
                             
                           
                         
                         | 
                         
                           τ 
                           1 
                         
                       
                       ) 
                     
                   
                   = 
                     
                    
                   
                     
                       
                         ∫ 
                         Z 
                       
                        
                       
                         
                           ( 
                           
                               
                           
                            
                           
                             
                               ∑ 
                               
                                 i 
                                 = 
                                 1 
                               
                               
                                 l 
                                 1 
                               
                             
                              
                             
                               
                                 ψ 
                                 
                                   1 
                                    
                                   
                                       
                                   
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                                
                               
                                 x 
                                 
                                   1 
                                    
                                   
                                       
                                   
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                             
                           
                           ) 
                         
                          
                         d 
                          
                         
                             
                         
                          
                         
                           τ 
                           1 
                         
                       
                     
                     = 
                     
                       
                         ∫ 
                         Z 
                       
                        
                       
                           
                       
                        
                       
                         
                           p 
                           ( 
                           
                               
                           
                            
                           
                             
                               x 
                               
                                 1 
                                  
                                 
                                     
                                 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                             | 
                             
                               τ 
                               1 
                             
                           
                           ) 
                         
                          
                         d 
                          
                         
                             
                         
                          
                         
                           τ 
                           1 
                         
                       
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       
                         ∫ 
                         Z 
                       
                        
                       
                           
                       
                        
                       
                         
                           τ 
                           1 
                         
                          
                         d 
                          
                         
                             
                         
                          
                         
                           τ 
                           1 
                         
                       
                     
                     = 
                     
                       
                         
                           
                             1 
                             2 
                           
                            
                           
                             
                                
                               
                                 τ 
                                 1 
                               
                                
                             
                             2 
                           
                         
                         + 
                         C 
                       
                       = 
                       
                         
                           
                              
                             
                               τ 
                               1 
                             
                              
                           
                           2 
                         
                         + 
                         
                           C 
                           1 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     specifies the conditional probability of observing a set {x 1i* } i=1   l     1    of x 1i*  extreme points: 
     extreme points within localized regions of the decision space Z, where conditional densities Ψ 1i* x 1i*  for x 1i*  extreme points that lie in the Z 2  decision region contribute to the cost or risk  (Z 2 |Ψ 1i* x 1i* ) of making a decision error, and conditional densities Ψ 1i* x 1i*  for x 1i*  extreme points that lie in the Z 1  decision region counteract the cost or risk  (Z 1 |Ψ 1i* x 1i* ) of making a decision error. 
     Therefore, the conditional probability function P(x 1i* |τ 1 ) for class Ω 1  is given by the integral 
         P ( x   1i* |τ 1 )=∫ Z τ 1   dτ   1 =∥τ 1 ∥ 2   +C   1 ,  (1.36)
 
     over the decision space Z, which has a solution in terms of the critical minimum eigenenergy ∥τ 1 ∥ min     c     2  exhibited by τ 1  and an integration constant C 1 . 
     The area P(x 2i* |τ 2 ) under the class-conditional density function p(x 2i* |τ 2 ) in Eq. (1.35) 
     
       
         
           
             
               
                 
                   
                     P 
                      
                     
                       ( 
                       
                         
                           x 
                           
                             2 
                              
                             
                                 
                             
                              
                             
                               i 
                               * 
                             
                           
                         
                         | 
                         
                           τ 
                           2 
                         
                       
                       ) 
                     
                   
                   = 
                     
                    
                   
                     
                       
                         ∫ 
                         Z 
                       
                        
                       
                         
                           ( 
                           
                               
                           
                            
                           
                             
                               ∑ 
                               
                                 i 
                                 = 
                                 1 
                               
                               
                                 l 
                                 2 
                               
                             
                              
                             
                               
                                 ψ 
                                 
                                   2 
                                    
                                   
                                       
                                   
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                                
                               
                                 x 
                                 
                                   2 
                                    
                                   
                                       
                                   
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                             
                           
                           ) 
                         
                          
                         d 
                          
                         
                             
                         
                          
                         
                           τ 
                           2 
                         
                       
                     
                     = 
                     
                       
                         ∫ 
                         Z 
                       
                        
                       
                           
                       
                        
                       
                         
                           p 
                           ( 
                           
                               
                           
                            
                           
                             
                               x 
                               
                                 2 
                                  
                                 
                                     
                                 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                             | 
                             
                               τ 
                               2 
                             
                           
                           ) 
                         
                          
                         d 
                          
                         
                             
                         
                          
                         
                           τ 
                           2 
                         
                       
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       
                         ∫ 
                         Z 
                       
                        
                       
                           
                       
                        
                       
                         
                           τ 
                           2 
                         
                          
                         d 
                          
                         
                             
                         
                          
                         
                           τ 
                           2 
                         
                       
                     
                     = 
                     
                       
                         
                           
                             1 
                             2 
                           
                            
                           
                             
                                
                               
                                 τ 
                                 2 
                               
                                
                             
                             2 
                           
                         
                         + 
                         C 
                       
                       = 
                       
                         
                           
                              
                             
                               τ 
                               2 
                             
                              
                           
                           2 
                         
                         + 
                         
                           C 
                           2 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     specifies the conditional probability of observing a set {x 2i* } i=1   l     2    of x 2i*  extreme points within localized regions of the decision space Z, where conditional densities Ψ 2i* x 2i*  Ψ 2i* k 2i*  for x 2i*  extreme points that lie in the Z 1  decision region contribute to the cost or risk  (Z 1 |Ψ 2i* x 2i* ) of making a decision error, and conditional densities Ψ 2i* x 2i*  for x 2i*  extreme points that lie in the Z 2  decision region counteract the cost or risk  (Z 2 |Ψ 2i* x 2i* ) of making a decision error. 
     Therefore, the conditional probability function P(x 2i* |τ 2 ) for class ω 2  is given by the integral 
         P ( x   2i* |τ 2 )=∫ Z τ 2   dτ   2 =∥τ 2 ∥ 2   +C   2 ,  (1.37)
 
     over the decision space Z, which has a solution in terms of the critical minimum eigenenergy ∥τ 2 ∥ min     c     2  exhibited by ρ 2  and an integration constant C 2 . 
     Linear eigenlocus transforms routinely accomplish an elegant, statistical balancing feat that involves finding the right mix of principal eigenaxis components on Ψ and τ. The scale factors {Ψ i* } i=1   l  of the principal eigenaxis components on Ψ play a fundamental role in this statistical balancing feat. 
     Using Eq. (1.32), the integrated lengths Σ i=1   l     1   Ψ 1i*  of of the principal eigenaxis components on Ψ 1  must satisfy the equation: 
       Σ i=1   l     1   Ψ 1i* =λ maxΨ   −1 Σ i=1   l     1     x   1i*   T (Σ j=1   l     1   Ψ 1j*   x   1j* −Σ j=1   l     2   Ψ 2j*   x   2j* ),  (1.38)
 
     and, using Eq. (1.33), the integrated lengths Σ i=1   l     2   Ψ 2i*  of the principal eigenaxis components on Ψ 2  must satisfy the equation: 
       Σ i=1   l     2   Ψ 2i* =λ maxΨ   −1 Σ i=1   l     2     x   2i*   T (Σ j=1   l     2   Ψ 2j*   x   2j* −Σ j=1   l     1   Ψ 1j*   x   1j* ),  (1.39)
 
     Returning to Eq. (1.27) where the axis of Ψ is in statistical equilibrium, it follows that the RHS of Eq. (1.38) must equal the RHS of Eq. (1.39): 
       Σ i=1   l     1   x 1i*   T (Σ j=1   l     1   Ψ 1j*   x   1j* −Σ j=1   l     2   Ψ 2j*   x   2j* )=Σ i=1   l     2   x 2i*   T (Σ j=1   l     2   Ψ 2j*   x   2j* −Σ j=1   l     2   Ψ 1j*   x   1j* ),  (1.40)
 
     whereby all of the x 1i*  ,and x 2i* , extreme points are distributed over the axes of τ i  and τ 2  in the symmetrically balanced manner: 
       Σ i=1   l     1   x 1i*   T (τ 1 −τ 2 )=Σ i=1   l     2   x 2i*   T (τ 2 −τ 1 ),  (1.41)
 
     where the components of the x 1i*  extreme vectors along the axis of τ 2  oppose the components of the x 1i*  extreme vectors along the axis of τ 1 , and the components of the x 2i*  extreme vectors along the axis of τ t  oppose the components of the x 2i*  extreme vectors along the axis of τ 2 . 
     Rewrite Eq. (1.41) as: 
       Σ i=1   l     1     x   1i*   T τ 1 +Σ i=1   l     2     x   2i*   T τ 1 =Σ i=1   l     1     x   1i*   T τ 2 +Σ i=1   l     2     x   2i*   T τ 2   (1.42)
 
     where the components of the x 1i*  and x 2i*  extreme vectors along the axes of τ 1  and ‘r 2  have forces associated with Bayes’ risks and Bayes&#39; counter risks that are functions of symmetrically balanced expected values and spreads of x 1i*  and x 2i*  extreme points located in the Z 1  and Z 2  or decision regions. Therefore, for any given collection of extreme points drawn from any given statistical distribution, all of the aggregate forces associated with Bayes&#39; risks and Bayes&#39; counter risks on the axis of τ 1  are balanced with all of the aggregate forces associated with Bayes&#39; risks and Bayes&#39; counter risks on the axis of τ 2 . 
     So, let 
     
       
         
           
             
               
                 x 
                 ^ 
               
               
                 i 
                 * 
               
             
              
             
               = 
               Δ 
             
             . 
           
         
       
     
     Using Eq. (1.42), it follows that the component of {circumflex over (x)} i*  along τ 1  is symmetrically balanced with the component of {circumflex over (x)} i*  along τ 2    
     
       
         
           
             
               
                 
                   
                     
                       comp 
                       
                         
                           τ 
                           1 
                         
                         → 
                       
                     
                     
                       ‵ 
                        
                       
                           
                       
                       · 
                       
                           
                       
                        
                       ′ 
                     
                     
                       ′ 
                       → 
                     
                   
                 
                 
                   
                     
                         
                     
                     
                       - 
                       
                         
                           τ 
                           2 
                         
                         → 
                       
                     
                   
                 
               
             
             
               ‵ 
                
               
                   
               
                
               
                 · 
                 
                     
                 
                  
                 ′ 
               
             
             
               ′ 
               → 
             
           
         
       
     
     so that the components 
     
       
         
           
             
               
                 comp 
                 
                   
                     τ 
                     1 
                   
                   → 
                 
               
               
                 
                   ‵ 
                    
                   
                       
                   
                   · 
                 
                  
                 
                     
                 
                 * 
               
               
                 ′ 
                 → 
               
             
             , 
             
                 
             
              
             
               and 
                
               
                   
               
                
               
                 
                   comp 
                   
                     
                       τ 
                       2 
                     
                     → 
                   
                 
                 
                   
                     ‵ 
                      
                     
                         
                     
                     · 
                   
                    
                   
                       
                   
                   * 
                 
                 
                   ′ 
                   → 
                 
               
             
             , 
           
         
       
     
     of clusters or aggregates of the extreme vectors from both pattern classes have equal forces associated with Bayes&#39; risks and Bayes&#39; counter risks on opposite sides of the axis of τ. 
     Given Eq. (1.42), the axis of τ can be regarded as a lever of uniform density, where the center of τ is ∥τ∥ min   2 , for which two equal weights 
     
       
         
           
             
                 
             
              
             
               comp 
                
               
                 -&gt; 
                 
                   τ 
                   1 
                 
               
                
               
                 
                   
                     ? 
                   
                    
                   
                       
                   
                    
                   and 
                    
                   
                       
                   
                    
                   comp 
                 
                  
                 
                   -&gt; 
                   
                     τ 
                     2 
                   
                 
                  
                 
                   ? 
                 
               
             
           
         
       
       
         
           
             
               ? 
             
              
             
               indicates text missing or illegible when filed 
             
           
         
       
     
     are placed on opposite sides of the fulcrum of τ, whereby the axis of τ is in statistical equilibrium. 
     Equation (1.40) indicates that the lengths {Ψ 1i* |Ψ 1i* &gt;0} i=1   l     1    and {Ψ 2i* |Ψ 2i* &gt;0} i=1   l     2    of the l Wolfe dual principal eigenaxis components on Ψ satisfy critical magnitude constraints, such that the Wolfe dual eigensystem in Eq. (1.17) determines well-proportioned lengths Ψ 1i*  or Ψ 2i*  for each Wolfe dual principal eigenaxis component on Ψ 1  or Ψ 2 , where each scale factor Ψ 1i*  or Ψ 2i*  determines a well-proportioned length for a correlated, constrained primal principal eigenaxis component or Ψ 1i* ,x 1i*  or Ψ 2i* ,x 2i*  on τ 1  or τ 2 . 
     Moreover, linear eigenlocus transforms generate scale factors for the Wolfe dual principal eigenaxis components on Ψ, which is constrained to satisfy the equation of statistical equilibrium in Eq. (1.27), such that the likelihood ratio Λ τ (x)=τ 1 −τ 2  and the classification system 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     are in statistical equilibrium, and the Bayes&#39; risk  (Z|Λ τ (x)) and the corresponding total allowed eigenenergies ∥τ 1 −τ 2 ∥ min     x     2  exhibited by the classification system 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     are minimized. 
     A system of data-driven, locus equations that determines the manner n which the total allowed eigenenergies of the scaled extreme points on τ 1 −τ 2  are symmetrically balanced about the fulcrum ∥τ∥ min     c      2  of τ is presented next. 
     Let there be l labeled, scaled extreme points on τ. Given the theorem of Karush, Kuhn, and Tucker and the KKT condition in Eq. (1.15), it follows that a Wolf dual quadratic Ψexists for which: 
       {Ψ 1i* &gt;0} i=1   l  
 
     such that the l constrained, primal principal eigenaxis components {Ψ 1i* x 1i* } i=1   l  on τ satisfy a system of l eigenlocus equations: 
       Ψ i*   [y   i ( x   i*   T τ+τ 0 )−1+ξ i ]=0,  i= 1  (1.43)
 
     Take any scaled extreme vector Ψ 1i* x 1i*  that belongs to class ω 1 . Using Eq. (1,43) and letting y i ++1, it can be shown that the total allowed eigenenergy ∥τ 1 ∥ min     c     2  exhibited by τ 1  is determined by the identity 
       ∥τ 1 ∥ min     c     2 −∥τ 1 ∥∥τ 2 ∥ cos θ τ     1     τ     2   ≡Σ i=1   l     1   Ψ 1i* (1−ξ i −τ 0 )  (1.44)
 
     so that a constrained, linear discriminant function τ T x+τ 0  satisfies the linear decision border D −1 (x): τ T x+τ 0 =1 in terms of the total allowed eigenenergy ∥τ 1 ∥ min     c     2  exhibited by τ 1 , where the functional ∥τ 1 ∥ min     c     2 −∥τ 1 ∥∥τ 2 ∥cosθ τ     1     τ     2    is constrained by the functional Σ i=1   l     1   Ψ 1i* (1−ξ i −τ 0 ). 
     Take any scaled extreme vector Ψ 2i* ,x 2i*  that belongs to class ω 2 . Using Eq. (1.43) and letting y i =−1, it can be shown that the total allowed eigenenergy ∥τ 1 ∥ min     c     2  exhibited by τ 2  is determined by the identity 
       ∥τ 1 ∥ min     c     2 − 81  τ 1 ∥ min     c     2 −∥τ 1 ∥∥τ 2 ∥ cos θ τ     2     τ     1   ≡Σ i=1   l     1   Ψ 2i* (1−ξ i −τ 0 )  (1.45)
 
     so that a constrained, linear discriminant function τ T x+τ 0  satisfies the linear decision border D − (x): τ T x+τ 0 =−1 in terms of the total allowed eigenenergy ∥τ 1 ∥ min     c     2  exhibited by τ 2 , where the functional ∥τ 2 ∥ min     c     2 −∥τ 2 ∥∥τ 1 ∥cosθ τ     2     τ     1    is constrained by the functional Σ i=1   l     2   Ψ 2i* (1−ξ i −τ 0 ). 
     Summation over the complete system of eigenlocus equations satisfied by τ 1    
       (Σ i=1   l     1   Ψ 1 i* x   1i*   T )τ=Σ i=1   l     1   Ψ 1i* (1−ξ i −τ 0 )
 
     and by τ 2    
       (−Σ i=1   l     2   Ψ 2 i* x   2i*   T )τ=Σ i=1   l     2   Ψ 2i* (1−ξ i −τ 0 )
 
     produces the following identity that is satisfied by the total allowed eigenenergy ∥τ 1 ∥ min     c     2  of τ 
       (τ 1 −τ 2 )τ≡Σ i=1   l     1   Ψ 1i* (1−ξ i −τ 0 )+Σ i=1   l     2   Ψ 2i* (1−ξ i −τ 0 )≡Σ i=1   l     1   Ψ 1i* (1−ξ i ),  (1.46)
 
     where the equilibrium constraint on Ψ in Eq.(1.27) has been used. 
     Thus, the total allowed eigenenergy ∥τ 1 ∥ min     c     2  exhibited by τ is specified by the integrated magnitudes Ψ 1i*  of the Wolfe dual principal eigenaxis components on Ψ 
       ∥τ∥ min     c     2 ≡Σ i=1   l Ψ 1i* (1−ξ i )≡Σ i=1   l Ψ 1i* −Σ i=1   l Ψ 1i* ξ i ,  (1.47)
 
     where the regularization parameters ξ i =ξ&lt;&lt; are seen to determine negligible constraints ∥τ∥ min     c     2 , so that a constrained, linear discriminant function τ T x+τ 0  satisfies the boundary value of a linear decision boundary D 0 (x): τ T x+τ 0  in terms of its total allowed eigenenergy ∥τ∥ min     c     2 , where the functional ∥τ∥ min     c     2  is constrained by the functional Σ i=1   l Ψ 2i* (1−ξ i ). 
     Using Eqs (1.44), (1.45), and (1.46), it follows that the symmetrically balanced constraints 
         E   Ψ     1   =Σ i=1   l     1   Ψ 1i* (1−ξ i −τ 0 ) and  E   Ψ     2   =Σ i=1   l     2   Ψ 2i* (1−ξ i −τ 0 )
 
     satisfied by a linear discriminant function on the respective linear decision borders D +1 (x) and D −1 (x), and the corresponding constraint 
         E   Ψ =Σ i=1   l     1   Ψ 1i* (1−ξ i −τ 0 )+Σ i=1   l     2   Ψ 2i* (1−ξ i −τ 0 )
 
     satisfied by a linear discriminant function on the linear decision boundary D 0 (x), ensure that the total allowed eigenenergies ∥τ 1 −τ 2 ∥ min     c     2  exhibited by the scaled extreme points on τ 1 τ 2  satisfy the law of cosines in the symmetrically balanced manner: 
       ∥τ∥ min     c     2 =[∥τ 1 ∥ min     c     2 −∥τ 1 ∥∥τ 2 ∥ cos θ τ     1     τ     2   ]+[∥τ 2 ∥ min     c     2 −∥τ 2 ∥∥τ 1 ∥ cos θ τ     2     τ     1   ].
 
     Furthermore, it has been shown that ∥τ 1 ∥ min     c     2  and ∥τ 2 ∥ min     c      2  are symmetrically balanced with each other in the following manner: 
       ∥τ 1 ∥ min     c     2 −∥τ 1 ∥∥τ 2 ∥ cos θ τ     1     τ     2   +δ( y )½Σ i=1   l Ψ i* ≡½∥τ∥ min     c     2 ,
 
     and 
       ∥τ 2 ∥ min     c     2 −∥τ 2 ∥∥τ 1 ∥ cos θ τ     2     τ     1   +δ( y )½Σ i=1   l Ψ i* ≡½∥τ∥ min     c     2 ,
 
     where the equalizer statistic 
     
       
         
           
             
               
                 
                   
                     
                         
                     
                      
                     
                       
                         
                           δ 
                            
                           
                             ( 
                             y 
                             ) 
                           
                         
                          
                         
                           1 
                           2 
                         
                          
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             l 
                           
                            
                           
                             
                               ψ 
                               
                                 i 
                                 * 
                               
                             
                              
                             
                               : 
                             
                              
                             
                               δ 
                                
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                           
                         
                       
                        
                       
                         = 
                         Δ 
                       
                        
                       
                         
                           ? 
                         
                         - 
                         
                           ξ 
                           i 
                         
                       
                     
                     ) 
                   
                    
                   
                     
 
                   
                    
                   
                     
                       ? 
                     
                      
                     
                       indicates text missing or illegible when filed 
                     
                   
                 
               
               
                 
                   ( 
                   1.48 
                   ) 
                 
               
             
           
         
       
     
     equalizes the total allowed eigenenergies ∥τ 1 ∥ min     c     2  and ∥τ 2 ∥ min     c     2  exhibited by τ t  and τ 2  so that the total allowed eigenenergies ∥τ 1 −τ 2 ∥ min     c     2  exhibited by the scaled extreme points on τ 1 −τ 2  are symmetrically balanced with each other about the fulcrum of τ: 
       ∥κ 1 ∥ min     c     2 +δ( y )½Σ i=1   l Ψ i* ≡∥κ 2 ∥ min     c     2 −δ( y )½Σ i=1   l Ψ i*    (1.49)
 
     which is located at the center of eigenenergy ∥τ∥ min     c     2  the geometric center of τ. Thereby, the eigenenergy ∥τ 1 ∥ min     c     2  associated with the position or location of the likelihood ratio p(Λ τ (x)|ω 1 ) given class ω 1  is symmetrically balanced with the eigenenergy ∥τ 2 ∥ min     c     2  associated with the position or location of the likelihood ratio p(Λ τ (x)|ω 2 ) given class ω 2  so that the likelihood ratio 
       Λ τ ( x )= p (Λ τ ( x )|ω 1 )− p (Λ τ ( x )|ω 1 )=τ 1 =τ 2  
 
     of the classification system 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     is in statistical equilibrium. 
     Returning to Eq. (1.36) 
         P ( x   1i* |τ 1 )=∫ Z τ 1   dτ   1 =∥τ 1 ∥ 2   +C   1  
 
     and Eq. (1.37) 
         P ( x   2i* |τ 2 )=∫ Z τ 2   dτ   2 =∥τ 2 ∥ 2   +C   2 ,
 
     it follows that the value for the integration constant C 1  in Eq. (1.36) is 
         C   1 =−∥τ 1 ∥∥τ 2 ∥ cos θ τ     1     τ     2    
 
     and the value for the integration constant C 2  in Eq. (1.37) is 
         C   2 =−∥τ 2 ∥∥τ 1 ∥ cos θ τ     2     τ     1   .
 
     Therefore, the area P(x 1i* |τ 1 ) under the class-conditional density function p(x 1i* |τ 1 ) in Eq. (1.36): 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           P 
                            
                           
                             ( 
                             
                               
                                 x 
                                 
                                   1 
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                               | 
                               
                                 τ 
                                 1 
                               
                             
                             ) 
                           
                         
                         = 
                           
                          
                         
                           
                             
                               ∫ 
                               Z 
                             
                              
                             
                               
                                 p 
                                  
                                 
                                   ( 
                                   
                                     
                                       x 
                                       
                                         1 
                                          
                                         
                                           i 
                                           * 
                                         
                                       
                                     
                                     | 
                                     
                                       τ 
                                       1 
                                     
                                   
                                   ) 
                                 
                               
                                
                               d 
                                
                               
                                   
                               
                                
                               
                                 τ 
                                 1 
                               
                             
                           
                           + 
                           
                             
                               δ 
                                
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                              
                             
                               1 
                               2 
                             
                              
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 l 
                               
                                
                               
                                 ψ 
                                 
                                   i 
                                   * 
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                          
                         
                           
                             
                               ∫ 
                               Z 
                             
                              
                             
                               
                                 τ 
                                 1 
                               
                                
                               d 
                                
                               
                                   
                               
                                
                               
                                 τ 
                                 1 
                               
                             
                           
                           + 
                           
                             
                               δ 
                                
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                              
                             
                               1 
                               2 
                             
                              
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 l 
                               
                                
                               
                                 ψ 
                                 
                                   i 
                                   * 
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                          
                         
                           
                             
                                
                               
                                 τ 
                                 1 
                               
                                
                             
                             
                               m 
                                
                               
                                   
                               
                                
                               i 
                                
                               
                                   
                               
                                
                               
                                 n 
                                 c 
                               
                             
                             2 
                           
                           - 
                           
                             
                                
                               
                                 τ 
                                 1 
                               
                                
                             
                              
                             
                                
                               
                                 τ 
                                 2 
                               
                                
                             
                              
                             cos 
                              
                             
                                 
                             
                              
                             
                               θ 
                               
                                 
                                   τ 
                                   1 
                                 
                                  
                                 
                                   τ 
                                   2 
                                 
                               
                             
                           
                           + 
                           
                             
                               δ 
                                
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                              
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 
                                   l 
                                   1 
                                 
                               
                                
                               
                                 ψ 
                                 
                                   1 
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         
                           ≡ 
                             
                            
                           
                             
                               1 
                               2 
                             
                              
                             
                               
                                  
                                 τ 
                                  
                               
                               
                                 m 
                                  
                                 
                                     
                                 
                                  
                                 i 
                                  
                                 
                                     
                                 
                                  
                                 
                                   n 
                                   c 
                                 
                               
                               2 
                             
                           
                         
                         , 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1.50 
                   ) 
                 
               
             
           
         
       
     
     over the decision space Z, is symmetrically balanced with the area under the class-conditional density function p(x 2i* |τ 2 ) in Eq. (1.37): 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           P 
                            
                           
                             ( 
                             
                               
                                 x 
                                 
                                   2 
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                               | 
                               
                                 τ 
                                 2 
                               
                             
                             ) 
                           
                         
                         = 
                           
                          
                         
                           
                             
                               ∫ 
                               Z 
                             
                              
                             
                               
                                 p 
                                  
                                 
                                   ( 
                                   
                                     
                                       x 
                                       
                                         2 
                                          
                                         
                                           i 
                                           * 
                                         
                                       
                                     
                                     | 
                                     
                                       τ 
                                       2 
                                     
                                   
                                   ) 
                                 
                               
                                
                               d 
                                
                               
                                   
                               
                                
                               
                                 τ 
                                 2 
                               
                             
                           
                           - 
                           
                             
                               δ 
                                
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                              
                             
                               1 
                               2 
                             
                              
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 l 
                               
                                
                               
                                 ψ 
                                 
                                   i 
                                   * 
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                          
                         
                           
                             
                               ∫ 
                               Z 
                             
                              
                             
                               
                                 τ 
                                 2 
                               
                                
                               d 
                                
                               
                                   
                               
                                
                               
                                 τ 
                                 2 
                               
                             
                           
                           - 
                           
                             
                               δ 
                                
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                              
                             
                               1 
                               2 
                             
                              
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 l 
                               
                                
                               
                                 ψ 
                                 
                                   i 
                                   * 
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                          
                         
                           
                             
                                
                               
                                 τ 
                                 2 
                               
                                
                             
                             
                               m 
                                
                               
                                   
                               
                                
                               i 
                                
                               
                                   
                               
                                
                               
                                 n 
                                 c 
                               
                             
                             2 
                           
                           - 
                           
                             
                                
                               
                                 τ 
                                 2 
                               
                                
                             
                              
                             
                                
                               
                                 τ 
                                 1 
                               
                                
                             
                              
                             cos 
                              
                             
                                 
                             
                              
                             
                               θ 
                               
                                 
                                   τ 
                                   2 
                                 
                                  
                                 
                                   τ 
                                   1 
                                 
                               
                             
                           
                           + 
                           
                             
                               δ 
                                
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                              
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 
                                   l 
                                   2 
                                 
                               
                                
                               
                                 ψ 
                                 
                                   2 
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         
                           ≡ 
                             
                            
                           
                             
                               1 
                               2 
                             
                              
                             
                               
                                  
                                 τ 
                                  
                               
                               
                                 m 
                                  
                                 
                                     
                                 
                                  
                                 i 
                                  
                                 
                                     
                                 
                                  
                                 
                                   n 
                                   c 
                                 
                               
                               2 
                             
                           
                         
                         , 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1.51 
                   ) 
                 
               
             
           
         
       
     
     over the decision space Z, where the area P(x 1i* |τ 1 ) under p(x 1i* |τ 1 ) and the area P(x 2i* |τ 2 ) under p(x 2i* |τ 2 ) are constrained to be equal to ½∥τ∥ min     c     2  by means of the equalizer statistic in Eq. (1.48). 
     It follows that the linear discriminant function Λ τ (x)=τ T x+τ 0  is the solution to the integral equation 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           f 
                            
                           
                             ( 
                             
                               Λ 
                               τ 
                             
                             ) 
                           
                         
                         = 
                           
                          
                         
                           
                             
                               ∫ 
                               
                                 Z 
                                 1 
                               
                             
                              
                             
                               
                                 τ 
                                 1 
                               
                                
                               d 
                                
                               
                                   
                               
                                
                               
                                 τ 
                                 1 
                               
                             
                           
                           + 
                           
                             
                               ∫ 
                               
                                 Z 
                                 2 
                               
                             
                              
                             
                               
                                 τ 
                                 1 
                               
                                
                               d 
                                
                               
                                   
                               
                                
                               
                                 τ 
                                 1 
                               
                             
                           
                           + 
                           
                             
                               δ 
                                
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                              
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 
                                   l 
                                   1 
                                 
                               
                                
                               
                                 ψ 
                                 
                                   1 
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         
                           = 
                             
                            
                           
                             
                               
                                 ∫ 
                                 
                                   Z 
                                   1 
                                 
                               
                                
                               
                                 
                                   τ 
                                   2 
                                 
                                  
                                 d 
                                  
                                 
                                     
                                 
                                  
                                 
                                   τ 
                                   2 
                                 
                               
                             
                             + 
                             
                               
                                 ∫ 
                                 
                                   Z 
                                   2 
                                 
                               
                                
                               
                                 
                                   τ 
                                   2 
                                 
                                  
                                 d 
                                  
                                 
                                     
                                 
                                  
                                 
                                   τ 
                                   2 
                                 
                               
                             
                             - 
                             
                               
                                 δ 
                                  
                                 
                                   ( 
                                   y 
                                   ) 
                                 
                               
                                
                               
                                 
                                   ∑ 
                                   
                                     i 
                                     = 
                                     1 
                                   
                                   
                                     l 
                                     2 
                                   
                                 
                                  
                                 
                                   ψ 
                                   
                                     2 
                                      
                                     
                                       i 
                                       * 
                                     
                                   
                                 
                               
                             
                           
                         
                         , 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1.52 
                   ) 
                 
               
             
           
         
       
     
     over the decision space Z=Z 1 +Z 2  where the dual likelihood ratios Λ Ψ (x)=Ψ 1 +Ψ 2 and Λ 96 (x)=τ 1 −τ 2  are in statistical equilibrium, so that all of the forces associated with Bayes&#39; counter risks  (Z 1 |τ 1 ) and Bayes&#39; risks  (Z 2 |τ 1 ) in the Z 1  and Z 2  decision regions: which are related to positions and potential locations of extreme points x 1i*  that are generated according to p(x|ω 1 ), are balanced with all of the forces associated with Bayes&#39; risks  (Z 1 |τ 2 ) and Bayes&#39; counter risks  (Z 2 |τ 2 ) in the Z 1  and Z 2  decision regions: which are related to positions and potential locations of extreme points x 2i*  that are generated according to p(x|ω 2 ), and the eigenenergy ∥τ 1 ∥ min     c     2  associated with the position or location of the likelihood ratio p(Λ τ (x)|ω 1 ) given class ω 1  is balanced with the eigenenergy ∥τ 2 ∥ min     c     2  associated with the position or location of the likelihood ratio p(Λ τ (x)|ω 2 ) given class ω 2 . 
     Equation (1.52) can be rewritten as: 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           f 
                            
                           
                             ( 
                             
                               Λ 
                               τ 
                             
                             ) 
                           
                         
                         = 
                           
                          
                         
                           
                             
                               ∫ 
                               
                                 Z 
                                 1 
                               
                             
                              
                             
                               
                                 τ 
                                 1 
                               
                                
                               d 
                                
                               
                                   
                               
                                
                               
                                 τ 
                                 1 
                               
                             
                           
                           - 
                           
                             
                               ∫ 
                               
                                 Z 
                                 1 
                               
                             
                              
                             
                               
                                 τ 
                                 2 
                               
                                
                               d 
                                
                               
                                   
                               
                                
                               
                                 τ 
                                 2 
                               
                             
                           
                           + 
                           
                             
                               δ 
                                
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                              
                             
                               
                                 ∑ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                                 
                                   l 
                                   1 
                                 
                               
                                
                               
                                 ψ 
                                 
                                   1 
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         
                           = 
                             
                            
                           
                             
                               
                                 ∫ 
                                 
                                   Z 
                                   2 
                                 
                               
                                
                               
                                 
                                   τ 
                                   2 
                                 
                                  
                                 d 
                                  
                                 
                                     
                                 
                                  
                                 
                                   τ 
                                   2 
                                 
                               
                             
                             + 
                             
                               
                                 ∫ 
                                 
                                   Z 
                                   2 
                                 
                               
                                
                               
                                 
                                   τ 
                                   1 
                                 
                                  
                                 d 
                                  
                                 
                                     
                                 
                                  
                                 
                                   τ 
                                   1 
                                 
                               
                             
                             - 
                             
                               
                                 δ 
                                  
                                 
                                   ( 
                                   y 
                                   ) 
                                 
                               
                                
                               
                                 
                                   ∑ 
                                   
                                     i 
                                     = 
                                     1 
                                   
                                   
                                     l 
                                     2 
                                   
                                 
                                  
                                 
                                   ψ 
                                   
                                     2 
                                      
                                     
                                       i 
                                       * 
                                     
                                   
                                 
                               
                             
                           
                         
                         , 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1.53 
                   ) 
                 
               
             
           
         
       
     
     where all of the eigenenergies ∥Ψ 1i* x 1i* ∥ min     c     2  and ∥Ψ 2i* x 2i* ∥ min     c     2  associated with Bayes&#39; counter risk  (Z 1 |τ 1 ) and Bayes&#39; risk  (Z 1 |τ 2 ) in the Z 1  decision region are symmetrically balanced with all of the eigenenergies ∥Ψ 2i* x 1i* ∥ min     c     2  and ∥Ψ 1i* x 1i* ∥ min     c     2  associated with Bayes&#39; counter risk  (Z 2 |τ 2 ) and Bayes&#39; risk  (Z 2 |τ 1 ) in the Z 2  decision region. 
     The equilibrium point of the integral equation in Eq. (1.52) and its derivative in Eq. (1.53) is a dual locus of principal eigenaxis components and likelihoods 
     
       
         
           
             
               
                 
                   ψ 
                    
                   
                     
                       = 
                       Δ 
                     
                     ψ 
                   
                    
                     
                    
                   
                     
                       
                         
                             
                           z 
                         
                          
                         p 
                       
                        
                       
                         ( 
                         
                           
                             
                               Λ 
                               ψ 
                             
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           | 
                           
                             ω 
                             1 
                           
                         
                         ) 
                       
                     
                     + 
                     
                       p 
                        
                       
                         ( 
                         
                           
                             
                               Λ 
                               ψ 
                             
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           | 
                           
                             ω 
                             2 
                           
                         
                         ) 
                       
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       ψ 
                       1 
                     
                     + 
                     
                       ψ 
                       2 
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         
                           l 
                           1 
                         
                       
                        
                       
                         
                           ψ 
                           
                             1 
                              
                             
                               i 
                               * 
                             
                           
                         
                          
                         
                           
                             x 
                             
                               1 
                                
                               
                                 i 
                                 * 
                               
                             
                           
                           
                              
                             
                               x 
                               
                                 1 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                              
                           
                         
                       
                     
                     + 
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         
                           l 
                           2 
                         
                       
                        
                       
                         
                           ψ 
                           
                             2 
                              
                             
                               i 
                               * 
                             
                           
                         
                          
                         
                           
                             x 
                             
                               2 
                                
                               
                                 i 
                                 * 
                               
                             
                           
                           
                              
                             
                               x 
                               
                                 2 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                              
                           
                         
                       
                     
                   
                 
               
             
           
         
       
     
     that is constrained to be in statistical equilibrium: 
     
       
         
           
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 
                   l 
                   1 
                 
               
                
               
                 
                   ψ 
                   
                     1 
                      
                     
                       i 
                       * 
                     
                   
                 
                  
                 
                   
                     x 
                     
                       1 
                        
                       
                         i 
                         * 
                       
                     
                   
                   
                      
                     
                       x 
                       
                         1 
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                   
                 
               
             
             = 
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 
                   l 
                   2 
                 
               
                
               
                 
                   ψ 
                   
                     2 
                      
                     
                       i 
                       * 
                     
                   
                 
                  
                 
                   
                     
                       x 
                       
                         2 
                          
                         
                           i 
                           * 
                         
                       
                     
                     
                        
                       
                         x 
                         
                           2 
                            
                           
                             i 
                             * 
                           
                         
                       
                        
                     
                   
                   . 
                 
               
             
           
         
       
     
     Therefore, the Bayes&#39; risk  (Z|Λ τ (x)) and the eigenenergy E min (Z|τ) linear classification system 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     are governed by the equilibrium point: 
       Σ i=1   l     1   Ψ 1i* −Σ i=1   l     2   Ψ 2i* =0
 
     of the integral equation f(Λ τ ) in Eq. (1.52). 
     The discrete linear classification theorem that is outlined next summarizes the properties of the system of fundamental data-driven locus equations that transforms two given sets of feature vectors into a linear classification system. 
     Take a collection of d-component random vectors x that are generated according to probability density functions p(x|ω 1 ) and p(x|ω 2 ) related to statistical distributions of random vectors x that have constant or unchanging statistics, and let 
     
       
         
           
             
               
                 Λ 
                 τ 
               
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 
                   
                     τ 
                     T 
                   
                    
                   x 
                 
                 + 
                 
                   τ 
                   0 
                 
               
                
               
                 ≷ 
                 
                   ω 
                   2 
                 
               
             
           
         
       
     
     denote the likelihood ratio test for a discrete, linear classification system, where ω 1  or ω 2  is the true data category, is a locus of principal eigenaxis components and likelihoods: 
     
       
         
           
             
                 
             
              
             
               
                 
                   
                     τ 
                      
                     
                       = 
                       Δ 
                     
                      
                       
                      
                     
                       ? 
                     
                   
                 
               
               
                 
                   
                     = 
                       
                      
                     
                       
                         τ 
                         1 
                       
                       - 
                       
                         τ 
                         2 
                       
                     
                   
                 
               
               
                 
                   
                     
                       = 
                         
                        
                       
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             
                               l 
                               1 
                             
                           
                            
                           
                               
                           
                            
                           
                             
                               ψ 
                               
                                 1 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                              
                             
                               x 
                               
                                 1 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                           
                         
                         - 
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             
                               l 
                               2 
                             
                           
                            
                           
                               
                           
                            
                           
                             
                               ψ 
                               
                                 2 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                              
                             
                               x 
                               
                                 2 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                           
                         
                       
                     
                     , 
                   
                 
               
             
           
         
       
       
         
           
             
               ? 
             
              
             
               indicates text missing or illegible when filed 
             
           
         
       
     
     where 
     
       
         
           
             
               
                 
                   
                       
                   
                    
                   
                     
                       x 
                       
                         1 
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                     
                       ~ 
                     
                      
                     
                       ? 
                     
                   
                   ) 
                 
                 , 
                 
                   
                     x 
                     
                       2 
                        
                       
                         i 
                         * 
                       
                     
                   
                    
                   
                     ~ 
                   
                    
                   
                     ? 
                   
                 
               
               ) 
             
             , 
             
               
 
             
              
             
               
                 ? 
               
                
               
                 indicates text missing or illegible when filed 
               
             
           
         
       
     
     Ψ 1i*  and Ψ 2i*  are scale factors that provide unit measures of likelihood for respective data points x 1i*  and x 2i*  which lie in either overlapping regions or tails regions of data distributions related to p(x|ω 1 ) and p(x|ω 2 ), and τ 0  is a functional of τ: 
       τ 0 =Σ i=1   l   y   i (1−ξ i )−Σ i=1   l   x   i*   T τ,
 
     where Σ i=1   l x i* =Σ i=1   l     2   x 1i* +Σ i=1   l     2   x 2i*  is a cluster of the data points x 1i*  and x 2i*  used to form Ε, y, are class membership statistics: if x 1i* ϵω 1 , assign y i =+1; if x 2i* ϵω 2 , assign y i =−1, and are regularization parameters: ξ i =ξ=0 for full rank Gram matrices or ξ i =ξ&lt;&lt; for low rank Gram matrices. 
     The linear discriminant function 
       Λ τ ( x )=τ T   x+τ   0  
 
     is the solution to the integral equation 
     
       
         
           
             
               
                 f 
                  
                 
                   ( 
                   
                     Λ 
                     τ 
                   
                   ) 
                 
               
               = 
               
                 
                   
                     
                       ∫ 
                       
                         Z 
                         1 
                       
                     
                      
                     
                       
                         τ 
                         1 
                       
                        
                       d 
                        
                       
                           
                       
                        
                       
                         τ 
                         1 
                       
                     
                   
                   + 
                   
                     
                       ∫ 
                       
                         Z 
                         2 
                       
                     
                      
                     
                       
                         τ 
                         1 
                       
                        
                       d 
                        
                       
                           
                       
                        
                       
                         τ 
                         1 
                       
                     
                   
                   + 
                   
                     
                       δ 
                        
                       
                         ( 
                         y 
                         ) 
                       
                     
                      
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         
                           l 
                           1 
                         
                       
                        
                       
                         ψ 
                         
                           1 
                            
                           
                             i 
                             * 
                           
                         
                       
                     
                   
                 
                 = 
                 
                   
                     
                       ∫ 
                       
                         Z 
                         1 
                       
                     
                      
                     
                       
                         τ 
                         2 
                       
                        
                       d 
                        
                       
                           
                       
                        
                       
                         τ 
                         2 
                       
                     
                   
                   + 
                   
                     
                       ∫ 
                       
                         Z 
                         2 
                       
                     
                      
                     
                       
                         τ 
                         2 
                       
                        
                       d 
                        
                       
                           
                       
                        
                       
                         τ 
                         2 
                       
                     
                   
                   - 
                   
                     
                       δ 
                        
                       
                         ( 
                         y 
                         ) 
                       
                     
                      
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         
                           l 
                           2 
                         
                       
                        
                       
                         ψ 
                         
                           2 
                            
                           
                             i 
                             * 
                           
                         
                       
                     
                   
                 
               
             
             , 
           
         
       
     
     over the decision space Z=Z 1 +Z 2 , where Z 1  and Z 2  are congruent decision regions: 
     
       
         
           
             
               
                   
               
                
               
                 
                   
                     Z 
                     1 
                   
                   ≃ 
                   
                     
                       ? 
                     
                      
                     
                         
                     
                      
                     and 
                      
                     
                         
                     
                      
                     
                       δ 
                        
                       
                         ( 
                         y 
                         ) 
                       
                     
                   
                 
                  
                 
                   = 
                   Δ 
                 
                  
                 
                   
                     
                       ? 
                     
                      
                     1 
                   
                   - 
                   
                     ξ 
                     i 
                   
                 
               
               ) 
             
             , 
             
               
 
             
              
             
               
                 ? 
               
                
               
                 indicates text missing or illegible when filed 
               
             
           
         
       
     
     such that the Bayes&#39; risk  (Z|Λ τ (x)) and the corresponding eigenenergy E min (Z|τ) of the linear classification system 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     are governed by the equilibrium point: 
       Σ i=1   l     1   Ψ 1i* −Σ i=1   l     2   Ψ 2i* =0.
 
     of the integral equation f(Λ τ (x)), where the equilibrium point is a dual locus of principal eigenaxis components and likelihoods 
     
       
         
           
             
                 
             
              
             
               
                 
                   
                     ψ 
                      
                     
                       = 
                       Δ 
                     
                      
                       
                      
                     
                       ? 
                     
                   
                 
               
               
                 
                   
                     = 
                       
                      
                     
                       
                         ψ 
                         1 
                       
                       + 
                       
                         ψ 
                         2 
                       
                     
                   
                 
               
               
                 
                   
                     = 
                       
                      
                     
                       
                         
                           ∑ 
                           
                             i 
                             = 
                             1 
                           
                           
                             l 
                             1 
                           
                         
                          
                         
                             
                         
                          
                         
                           
                             ψ 
                             
                               1 
                                
                               
                                 i 
                                 * 
                               
                             
                           
                            
                           
                             
                               x 
                               
                                 1 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                             
                                
                               
                                 x 
                                 
                                   1 
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                                
                             
                           
                         
                       
                       + 
                       
                         
                           ∑ 
                           
                             i 
                             = 
                             1 
                           
                           
                             l 
                             2 
                           
                         
                          
                         
                             
                         
                          
                         
                           
                             ψ 
                             
                               2 
                                
                               
                                 i 
                                 * 
                               
                             
                           
                            
                           
                             
                               x 
                               
                                 2 
                                  
                                 
                                   i 
                                   * 
                                 
                               
                             
                             
                                
                               
                                 x 
                                 
                                   2 
                                    
                                   
                                     i 
                                     * 
                                   
                                 
                               
                                
                             
                           
                         
                       
                     
                   
                 
               
             
           
         
       
       
         
           
             
               ? 
             
              
             
               indicates text missing or illegible when filed 
             
           
         
       
     
     that is constrained to be in statistical equilibrium: 
     
       
         
           
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 
                   l 
                   1 
                 
               
                
               
                   
               
                
               
                 
                   ψ 
                   
                     1 
                      
                     
                       i 
                       * 
                     
                   
                 
                  
                 
                   
                     x 
                     
                       1 
                        
                       
                         i 
                         * 
                       
                     
                   
                   
                      
                     
                       x 
                       
                         1 
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                   
                 
               
             
             = 
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 
                   l 
                   2 
                 
               
                
               
                   
               
                
               
                 
                   ψ 
                   
                     2 
                      
                     
                       i 
                       * 
                     
                   
                 
                  
                 
                   
                     
                       x 
                       
                         2 
                          
                         
                           i 
                           * 
                         
                       
                     
                     
                        
                       
                         x 
                         
                           2 
                            
                           
                             i 
                             * 
                           
                         
                       
                        
                     
                   
                   . 
                 
               
             
           
         
       
     
     Therefore, the forces associated with Bayes&#39; counter risk  (Z 1 |p(Λ τ (x)|ω 1 )) and Bayes&#39; risk  (Z 2 |p(Λ τ (x)|ω 1 )) in the Z 1  and Z 2 . decision regions: which are related to positions and potential locations of data points x 1i*  , that are generated according to p(x|ω 1 ) are balanced with the forces associated with Bayes&#39; risk  (Z 1 |p(Λ τ (x)|ω 2 )) ) and Bayes&#39; counter risk  (Z 2 |p(Λ τ (x)|ω 2 )) in the Z 1  and Z 2  decision regions: which are related to positions and potential locations of data points x 2i*  that are generated according to p(x)|ω 2 ). 
     Furthermore, the eigenenergy E min p(Z|p(Λ τ (x)|ω 2 )) associated with the position or location of the likelihood ratio p(Λ τ (x)|ω 1 ) given class ω 1  is balanced with the eigenergy E min p(Z|p(Λ τ (x)|ω 2 )) associated with the position or location of the likelihood ratio p(Λ τ (x)|ω 2 ) given class ω 2 : 
       ∥τ 1 ∥ min     c     2 +δ( y )½Σ i=1   l Ψ i* ≡∥τ 2 ∥ min     c     2 −δ( y )½Σ i=1   l Ψ i*  
 
     where the total eigenenergy 
     
       
         
           
             
               
                 
                   
                     
                        
                       τ 
                        
                     
                     
                       min 
                       c 
                     
                     2 
                   
                   = 
                     
                    
                   
                     
                        
                       
                         
                           τ 
                           1 
                         
                         - 
                         
                           τ 
                           2 
                         
                       
                        
                     
                     
                       min 
                       c 
                     
                     2 
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       [ 
                       
                         
                           
                              
                             
                               τ 
                               1 
                             
                              
                           
                           
                             min 
                             c 
                           
                           2 
                         
                         - 
                         
                           
                              
                             
                               τ 
                               1 
                             
                              
                           
                            
                           
                              
                             
                               τ 
                               2 
                             
                              
                           
                            
                           cos 
                            
                           
                               
                           
                            
                           
                             θ 
                             
                               
                                 τ 
                                 1 
                               
                                
                               
                                 τ 
                                 2 
                               
                             
                           
                         
                       
                       ] 
                     
                     + 
                   
                 
               
             
             
               
                 
                     
                    
                   
                     [ 
                     
                       
                         
                            
                           τ 
                            
                         
                         
                           min 
                           c 
                         
                         2 
                       
                       - 
                       
                         
                            
                           
                             τ 
                             2 
                           
                            
                         
                          
                         
                            
                           
                             τ 
                             1 
                           
                            
                         
                          
                         cos 
                          
                         
                             
                         
                          
                         
                           θ 
                           
                             
                               τ 
                               2 
                             
                              
                             
                               τ 
                               1 
                             
                           
                         
                       
                     
                     ] 
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         
                           l 
                           1 
                         
                       
                        
                       
                           
                       
                        
                       
                         
                           ψ 
                           
                             1 
                              
                             
                               i 
                               * 
                             
                           
                         
                          
                         
                           ( 
                           
                             1 
                             - 
                             
                               ξ 
                               i 
                             
                           
                           ) 
                         
                       
                     
                     + 
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         
                           l 
                           2 
                         
                       
                        
                       
                           
                       
                        
                       
                         
                           ψ 
                           
                             2 
                              
                             
                               i 
                               * 
                             
                           
                         
                          
                         
                           ( 
                           
                             1 
                             - 
                             
                               ξ 
                               i 
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         1 
                       
                       
                         l 
                         1 
                       
                     
                      
                     
                         
                     
                      
                     
                       
                         ψ 
                         
                           i 
                           * 
                         
                       
                        
                       
                         ( 
                         
                           1 
                           - 
                           
                             ξ 
                             i 
                           
                         
                         ) 
                       
                     
                   
                 
               
             
           
         
       
     
     of the discrete, linear classification system 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     is determined by the eigenenergies associated with the position or location of the likelihood ratio τ=τ 1 −τand the locus of a corresponding, linear decision boundary τ T x+τ 0 =0. 
     It follows that the discrete, linear classification system 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     is in statistical equilibrium: 
     
       
         
           
             
               
                 f 
                  
                 
                   ( 
                   
                     Λ 
                     τ 
                   
                   ) 
                 
               
               = 
               
                 
                   
                     
                       ∫ 
                       
                         Z 
                         1 
                       
                     
                      
                     
                       
                         τ 
                         1 
                       
                        
                       d 
                        
                       
                           
                       
                        
                       
                         τ 
                         1 
                       
                     
                   
                   - 
                   
                     
                       ∫ 
                       
                         Z 
                         1 
                       
                     
                      
                     
                       
                         τ 
                         2 
                       
                        
                       d 
                        
                       
                           
                       
                        
                       
                         τ 
                         2 
                       
                     
                   
                   + 
                   
                     
                       δ 
                        
                       
                         ( 
                         y 
                         ) 
                       
                     
                      
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         
                           l 
                           1 
                         
                       
                        
                       
                         ψ 
                         
                           1 
                            
                           
                             i 
                             * 
                           
                         
                       
                     
                   
                 
                 = 
                 
                   
                     
                       ∫ 
                       
                         Z 
                         2 
                       
                     
                      
                     
                       
                         τ 
                         2 
                       
                        
                       d 
                        
                       
                           
                       
                        
                       
                         τ 
                         2 
                       
                     
                   
                   - 
                   
                     
                       ∫ 
                       
                         Z 
                         2 
                       
                     
                      
                     
                       
                         τ 
                         1 
                       
                        
                       d 
                        
                       
                           
                       
                        
                       
                         τ 
                         1 
                       
                     
                   
                   - 
                   
                     
                       δ 
                        
                       
                         ( 
                         y 
                         ) 
                       
                     
                      
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         
                           l 
                           2 
                         
                       
                        
                       
                         ψ 
                         
                           2 
                            
                           
                             i 
                             * 
                           
                         
                       
                     
                   
                 
               
             
             , 
           
         
       
     
     where the forces associated with Bayes&#39; counter risk  (Z 1 |p(Λ τ (x)|ω 1 )) and Bayes&#39; risk  (Z 1 |p(Λ τ (x)|ω 2 )) in the Z 1  decision region are balanced with the forces associated with Bayes&#39; counter risk  (Z 2 |p(Λ τ (x)|ω 2 )) and Bayes&#39; risk  (Z 2 |p(Λ τ (x)|ω 1 )) in the Z 2  decision region such that the Bayes&#39; risk  (Z|Λ τ (s)) of the classification system is minimized, and the eigenenergies associated with Bayes&#39; counter risk  (Z 1 |τ 1 ) and Bayes&#39; risk  (Z 1 |τ 2 ) in the Z 1  decision region are balanced with the eigenenergies associated with Bayes&#39; counter risk  (Z 2 |τ 2 ) and Bayes&#39; risk  (Z 2 |τ 1 ) in the Z 2  decision region such that the eigenenergy E min  (Z|τ) of the classification system is minimized.
 
Thus, any given discrete, linear classification system
 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     exhibits an error rate that is consistent with the Bayes&#39; risk  (Z 1 |τ 1 (x)) and the corresponding eigenenergy E min (Z|τ) of the classification system: for all random vectors x that are generated according to p(x|ω 1 ) and p(x|ω 2 ), where p(x|ω 1 ) and p(x|ω 2 ) are related to statistical distributions of random vectors x that have similar covariance functions and constant or unchanging statistics. 
     Thereby, a discrete, linear classification system 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     seeks a point of statistical equilibrium where the opposing forces and influences of the classification system are balanced with each other, such that the eigenenergy and the Bayes&#39; risk of the classification system are minimized, and the classification system is in statistical equilibrium. 
     Furthermore, the eigenenergy ∥τ∥ min     c     2 −∥τ 1 −τ 2 ∥ min     c     2  is the state of a discrete, linear classification system 
     
       
         
           
             
               
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     that is associated with the position or location of a dual likelihood ratio: 
     
       
         
           
             ψ 
              
             
               = 
               Δ 
             
              
             
               
                 
                   
                     ? 
                   
                    
                   
                     p 
                      
                     
                       ( 
                       
                         
                           
                             Λ 
                             ψ 
                           
                            
                           
                             ( 
                             x 
                             ) 
                           
                         
                          
                         
                           ω 
                           1 
                         
                       
                       ) 
                     
                   
                 
                 + 
                 
                   p 
                    
                   
                     ( 
                     
                       
                         
                           Λ 
                           ψ 
                         
                          
                         
                           ( 
                           x 
                           ) 
                         
                       
                        
                       
                         ω 
                         2 
                       
                     
                     ) 
                   
                 
               
               = 
               
                 
                   
                     ψ 
                     1 
                   
                   + 
                   
                     ψ 
                     2 
                   
                 
                 = 
                 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         1 
                       
                       
                         l 
                         1 
                       
                     
                      
                     
                       
                         ψ 
                         
                           1 
                            
                           
                             i 
                             * 
                           
                         
                       
                        
                       
                         
                           x 
                           
                             1 
                              
                             
                               i 
                               * 
                             
                           
                         
                         
                            
                           
                             x 
                             
                               1 
                                
                               
                                 i 
                                 * 
                               
                             
                           
                            
                         
                       
                     
                   
                   + 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         1 
                       
                       
                         l 
                         2 
                       
                     
                      
                     
                         
                     
                      
                     
                       
                         ψ 
                         
                           2 
                            
                           
                             i 
                             * 
                           
                         
                       
                        
                       
                         
                           x 
                           
                             2 
                              
                             
                               i 
                               * 
                             
                           
                         
                         
                            
                           
                             x 
                             
                               2 
                                
                               
                                 i 
                                 * 
                               
                             
                           
                            
                         
                       
                     
                   
                 
               
             
           
         
       
       
         
           
             
               ? 
             
              
             
               indicates text missing or illegible when filed 
             
           
         
       
     
     which is constrained to be in statistical equilibrium: 
     
       
         
           
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 
                   l 
                   1 
                 
               
                
               
                 
                   ψ 
                   
                     1 
                      
                     
                         
                     
                      
                     
                       i 
                       * 
                     
                   
                 
                  
                 
                   
                     x 
                     
                       1 
                        
                       
                           
                       
                        
                       
                         i 
                         * 
                       
                     
                   
                   
                      
                     
                       x 
                       
                         1 
                          
                         
                             
                         
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                   
                 
               
             
             = 
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 
                   l 
                   2 
                 
               
                
               
                 
                   ψ 
                   
                     2 
                      
                     
                         
                     
                      
                     
                       i 
                       * 
                     
                   
                 
                  
                 
                   
                     x 
                     
                       2 
                        
                       
                           
                       
                        
                       
                         i 
                         * 
                       
                     
                   
                   
                      
                     
                       x 
                       
                         2 
                          
                         
                             
                         
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                   
                 
               
             
           
         
       
     
     and the locus of a corresponding, linear decision boundary τ T x+τ 0 =0. 
     In summary, discrete, linear classification systems 
     
       
         
           
             
                 
             
              
             
               
                 
                   
                     τ 
                     T 
                   
                    
                   x 
                 
                 + 
                 
                   τ 
                   0 
                 
               
               ≷ 
               
                 ? 
               
             
           
         
       
       
         
           
             
               ? 
             
              
             
               indicates text missing or illegible when filed 
             
           
         
       
     
     have the following unique and advantageous features. 
     Discrete, linear classification systems are a class of high-performance learning machines, where the architecture of any given learning machine satisfies equations of statistical equilibrium along with equations of minimization of eigenenergy and Bayes&#39; risk. Any given learning machine Λ τ (x)=τ T x+τ 0  is the solution to fundamental integral equations of likelihood ratios and corresponding decision boundaries, so that the learning machine finds a point of statistical equilibrium where the opposing forces and influences of a binary classification system are balanced with each other, and the eigenenergy and the corresponding Bayes&#39; risk of the learning machine are minimized. Thereby, the generalization error of any given learning machine is a function of the amount of overlap between data distributions, where any given discrete, linear classification system 
     
       
         
           
             
                 
             
              
             
               
                 
                   
                     τ 
                     T 
                   
                    
                   x 
                 
                 + 
                 
                   τ 
                   0 
                 
               
               ≷ 
               
                 ? 
               
             
           
         
       
       
         
           
             
               ? 
             
              
             
               indicates text missing or illegible when filed 
             
           
         
       
     
     generates the best possible linear decision boundary for a given collection of training data. 
     Thus, for any given set of feature vectors drawn from statistical distributions that have similar covariance functions and constant or unchanging statistics, the generalization error of each learning machine is Bayes&#39; error: which is the lowest error rate that can be achieved by a discriminant function and the best generalization error that can be achieved by a learning machine, so that the accuracy of any given learning machine Λ τ (x)=τ T x+τ 0  is the best possible for a given collection of training data. Accordingly, any given learning machine is a scalable, individual component of an optimal ensemble system, where any given ensemble system of learning machines exhibits optimal generalization performance for an M-class feature space. Optimal ensemble systems of discrete, linear discriminant functions are outlined next. 
     Let Λ τ     ij   (x) denote a discrete, linear discriminant function Λ τ (x)=τ T x+τ 0  for two given pattern classes ω i  and ω j , where the feature vectors in class ω I  have the training label +1, and the feature vectors in class ω i  have the training label −1. The discriminant function Λ τ     ij   (x) is an indicator function χ ω     i    for feature vectors x that belong to class ω i , where χ ω     i    denotes the event that an unknown feature vector xϵω 1  lies in the decision region Z 1  so that sign (Λ τ     ij   (x) =1. 
     Thereby, for any given M-class feature space {ω i } i=1   M , an ensemble of M-1 discrete, linear discriminant functions Σ j=1   M Λ τ     ij    (x) for which the discriminant function Λ τ     ij    (x) is an indicator function χ ω     i    for class ω i , provides M-1 characteristic functions χ ω     i    for feature vectors x that belong to class ω i : 
     
       
         
           
             
               E 
                
               
                 [ 
                 
                   χ 
                   
                     ω 
                     i 
                   
                 
                 ] 
               
             
             = 
             
               
                 
                   ∑ 
                   
                     j 
                     = 
                     1 
                   
                   
                     M 
                     - 
                     1 
                   
                 
                  
                 
                   P 
                    
                   
                     ( 
                     
                       
                         sign 
                          
                         
                           ( 
                           
                             
                               Λ 
                               
                                 τ 
                                 ij 
                               
                             
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           ) 
                         
                       
                       = 
                       1 
                     
                     ) 
                   
                 
               
               = 
               
                 
                   
                     ∑ 
                     
                       j 
                       = 
                       1 
                     
                     
                       M 
                       - 
                       1 
                     
                   
                    
                   
                     sign 
                      
                     
                       ( 
                       
                         
                           Λ 
                           
                             τ 
                             ij 
                           
                         
                          
                         
                           ( 
                           x 
                           ) 
                         
                       
                       ) 
                     
                   
                 
                 = 
                 1. 
               
             
           
         
       
     
     Further, because linear eigenlocus decision rules involve linear combinations of extreme vectors, scaled extreme points, class membership statistics, and regularization parameters: 
     
       
         
           
             
                 
             
              
             
               
                 
                   Λ 
                   τ 
                 
                  
                 
                   ( 
                   x 
                   ) 
                 
               
               = 
               
                 
                   
                     
                       ( 
                       
                         x 
                         - 
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             l 
                           
                            
                           
                             x 
                             
                               i 
                               * 
                             
                           
                         
                       
                       ) 
                     
                      
                     
                       τ 
                       1 
                     
                   
                   - 
                   
                     
                       ( 
                       
                         x 
                         - 
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             l 
                           
                            
                           
                             x 
                             
                               i 
                               * 
                             
                           
                         
                       
                       ) 
                     
                      
                     
                       τ 
                       2 
                     
                   
                   + 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         1 
                       
                       l 
                     
                      
                     
                       
                         y 
                         i 
                       
                        
                       
                         ( 
                         
                           1 
                           - 
                           
                             ξ 
                             i 
                           
                         
                         ) 
                       
                     
                   
                 
                 ≷ 
                 
                   ? 
                 
               
             
           
         
       
       
         
           
             
                 
             
              
             where 
           
         
       
       
         
           
             
                 
             
              
             
               
                 
                   τ 
                   1 
                 
                 = 
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     
                       l 
                       1 
                     
                   
                    
                   
                     
                       ψ 
                       
                         1 
                          
                         
                             
                         
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                     
                       x 
                       
                         1 
                          
                         
                             
                         
                          
                         
                           i 
                           * 
                         
                       
                     
                   
                 
               
               , 
               
                 
                   and 
                    
                   
                       
                   
                    
                   
                     τ 
                     2 
                   
                 
                 = 
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     
                       l 
                       1 
                     
                   
                    
                   
                     
                       ψ 
                       
                         1 
                          
                         
                             
                         
                          
                         
                           i 
                           * 
                         
                       
                     
                      
                     
                       x 
                       
                         1 
                          
                         
                             
                         
                          
                         
                           i 
                           * 
                         
                       
                     
                   
                 
               
               , 
               
                 
 
               
                
               
                 
                   ? 
                 
                  
                 
                   indicates text missing or illegible when filed 
                 
               
             
           
         
       
     
     it follows that linear combinations of linear eigenlocus discriminant functions can be used to build optimal statistical pattern recognition systems P (x), where the overall system complexity is scale-invariant for the feature space dimension and the number of pattern classes. Thus, linear eigenlocus decision rules Λ τ (x)=τ T x+τ 0  are scalable modules for optimal linear classification systems. 
     Given that a discrete, linear discriminant function Λ τ (x)=τ T x+τ 0  is an indicator function χ ω     i    for any given class of feature vectors ω i  that have the training label +1, it follows that the decision function 
     
       
         
           
             
               
                 sign 
                  
                 
                   
                     
                       
                         
                           ( 
                           
                             
                               Λ 
                               τ 
                             
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           ) 
                         
                          
                       
                     
                     
                       
                         
                           τ 
                            
                           
                               
                           
                            
                           x 
                         
                         + 
                         
                           τ 
                           0 
                         
                       
                     
                   
                 
               
               ) 
             
             , 
           
         
       
     
     where 
     
       
         
           
             
                 
             
              
             
               
                 sign 
                  
                 
                   ( 
                   x 
                   ) 
                 
               
                
                
               
                 ? 
               
             
           
         
       
       
         
           
             
               ? 
             
              
             
               indicates text missing or illegible when filed 
             
           
         
       
     
     for x≠0, provides a natural means for discriminating between multiple classes of data, where decisions can be made that are based on the largest probabilistic output of decision banks DB ω     i   (x) formed by linear combinations of linear eigenlocus decision functions sign(Λ x (x)): 
         DB   ω     i   ( x )=Σ j=1   M−1 sign(τ T   x+τ   0 )
 
     where the decision bank DB ω     τ   (x) for a pattern class ω i  is an ensemble Σ j=1   M−1 sign(Λ τ (x)) of M−1 decision functions {sign(Λ τ     j   (x))} j=1   M−1  for which the pat ern vectors in the given class ω 1  have the training label +1, and the pattern vectors in all of the other pattern classes have the training label −1. 
     The design of optimal, statistical pattern recognition systems P (x) involves designing M decision banks, where each decision bank contains an ensemble of M−1 linear decision functions sign(Λ τ (x)), and each decision function is determined by a feature extractor and a linear discriminant function Λ τ (x). A feature extractor generates d-dimensional feature vectors from collections of digital signals, digital waveforms, digital images, or digital videos for all of the M pattern classes. 
     Take M sets of d-dimensional feature vectors that have been extracted from collections of digital signals, digital waveforms, digital images, or digital videos for M pattern classes. Optimal, statistical pattern recognition systems P (x)are produced in the following manner. 
     Produce a decision bank DB ω     i   (Σ j=1   M−1 sign(Λ τ     j   (x)) for each pattern class ω 1  that consists of a bank or ensemble sign Σ j=1   M−1 sign(Λ τ     j   (x)) of M−1 decision functions {sign(Λ τ     j   (x))} j=1   M−1 . Accordingly, generate M−1 linear discriminant functions Λ τ (x), where the feature vectors in the given class ω 1  have the training label +1 and the feature vectors in all of the other pattern classes have the training label −1. 
     An optimal, statistical pattern recognition system  (X) 
     
       
         
           
             
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 { 
                 
                   
                     DB 
                     
                       ω 
                       i 
                     
                   
                   ( 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         1 
                       
                       
                         M 
                         - 
                         1 
                       
                     
                      
                     
                       sign 
                        
                       
                         ( 
                         
                           
                             Λ 
                             
                               τ 
                               j 
                             
                           
                            
                           
                             ( 
                             x 
                             ) 
                           
                         
                         ) 
                       
                     
                   
                   ) 
                 
                 } 
               
               
                 i 
                 = 
                 1 
               
               M 
             
           
         
       
     
     contains M decision banks {DB ω     i   (x)} i=1   M , i.e., M ensembles Σ j=1   M−1 sign(Λ τ     j   (x)) of optimal decision functions sign(Λ τ (x)), all of which provide a set of M×(M−1) decision statistics 
     
       
         
           
             
               { 
               
                 sign 
                  
                 
                   ( 
                   
                     
                       Λ 
                       
                         τ 
                         j 
                       
                     
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   ) 
                 
               
               } 
             
             
               j 
               = 
               1 
             
             
               M 
               × 
               
                 ( 
                 
                   M 
                   - 
                   1 
                 
                 ) 
               
             
           
         
       
     
     that minimize the probability of decision error for an M-class feature space, such that the maximum value selector of the pattern recognition system  (x) chooses the pattern class ω 1  for which a decision bank DB ω     i    (x) has the maximum probabilistic output: 
     
       
         
           
             
               
                  
                 
                   ( 
                   x 
                   ) 
                 
               
               = 
               
                 
                   
                     
                       
                         Arg 
                          
                         
                             
                         
                          
                         Max 
                       
                     
                   
                   
                     
                       
                         
                           i 
                           ∈ 
                           1 
                         
                         , 
                         … 
                       
                     
                   
                 
                  
                 
                     
                 
                  
                 
                   ( 
                   
                     
                       DB 
                       
                         ω 
                         i 
                       
                     
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   ) 
                 
               
             
             , 
           
         
       
     
     where the probabilistic output of each decision bank DB ω     i   (x) is determined by a set of M−1 characteristic functions: 
     
       
         
           
             
               E 
                
               
                 [ 
                 
                   χ 
                   
                     ω 
                     i 
                   
                 
                 ] 
               
             
             = 
             
               
                 
                   ∑ 
                   
                     j 
                     = 
                     1 
                   
                   
                     M 
                     - 
                     1 
                   
                 
                  
                 
                   P 
                    
                   
                     ( 
                     
                       
                         sign 
                          
                         
                           ( 
                           
                             
                               Λ 
                               
                                 τ 
                                 ij 
                               
                             
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           ) 
                         
                       
                       = 
                       1 
                     
                     ) 
                   
                 
               
               = 
               
                 
                   
                     ∑ 
                     
                       j 
                       = 
                       1 
                     
                     
                       M 
                       - 
                       1 
                     
                   
                    
                   
                     sign 
                      
                     
                       ( 
                       
                         
                           Λ 
                           
                             τ 
                             ij 
                           
                         
                          
                         
                           ( 
                           x 
                           ) 
                         
                       
                       ) 
                     
                   
                 
                 = 
                 1. 
               
             
           
         
       
     
     For feature vectors drawn from statistical distributions that have similar covariance functions and constant or unchanging mean and covariance statistics, statistical pattern recognition systems  (x) that are formed by the ensembles of linear decision functions outlined above generate a set of linear decision boundaries and decision statistics that minimize the probability of decision error, i.e., the Bayes&#39; error. Accordingly, any statistical pattern recognition system  (x) that is formed by the ensembles of linear decision functions outlined above achieves Bayes&#39; error, which is the lowest error rate that can be achieved by a discriminant function and the best generalization error that can be achieved by a learning machine. 
     Feature vectors that have been extracted from collections of digital signals, digital waveforms, digital images, or digital videos can be fused with each other by designing decision banks for data obtained from different sources and combining the outputs of the decision banks. The method is outlined for two different data sources and is readily extended to L sources of data. 
     Take M sets of d-dimensional and n-dimensional feature vectors that have been extracted from two different collections of digital signals, digital waveforms, digital images, or digital videos for M pattern classes. Optimal, statistical pattern recognition systems  (x) are produced in the following manner. 
     Given M pattern classes {ω i } i=1   M , let DB ω     i1    and DB ω     i2    denote the decision banks for the d-dimensional and n-dimensional feature vectors respectively, where feature vectors in class ω i  have the training label +1 and feature vectors in all of the other pattern classes have the training label −1. Produce the decision banks 
         DB   ω     i1   (Σ j=1   M−1 sign(Λ τ     i   ( x )) and DB ω     i2   (Σ j=1   M−1 sign(Λ τ     j   (x))
 
     for each pattern class ω 1 , where DB ω     i1    and DB ω     i2    consist of a bank or ensemble Σ j=1   M−1 sign(Λ τ     j   (x)) of M−1 linear decision functions sign(Λ τ (x)). Accordingly, for each decision bank, generate M−1 linear discriminant functions Λ τ (x), where the feature or pattern vectors in the given class ω 1  have the training label +1 and the feature or pattern vectors in all of the other pattern classes have the training label −1. 
     For each pattern class ω 1 , the decision banks DB ω     i1    DB ω     i2    and generate two sets of M−1 decision statistics 
     
       
         
           
             
               
                 DB 
                 
                   ω 
                   
                     i 
                      
                     
                         
                     
                      
                     1 
                   
                 
               
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 { 
                 
                   sign 
                    
                   
                     ( 
                     
                       
                         Λ 
                         
                           τ 
                           j 
                         
                       
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                     ) 
                   
                 
                 } 
               
               
                 j 
                 = 
                 1 
               
               
                 M 
                 - 
                 1 
               
             
           
         
       
       
         
           and 
         
       
       
         
           
             
               
                 DB 
                 
                   ω 
                   
                     i 
                      
                     
                         
                     
                      
                     2 
                   
                 
               
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 { 
                 
                   sign 
                    
                   
                     ( 
                     
                       
                         Λ 
                         
                           τ 
                           j 
                         
                       
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                     ) 
                   
                 
                 } 
               
               
                 j 
                 = 
                 1 
               
               
                 M 
                 - 
                 1 
               
             
           
         
       
     
     such that the maximum value selector of the statistical pattern recognition system  (x) 
     
       
         
           
             
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 { 
                 
                   
                     ∑ 
                     
                       j 
                       = 
                       1 
                     
                     2 
                   
                    
                   
                     
                       DB 
                       
                         ω 
                         ij 
                       
                     
                      
                     
                       ( 
                       
                         
                           ∑ 
                           
                             k 
                             = 
                             1 
                           
                           
                             M 
                             - 
                             1 
                           
                         
                          
                         
                           sign 
                            
                           
                             ( 
                             
                               
                                 Λ 
                                 
                                   τ 
                                   k 
                                 
                               
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                             ) 
                           
                         
                       
                       ) 
                     
                   
                 
                 } 
               
               
                 i 
                 = 
                 1 
               
               M 
             
           
         
       
     
     chooses the pattern class ω i  for which the fused decision banks Σ j=1   2 DB ω     ij   (x) have the maximum probabilistic output: 
     
       
         
           
             
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 
                   
                     
                       Arg 
                        
                       
                           
                       
                        
                       Max 
                     
                   
                 
                 
                   
                     
                       
                         i 
                         ∈ 
                         1 
                       
                       , 
                       … 
                     
                   
                 
               
                
               
                   
               
                
               
                 
                   ( 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         1 
                       
                       2 
                     
                      
                     
                       
                         DB 
                         
                           ω 
                           ij 
                         
                       
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                   
                   ) 
                 
                 . 
               
             
           
         
       
     
     The method is readily extended to L different data sources. Given that fusion of decision banks based on different data sources involves linear combinations of decision banks, it follows that optimal, statistical pattern recognition systems  (x) can be designed for feature vectors that have been extracted from L different sources of digital data: 
     
       
         
           
             
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 { 
                 
                   
                     ∑ 
                     
                       j 
                       = 
                       1 
                     
                     L 
                   
                    
                   
                     
                       DB 
                       
                         ω 
                         ij 
                       
                     
                     ( 
                     
                       
                         ∑ 
                         
                           k 
                           = 
                           1 
                         
                         
                           M 
                           - 
                           1 
                         
                       
                        
                       
                         sign 
                          
                         
                           ( 
                           
                             
                               Λ 
                               
                                 τ 
                                 k 
                               
                             
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           ) 
                         
                       
                     
                     ) 
                   
                 
                 } 
               
               
                 i 
                 = 
                 1 
               
               M 
             
           
         
       
     
     such that the maximum value selector of an optimal, statistical pattern recognition system  (x) chooses the pattern class ω i  for which the L fused decision banks Σ j=1   2 DB ω     ij   (x)have the maximum probabilistic output: 
     
       
         
           
             
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 
                   
                     
                       Arg 
                        
                       
                           
                       
                        
                       Max 
                     
                   
                 
                 
                   
                     
                       
                         i 
                         ∈ 
                         1 
                       
                       , 
                       … 
                     
                   
                 
               
                
               
                   
               
                
               
                 
                   ( 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         1 
                       
                       L 
                     
                      
                     
                       
                         DB 
                         
                           ω 
                           ij 
                         
                       
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                   
                   ) 
                 
                 . 
               
             
           
         
       
     
     For the problem of learning discriminant functions and decision boundaries, an important problem involves the identification and exploitation of distinguishing features that are simple to extract, invariant to irrelevant transformations, insensitive to noise, and useful for discriminating between objects in different categories. Useful sets of distinguishing features for discrimination tasks must exhibit sufficient class separability: i.e., a negligible overlap exists between all data distributions. Further, the criteria to evaluate the effectiveness of feature vectors must be a measure of the overlap or class separability among data distributions and not a measure of fit such as the mean-square error of a statistical model. 
     Because linear eigenlocus classification systems optimize trade-offs between Bayes&#39; counter risks and Bayes&#39; risks for any data distributions that have similar covariance functions, linear eigenlocus classification systems provide measures of data distribution overlap and Bayes&#39; error rate for any two given sets of feature vectors drawn from statistical distribution that have similar covariance functions and constant or unchanging mean and covariance statistics. Accordingly, linear eigenlocus classification systems can be used to predict how well they will generalize to new patterns. 
     Linear eigenlocus decision functions provide a practical statistical gauge for measuring data distribution overlap and Bayes&#39; error rate for two given sets of feature or pattern vectors. To measure Bayes&#39; error rate and data distribution overlap, generate a linear eigenlocus classification system 
     
       
         
           
             
               
                  
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     using feature vectors that have been extracted from any given collections of digital signals, digital waveforms, digital images, or digital videos for two pattern classes. While equal numbers of training examples are not absolutely necessary, the number of training examples from each of the pattern classes should be reasonably balanced with each other. 
     Apply the decision function sign(τ T x+τ 0 ) to a collection of feature vectors which have not been used to build the classification system 
     
       
         
           
             
               
                  
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
             . 
           
         
       
     
     Compare tree Known class memberships to the predicted class memberships, and determine the error rate for each pattern class based on the frequency of incorrect predictions for each pattern class. Determine the data distribution overlap and the Bayes&#39; error rate based on the error rates of the collection of unknown feature vectors. 
     If data collection is cost prohibitive, use k-fold cross validation, where a collection of feature vectors is split randomly into k partitions. Generate a linear classification system 
     
       
         
           
             
               
                  
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     using a data set consisting of k−1 of the original k parts and use the remaining portion for testing. Repeat this process k times. Bayes&#39; error rate and data distribution overlap is the average over the k test runs. 
     Linear decision functions sign(τ T x+τ 0 ) can also be used to identify homogeneous data distributions. Generate a linear classification system 
     
       
         
           
             
               
                  
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     using samples drawn from two distributions. Apply the decision function sign(τ T x+τ 0 ) to samples which have not been used to build the classification system 
     
       
         
           
             
               
                  
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
             . 
           
         
       
     
     Given homogeneous data distributions, essentially all of the training data are transformed into constrained, primal principal eigenaxis components, such that the error rate of the linear classification system 
     
       
         
           
             
               
                  
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     is ≈50%. 
     If data collection is cost prohibitive, use k-fold cross validation, where a collection of feature vectors is split randomly into k partitions. Generate a linear classification system 
     
       
         
           
             
               
                  
                 
                   τ 
                   T 
                 
                  
                 x 
               
               + 
               
                 τ 
                 0 
               
             
              
             
               ≷ 
               
                 ω 
                 2 
               
             
           
         
       
     
     using a aata set consisting of k−1 of the original k parts and use the remaining portion for testing. Repeat this process k times. Bayes&#39; error rate and data distribution overlap is the average over the k test runs. 
     The machine learning methods disclosed herein may be readily utilized in a wide variety of applications to construct optimal statistical pattern recognition systems or optimal linear classification systems, where the data corresponds to a phenomenon of interest, e.g., outputs of sensors: radar and hyperspectral or multispectral images, biometrics, digital communication signals, text, images, digital waveforms, etc. More specifically, the applications include, for example and without limitation, general pattern recognition (including image recognition, waveform recognition, object detection, spectrum identification, and speech and handwriting recognition, data classification, (including text, image, and waveform categorization), bioinformatics (including automated diagnosis systems, biological modeling, and bioimaging classification), etc. One skilled in the art will recognize that any suitable computer system may be used to execute the machine learning methods disclosed herein. The computer system may include, without limitation, a mainframe computer system, a workstation, a personal computer system, a personal digital assistant, or other device or apparatus having at least one processor that executes instructions from a memory medium. The computer system may further include a display device or monitor for displaying operations associated with the learning machine and one or more memory mediums on which computer programs or software components may be stored. In addition, the memory medium may be entirely or partially located in one or more associated computers or computer systems which connect to the computer system over a network, such as the Internet. 
     The machine learning method described herein may also be executed in hardware, a combination of software and hardware, or in other suitable executable implementations. 
     The learning machine methods implemented in software may be executed by the processor of the computer system or the processor or processors of the one or more associated computer systems connected to the computer system. 
       FIG. 6  illustrates a flowchart of processing performed in training a linear classifier in accordance with the preferred embodiment. At step  100 , a set of labeled feature vectors is received. At step  102 , the inner product statistic used to form the Gram matrix is chosen. At step  104 , general processing is being done on the training data to identity the extreme feature vectors. At step  106 , general processing is being done on the extreme points to obtain scale factors for the extreme vectors. At step  108 , general processing is being done to produce the optimal linear classification system. 
     A computer-implemented, optimal linear classification system is obtained by solving the inequality constrained optimization problem: 
       minΨ(τ)=∥τ∥ 2 /2+ C/ 2Σ i=1   N ξ i   2   , s.t. y   i ( x   i   T τ+τ 0 )≥1−ξ i , ξ i ≥0, i= 1 ,   (1.54)
 
     The strong dual solution of Eq. (1.54) is obtained by solving a dual optimization problem: 
     
       
         
           
             
               
                 
                   
                     
                       max 
                        
                       
                           
                       
                        
                       Ξ 
                        
                       
                           
                       
                        
                       
                         ( 
                         ψ 
                         ) 
                       
                     
                     = 
                     
                       
                         
                           ∑ 
                           
                             i 
                             = 
                             1 
                           
                           N 
                         
                          
                         
                           ψ 
                           i 
                         
                       
                       - 
                       
                         
                           ∑ 
                           
                             i 
                             , 
                             
                               j 
                               = 
                               1 
                             
                           
                           N 
                         
                          
                         
                           
                             ψ 
                             i 
                           
                            
                           
                             ψ 
                             j 
                           
                            
                           
                             y 
                             i 
                           
                            
                           
                             y 
                             j 
                           
                            
                           
                               
                           
                            
                           
                             
                               
                                 
                                   x 
                                   i 
                                   T 
                                 
                                  
                                 
                                   x 
                                   j 
                                 
                               
                               + 
                               
                                 
                                   δ 
                                   ij 
                                 
                                 / 
                                 C 
                               
                             
                             2 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   1.55 
                   ) 
                 
               
             
           
         
       
     
     which is subject to the algebraic constraints Σ i=1 Ψ i y i =0, and Ψ i ≥0, where δ ij  is the Kronecker δ defined as unity for i=j and 0 otherwise. Equation (1.55) is a quadratic programming problem that can be written in vector notation by letting 
     
       
         
           
             
                 
             
              
             
               
                 Q 
                  
                 
                   = 
                   Δ 
                 
                  
                 
                   
                     ? 
                   
                    
                   
                       
                   
                    
                   
                     : 
                     T 
                   
                    
                   
                       
                   
                    
                   
                     and 
                      
                     
                         
                     
                      
                     X 
                      
                     
                       
                         = 
                         
                           ? 
                         
                       
                       Δ 
                     
                      
                     
                       ? 
                     
                   
                 
               
               , 
               
                 
 
               
                
               
                 
                   ? 
                 
                  
                 
                   indicates text missing or illegible when filed 
                 
               
             
           
         
       
     
     where D is an N×N diagonal matrix of training labels (class membership statistics) y i  and the N×d data matrix X is 
     
       
         
           
             
                 
             
              
             
               X 
               = 
               
                 ( 
                 
                   
                     
                       
                         
                           
                             
                               x 
                               1 
                             
                             , 
                           
                         
                         
                           
                             
                               x 
                               2 
                             
                             , 
                           
                         
                         
                           … 
                         
                       
                     
                      
                     
                       ? 
                     
                   
                   , 
                   
                     
 
                   
                    
                   
                     
                       ? 
                     
                      
                     
                       indicates text missing or illegible when filed 
                     
                   
                 
               
             
           
         
       
     
     The matrix version of the Lagrangian dual problem: 
     
       
         
           
             
               max 
                
               
                   
               
                
               Ξ 
                
               
                   
               
                
               
                 ( 
                 ψ 
                 ) 
               
             
             = 
             
               
                 
                   1 
                   T 
                 
                  
                 ψ 
               
               - 
               
                 
                   
                     ψ 
                     T 
                   
                    
                   Q 
                    
                   
                       
                   
                    
                   ψ 
                 
                 2 
               
             
           
         
       
     
     is subject to the constraints Ψ T y=0 and Ψ i ≥0. 
     In order to solve Eq. (1.54), values for the parameters ξ i  and C must be properly specified. 
     For N training vectors of dimension d, where d&lt;N, all of the regularization parameters {ξ i } i=1   N  in Eq. (1.54) and all of its derivatives are set equal to a very small value: ξ i =ξ&lt;&lt;. The regularization constant C is set equal to 
     
       
         
           
             
               
                 1 
                 ξ 
               
               : 
               C 
             
             = 
             
               
                 1 
                 ξ 
               
               . 
             
           
         
       
     
     For N training vectors of dimension d, where N&lt;d, all of the regularization parameters {ξ i } i=1   N  in Eq. (1.54) and all of its derivatives are set equal to zero: ξ i ξ=0. The regularization constant C is set equal to infinity: C=∞. 
     Solving Eq. (1.55) produces a principal eigenvector Ψ of N parameters. A linear discriminant function 
         D ( x )=τ T   x+τ   0  
 
     is formed by setting 
       τ=X T Ψ,
 
     where X≙, D y  is an N×N diagonal matrix of training labels (class membership statistics) y i  and X is an N×d data matrix 
     
       
         
           
             
                 
             
              
             
               X 
               = 
               
                 ( 
                 
                   
                     
                       
                         
                           
                             x 
                             1 
                           
                           , 
                         
                       
                       
                         
                           
                             x 
                             2 
                           
                           , 
                         
                       
                       
                         … 
                       
                     
                   
                    
                   
                     ? 
                   
                    
                   
                     
 
                   
                    
                   
                     ? 
                   
                    
                   
                     indicates text missing or illegible when filed 
                   
                 
               
             
           
         
       
     
     and by setting 
       τ 0 =Σ i=1   l   y   i (1−ξ i )−Σ i=1   l   x   i*   T   τ,  
 
     where the vector x i*  is correlated with Ψ 1* &gt;0. 
     A linear decision function sign(D(x)) is formed by the vector expression 
       (sign( D ( x ) τ x+τ   0 ),
 
     where 
     
       
         
           
             
                 
             
              
             
               
                 sign 
                  
                 
                   ( 
                   x 
                   ) 
                 
               
                
                
               
                 ? 
               
             
           
         
       
       
         
           
             
               ? 
             
              
             
               indicates text missing or illegible when filed 
             
           
         
       
     
     for x≠0. 
     Equal numbers of training examples are not absolutely necessary for optimal estimates of linear decision boundaries. Even so, the number of training examples from each of the pattern classes should be reasonably balanced with each other. Therefore, it is recommended, but is not absolutely necessary that Eq. (1.55) be applied to equal numbers of training examples from each pattern class. 
     Linear eigenlocus transforms involve solving variants of the inequality constrained optimization problem for linear kernel support vector machines (SVMs): Software for linear eigenlocus transforms can be obtained by using software packages that solve quadratic programming problems, or via LIBSVM (A Library for Support Vector Machines), SVMlight (an implementation of Support Vector Machines (SVMs) in C), or MATLAB SVM toolboxes.