Abstract:
Given the integral Z:=         ƒ(t)g(t)dv(t) of the product of two functions ƒ and g defined on a space          , a pivot function r:                               +  is optimized to minimize the bound defined by the inequality 
     
       
         
           
             
                 
             
              
             
               Z 
               ≤ 
               
                 
                   
                     ( 
                     
                       
                         ∫ 
                          
                       
                        
                       
                         
                           
                             f 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                           p 
                         
                          
                         
                           
                             r 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                           p 
                         
                          
                         
                            
                           
                             v 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                       
                     
                     ) 
                   
                   
                     1 
                     p 
                   
                 
                  
                 
                   
                     ( 
                     
                       
                         ∫ 
                          
                       
                        
                       
                         
                           
                             g 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                           q 
                         
                          
                         
                           
                             r 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                           
                             - 
                             q 
                           
                         
                          
                         
                            
                           
                             v 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                       
                     
                     ) 
                   
                   
                     1 
                     q 
                   
                 
               
             
           
         
       
     
     where v is a measure on the space          , p≧1 and 
     
       
         
           
             
               
                 1 
                 p 
               
               + 
               
                 1 
                 q 
               
             
             = 
             1 
           
         
       
     
     to determine an optimized pivot function r opt (t). The product ƒg may be evaluated as ƒg=ƒ p r opt   p  or ƒg=g q r opt   −q . The integral Z:=         ƒ(t)g(t)dv(t) may e evaluated as the product 
     
       
         
           
             
               
                 ( 
                 
                   
                     ∫ 
                      
                   
                    
                   
                     
                       
                         f 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       p 
                     
                      
                     
                       
                         r 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       p 
                     
                      
                     
                        
                       
                         v 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                 
                 ) 
               
               
                 1 
                 p 
               
             
              
             
               
                 ( 
                 
                   
                     ∫ 
                      
                   
                    
                   
                     
                       
                         g 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       q 
                     
                      
                     
                       
                         r 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       
                         - 
                         q 
                       
                     
                      
                     
                        
                       
                         v 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                 
                 ) 
               
               
                 1 
                 q 
               
             
           
         
       
     
     with r(t)=r opt (t). The method is suitably performed by an electronic data processing device. More generally, an integral Z:=         Π k=1   K ƒ k (t)dv(t) where K≧2 may be evaluated by optimizing a pivot function q(t)=Π k=1   K q k (t) to minimize a bound defined by the inequality 
     
       
         
           
             
               
                 ∫ 
                  
               
                
               
                 
                   ∏ 
                   
                     
                       k 
                       ′ 
                     
                     = 
                     1 
                   
                   K 
                 
                  
                 
                   
                     
                       f 
                       k 
                     
                      
                     
                       ( 
                       t 
                       ) 
                     
                   
                    
                   
                      
                     
                       v 
                        
                       
                         ( 
                         t 
                         ) 
                       
                     
                   
                 
               
             
             ≤ 
             
               
                 ∏ 
                 
                   k 
                   = 
                   1 
                 
                 K 
               
                
               
                 
                   ( 
                   
                     
                       ∫ 
                        
                     
                      
                     
                       
                         
                           ( 
                           
                             
                               
                                 f 
                                 k 
                               
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                             
                               
                                 q 
                                 k 
                               
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                           
                           ) 
                         
                         
                           1 
                           
                             ρ 
                             k 
                           
                         
                       
                        
                       
                         
                           ∏ 
                           
                             
                               k 
                               ′ 
                             
                             = 
                             1 
                           
                           K 
                         
                          
                         
                           
                             
                               q 
                               
                                 k 
                                 ′ 
                               
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                            
                           
                              
                             
                               v 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                   ) 
                 
                 
                   ρ 
                   k 
                 
               
             
           
         
       
     
     where ρ k ≧0 for k=1, . . . , K and Σ k=1   K ρ k =1 to determine an optimized pivot function q opt (t).

Description:
BACKGROUND 
       [0001]    The following relates to the information processing arts, computational arts, numerical integration arts including inference arts such as Bayesian inference, and related arts. 
         [0002]    Tasks such as machine learning, modeling, and so forth commonly involve evaluation of an intractable sum or integral. For example, in Bayesian inference a posterior predictive distribution of the form: 
         [0000]        p ( x |,α)=∫ Θ   p ( x |θ) p (θ| X ,α) dθ 
 
         [0000]    is evaluated, where θ is a parameters vector defined on the parameter space Θ, α is a hyperparameter vector, X is a training data set (also called the “evidence”), and x is a data point to be predicted. Computing the posterior predictive distribution requires evaluating an integral of a product of two functions, namely p(x|θ) and p(θ|X,α), which is often not possible to perform analytically. 
         [0003]    Numerical approximations methods have been derived to evaluate such integrals. Sampling techniques such as Gibbs sampling are sometimes used to explore the space and compute sufficient statistics to estimate such an integral. However, sampling techniques are stochastic by nature and typically do not provide guaranteed convergence, but only convergence with a high probability. Other approaches minimize an objective function, such as a loss function, a negative log-likelihood or a energy criterion. 
         [0004]    In some approaches, an upper (or lower) bounding function is defined such that the integral is known to be less than (or greater than) the bound everywhere over the region of interest. In such approaches, the “tightness” of the bound determines accuracy of (or at least an error bar for) the estimated integral. A loose bound which has large gaps between the bounding function and the true integral over the region will typically result in a large overestimate of the integral (for an upper bound). A tight bound of the bounding function to the true integral over the region provides accuracy and a small error bar. 
         [0005]    Accuracy can be improved by partitioning the region over which the integral is to be computed and defining an upper (or lower) bound for each partition. Bouchard et al., U.S. Pat. No. 8,190,550 titled “Split Variational Inference” discloses some such approaches, including a disclosed embodiment which partitions a region of interest into a plurality of soft bin regions that span the region of interest, estimates an integral over each soft bin region of a function defined over the region of interest, and outputs a value equal to or derived from the sum of the estimated integrals over the soft bin regions spanning the region of interest. 
       BRIEF DESCRIPTION 
       [0006]    In some embodiments disclosed herein, a non-transitory storage medium storing instructions readable and executable by an electronic data processing device to perform a method of evaluating an integral Z:=         Π k=1   K ƒ k (t)dv(t) where K≧2, v is a measure on a space           and ƒ k (t), k=1, . . . , K are functions defined in the space          , the method comprising operations including optimizing a pivot function q(t)=Π k=1   K q k (t) to minimize a bound defined by the inequality          Π k′=1   K ƒ k (t)dv(t)≧Π k=1   K   
         [0000]    
       
         
           
             
               ( 
               
                 
                   ∫ 
                    
                 
                  
                 
                   
                     
                       ( 
                       
                         
                           
                             f 
                             k 
                           
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                         
                           
                             q 
                             k 
                           
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                       
                       ) 
                     
                     
                       1 
                       
                         ρ 
                         k 
                       
                     
                   
                    
                   
                     
                       ∏ 
                       
                         
                           k 
                           ′ 
                         
                         = 
                         1 
                       
                       K 
                     
                      
                     
                       
                         
                           q 
                           
                             k 
                             ′ 
                           
                         
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                        
                       
                          
                         
                           v 
                            
                           
                             ( 
                             x 
                             ) 
                           
                         
                       
                     
                   
                 
               
               ) 
             
             
               ρ 
               k 
             
           
         
       
     
         [0000]    where ρ k ≧0 for k=1, . . . , K and Σ k=1   K ρ k =1 to determine an optimized pivot function q opt (t) and optimized values ρ 1     opt   , . . . , ρ K     opt   , and evaluating Z as the product 
         [0000]    
       
         
           
             
               ∏ 
               
                 k 
                 = 
                 1 
               
               K 
             
              
             
               
                 ( 
                 
                   
                     ∫ 
                      
                   
                    
                   
                     
                       
                         ( 
                         
                           
                             
                               f 
                               k 
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                           
                             
                               q 
                               k 
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                         ) 
                       
                       
                         1 
                         
                           ρ 
                           k 
                         
                       
                     
                      
                     
                       
                         ∏ 
                         
                           
                             k 
                             ′ 
                           
                           = 
                           1 
                         
                         K 
                       
                        
                       
                         
                           
                             q 
                             
                               k 
                               ′ 
                             
                           
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                          
                         
                            
                           
                             v 
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                         
                       
                     
                   
                 
                 ) 
               
               
                 ρ 
                 k 
               
             
           
         
       
     
         [0000]    with q(t)=q opt (t) and ρ k =ρ k     opt    for k=1, . . . , K. 
         [0007]    In some embodiments disclosed herein, an apparatus comprises a non-transitory storage medium as set forth in the immediately preceding paragraph and an electronic data processing device configured to read and execute the instructions stored on the non-transitory storage medium. 
         [0008]    In some embodiments disclosed herein, a method operating on two functions ƒ and g each mapping from a space           into            +  comprises optimizing a pivot function r:                               +  to minimize a bound defined by the inequality Z≦ 
         [0000]    
       
         
           
             
               
                 ( 
                 
                   
                     ∫ 
                     T 
                   
                    
                   
                     
                       
                         f 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       p 
                     
                      
                     
                       
                         r 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       p 
                     
                      
                     
                         
                     
                      
                     
                        
                       
                         v 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                 
                 ) 
               
               
                 1 
                 p 
               
             
              
             
               
                 ( 
                 
                   
                     ∫ 
                     T 
                   
                    
                   
                     
                       
                         g 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       q 
                     
                      
                     
                       
                         r 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       
                         - 
                         q 
                       
                     
                      
                     
                         
                     
                      
                     
                        
                       
                         v 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                 
                 ) 
               
               
                 1 
                 q 
               
             
           
         
       
     
         [0000]    where v is a measure on the space           and p≧1 and 
         [0000]    
       
         
           
             
               
                 1 
                 p 
               
               + 
               
                 1 
                 q 
               
             
             = 
             1 
           
         
       
     
         [0000]    to determine an optimized pivot function r opt (t). The method may further comprise evaluating the product ƒg as one of ƒg=ƒ p r opt   p  and ƒg=g q r opt   −q . The method may additionally or alternatively further comprise evaluating the integral Z:=         ƒ(t)g(t)dv(t) as the product 
         [0000]    
       
         
           
             
               
                 ( 
                 
                   
                     ∫ 
                     T 
                   
                    
                   
                     
                       
                         f 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       p 
                     
                      
                     
                       
                         r 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       p 
                     
                      
                     
                         
                     
                      
                     
                        
                       
                         v 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                 
                 ) 
               
               
                 1 
                 p 
               
             
              
             
               
                 ( 
                 
                   
                     ∫ 
                     T 
                   
                    
                   
                     
                       
                         g 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       q 
                     
                      
                     
                       
                         r 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                       
                         - 
                         q 
                       
                     
                      
                     
                         
                     
                      
                     
                        
                       
                         v 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                 
                 ) 
               
               
                 1 
                 q 
               
             
           
         
       
     
         [0000]    with r(t)=r opt (t). The method is suitably performed by an electronic data processing device. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]      FIG. 1  diagrammatically shows an electronic data processing device configured to perform a variational Hölder&#39;s inference technique disclosed herein. 
           [0010]      FIGS. 2 and 3  plot examples of truncated Gaussian integration performed using the variational Hölder&#39;s inference technique. 
       
    
    
     DETAILED DESCRIPTION 
       [0011]    In existing integral evaluation approaches employing an upper (or lower) bound, a problem exists in that substantial computer resources are required to evaluate the integral. The evaluation of the integral is computationally expensive due to the need to determine a tight bound. In approaches employing partitioning of the region of interest, the number of computations is multiplied by the number of partitions. 
         [0012]    Disclosed herein are improved systems and methods providing various benefits, such as: reduced computer resources required for evaluating the integral; more accurate integration; and applicability to a wide range of classes of integrals in which the integrand can be formulated as the product of two or more functions each of which is individually readily integrable. The disclosed systems and methods additionally or alternatively may be employed to estimate the normalized integrand distribution. 
         [0013]    The disclosed approaches for computationally efficient integral evaluation are based on Hölder&#39;s inequality, which was originally promulgated by Otto Ludwig Hölder (Dec. 22, 1859-Aug. 29, 1937). Hölder&#39;s inequality is suitably defined as follows. Denote by ƒ and g two functions mapping from a space           into            + . For example, in a probabilistic inference problem, the functions ƒ and g typically correspond to the prior and the likelihood term of the probabilistic inference problem. The integral Z of a product ƒg of these functions is to be evaluated, that is, Z:=∫         ƒ(t)g(t)dv(t), where v is a measure on the space          . It is assumed that this integral Z is intractable (or at least is not readily evaluated analytically). However, the individual integrals are readily integrated, that is, there exist efficient methods to compute the individual integrals ∫         ƒdv and ∫         gdv. A known upper bound on the integral Z is the Hölder inequality, which can be written as follows: 
         [0000]    
       
         
           
             
               
                 
                   Z 
                   ≤ 
                   
                     
                       
                         ( 
                         
                           
                             ∫ 
                             T 
                           
                            
                           
                             
                               
                                 f 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                               p 
                             
                              
                             
                                 
                             
                              
                             
                                
                               
                                 v 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                             
                           
                         
                         ) 
                       
                       
                         1 
                         p 
                       
                     
                      
                     
                       
                         ( 
                         
                           
                             ∫ 
                             T 
                           
                            
                           
                             
                               
                                 g 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                               q 
                             
                              
                             
                                 
                             
                              
                             
                                
                               
                                 v 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                             
                           
                         
                         ) 
                       
                       
                         1 
                         q 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
         [0000]    where p is a scalar and p≧1 and q is defined in terms of p as 
         [0000]    
       
         
           
             
               
                 1 
                 p 
               
               + 
               
                 1 
                 q 
               
             
             = 
             1. 
           
         
       
     
         [0014]    Hölder&#39;s inequality (e.g. as expressed in Equation (1)) might appear to be a good candidate for evaluation of integrals such as integrals implicated in evaluating a posterior predictive distribution in Bayesian inference. However, in practice Hölder&#39;s inequality is often not useful for such purposes because the bound is too loose. Hölder&#39;s inequality provides a tight bound when g=ƒ p-1  (or equivalently when ƒ=g q-1 ), but in practical applications such as Bayesian inference this equality cannot be achieved for any value of p, and the bound is loose. Accordingly, Hölder&#39;s inequality is not of practical use in evaluating many integrals of the form Z:=         ƒ(t)g(t)dv(t). 
         [0015]    Disclosed herein are integration approaches that are implementable with reduced computer resources (i.e. are less computationally expensive) as compared with other existing integration approaches. The disclosed integration approaches modify Hölder&#39;s inequality to incorporate a pivot function r as follows. For any function r:                               +  the following inequality holds: 
         [0000]    
       
         
           
             
               
                 
                   Z 
                   ≤ 
                   
                     
                       
                         ( 
                         
                           
                             ∫ 
                             T 
                           
                            
                           
                             
                               
                                 f 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                               p 
                             
                              
                             
                               
                                 r 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                               p 
                             
                              
                             
                                 
                             
                              
                             
                                
                               
                                 v 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                             
                           
                         
                         ) 
                       
                       
                         1 
                         p 
                       
                     
                      
                     
                       
                         
                           ( 
                           
                             
                               ∫ 
                               T 
                             
                              
                             
                               
                                 
                                   g 
                                    
                                   
                                     ( 
                                     t 
                                     ) 
                                   
                                 
                                 q 
                               
                                
                               
                                 
                                   r 
                                    
                                   
                                     ( 
                                     t 
                                     ) 
                                   
                                 
                                 
                                   - 
                                   q 
                                 
                               
                                
                               
                                   
                               
                                
                               
                                  
                                 
                                   v 
                                    
                                   
                                     ( 
                                     t 
                                     ) 
                                   
                                 
                               
                             
                           
                           ) 
                         
                         
                           1 
                           q 
                         
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
         [0016]    This proposition is suitably demonstrated as follows. Replace ƒ by ƒr and similarly replace g by 
         [0000]    
       
         
           
             g 
             r 
           
         
       
     
         [0000]    in Hölder&#39;s inequality as presented in Equation (1). The expression for the integral Z is then written as: 
         [0000]    
       
         
           
             
               
                 
                   Z 
                   = 
                   
                     
                       ∫ 
                       T 
                     
                      
                     
                       
                         ( 
                         
                           
                             f 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                            
                           
                             r 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                         ) 
                       
                        
                       
                         
                           g 
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                         
                           r 
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                       
                        
                       
                          
                         
                           v 
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
         [0000]    Note that the integral Z of Equation (3) is exactly equal to the original integral Z:=         ƒ(t)g(t)dv(t) since the r(t) and 
         [0000]    
       
         
           
             1 
             
               r 
                
               
                 ( 
                 t 
                 ) 
               
             
           
         
       
     
         [0000]    terms of Equation (3) cancel. Applying Hölder&#39;s inequality (Equation (1)) to the integral of Equation (3) yields: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       ∫ 
                       T 
                     
                      
                     
                       
                         ( 
                         
                           
                             f 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                            
                           
                             r 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                         ) 
                       
                        
                       
                         
                           g 
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                         
                           r 
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                       
                        
                       
                          
                         
                           v 
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                       
                     
                   
                   ≤ 
                   
                     
                       
                         ( 
                         
                           
                             ∫ 
                             T 
                           
                            
                           
                             
                               
                                 f 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                               p 
                             
                              
                             
                               
                                 r 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                               p 
                             
                              
                             
                                
                               
                                 v 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                             
                           
                         
                         ) 
                       
                       
                         1 
                         p 
                       
                     
                      
                     
                       
                         ( 
                         
                           
                             ∫ 
                             T 
                           
                            
                           
                             
                               
                                 g 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                               q 
                             
                              
                             
                               
                                 r 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                               
                                 - 
                                 q 
                               
                             
                              
                             
                                
                               
                                 v 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                             
                           
                         
                         ) 
                       
                       
                         1 
                         q 
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
         [0000]    which is Equation (2) since the left-hand size is exactly equal to Z. 
         [0017]    Equation (2) is referred to herein as a “variational Hölder&#39;s inequality”, and serves as the basis for a variational inference algorithm disclosed in the following which comprises minimizing the bound defined by Equation (2) with respect to the pivot function r. 
         [0018]    As an initial point, it is observed herein that the upper bound defined by the variational Hölder&#39;s inequality (Equation (2)) actually becomes exact if r is chosen to be proportional to r* which is defined as follows: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         r 
                         * 
                       
                        
                       
                         ( 
                         t 
                         ) 
                       
                     
                     := 
                     
                       
                         
                           
                             g 
                             
                               1 
                               p 
                             
                           
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                         
                           
                             f 
                             
                               1 
                               q 
                             
                           
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                       
                        
                       
                           
                       
                        
                       
                         ∀ 
                         
                           t 
                           ∈ 
                           T 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
         [0000]    That is, for r=cr* with c being a positive scalar and r* as defined in Equation (5), the variational Hölder&#39;s inequality of Equation (2) becomes an equality. This can be demonstrated as follows. For r=cr*, the first integral of the right-hand side of Equation (2) is 
         [0000]    
       
         
           
             cZ 
             
               1 
               p 
             
           
         
       
     
         [0000]    because it integrates 
         [0000]    
       
         
           
             
               
                 f 
                 p 
               
                
               
                 ( 
                 
                   cr 
                   * 
                 
                 ) 
               
             
             = 
             
               
                 
                   c 
                   p 
                 
                  
                 
                   f 
                   
                     p 
                     - 
                     
                       p 
                       q 
                     
                   
                 
                  
                 g 
               
               = 
               
                 
                   
                     c 
                     p 
                   
                    
                   
                     f 
                     
                       p 
                       - 
                       
                         p 
                          
                         
                           ( 
                           
                             1 
                             - 
                             
                               1 
                               p 
                             
                           
                           ) 
                         
                       
                     
                   
                    
                   g 
                 
                 = 
                 
                   
                     c 
                     p 
                   
                    
                   
                     fg 
                     . 
                   
                 
               
             
           
         
       
     
         [0000]    Similarly, the second integral of the right-hand side of Equation (2) is 
         [0000]    
       
         
           
             
               1 
               c 
             
              
             
               Z 
               
                 1 
                 q 
               
             
           
         
       
     
         [0000]    because 
         [0000]    
       
         
           
             
               
                 
                   g 
                   q 
                 
                  
                 
                   ( 
                   
                     cr 
                     * 
                   
                   ) 
                 
               
               
                 - 
                 q 
               
             
             = 
             
               
                 
                   1 
                   
                     c 
                     q 
                   
                 
                  
                 
                   g 
                   
                     q 
                     - 
                     
                       q 
                        
                       
                         ( 
                         
                           1 
                           - 
                           
                             1 
                             q 
                           
                         
                         ) 
                       
                     
                   
                 
                  
                 
                   f 
                   
                     q 
                     p 
                   
                 
               
               = 
               
                 
                   1 
                   
                     c 
                     q 
                   
                 
                  
                 
                   fg 
                   . 
                 
               
             
           
         
       
     
         [0000]    The product of these integrals is 
         [0000]    
       
         
           
             
               
                 cZ 
                 
                   1 
                   p 
                 
               
               × 
               
                 1 
                 c 
               
                
               
                 Z 
                 
                   1 
                   q 
                 
               
             
             = 
             
               Z 
               . 
             
           
         
       
     
         [0019]    It is also noted that for the case of p=q=2 (which satisfies the conditions p≧1 and 
         [0000]    
       
         
           
             
               
                 1 
                 p 
               
               + 
               
                 1 
                 q 
               
             
             = 
             1 
           
         
       
     
         [0000]    required for Hölder&#39;s inequality), the function r* given in Equation (5) corresponds to the square root of the ratio between the two factors ƒ and g, that is, 
         [0000]    
       
         
           
             
               r 
               * 
             
             = 
             
               
                 
                   g 
                   f 
                 
               
               . 
             
           
         
       
     
         [0020]    Equation (2) using a pivot function r=cr* with r* given in Equation (5) thus yields an exact evaluation of Z. In practice however, the integrals in Equation (2) formulated using r* as given in Equation (5) are typically not easier to evaluate than the original integral Z. 
         [0021]    As further disclosed herein, computationally efficient evaluation of the integral Z using the variational Hölder&#39;s inequality of Equation (2) is achievable by defining a family of variational functions          ={r(•,θ),θεΘ} parameterized by θεΘ for which the integrals of the form: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         I 
                         f 
                       
                        
                       
                         ( 
                         
                           θ 
                           ; 
                           p 
                         
                         ) 
                       
                     
                     := 
                     
                       
                         ∫ 
                          
                       
                        
                       
                         
                           
                             ( 
                             
                               
                                 f 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                                
                               
                                 r 
                                  
                                 
                                   ( 
                                   
                                     t 
                                     ; 
                                     θ 
                                   
                                   ) 
                                 
                               
                             
                             ) 
                           
                           p 
                         
                          
                         
                            
                           
                             v 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                          
                         
                             
                         
                          
                         and 
                       
                     
                   
                    
                   
                       
                   
                    
                   
                     
 
                   
                    
                   
                     
                       
                         I 
                         g 
                       
                        
                       
                         ( 
                         
                           θ 
                           ; 
                           q 
                         
                         ) 
                       
                     
                     := 
                     
                       
                         ∫ 
                          
                       
                        
                       
                         
                           
                             ( 
                             
                               
                                 g 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                               
                                 r 
                                  
                                 
                                   ( 
                                   
                                     t 
                                     ; 
                                     θ 
                                   
                                   ) 
                                 
                               
                             
                             ) 
                           
                           q 
                         
                          
                         
                            
                           
                             v 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
     
         [0000]    are tractable. Choosing the best function r(•,θ) in the family           is disclosed herein to significantly tighten the bound as compared with Hölder inequality (Equation (1)). In one suitable optimization formulation, the upper bound Ī:Θ                     +  is minimized with respect to θ, p and q (where p and q are not independent but rather are constrained by p≧1 and 
         [0000]    
       
         
           
             
               
                 
                   
                     1 
                     p 
                   
                   + 
                   
                     1 
                     q 
                   
                 
                 = 
                 1 
               
               ) 
             
             , 
           
         
       
     
         [0000]    where: 
         [0000]    
       
         
           
             
               
                 
                   
                     log 
                      
                     
                         
                     
                      
                     
                       I 
                        
                       
                         ( 
                         θ 
                         ) 
                       
                     
                   
                   := 
                   
                     
                       
                         1 
                         p 
                       
                        
                       log 
                        
                       
                           
                       
                        
                       
                         
                           I 
                           f 
                         
                          
                         
                           ( 
                           
                             θ 
                             ; 
                             p 
                           
                           ) 
                         
                       
                     
                     + 
                     
                       
                         1 
                         q 
                       
                        
                       log 
                        
                       
                           
                       
                        
                       
                         
                           I 
                           g 
                         
                          
                         
                           ( 
                           
                             θ 
                             ; 
                             q 
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
         [0000]    For variational functions r(•,θ) that are differentiable with respect to θ, the optimal parameters θ can be obtained using an optimization algorithm such as (by way of illustrative example) a standard gradient descent algorithm. 
         [0022]    In some embodiments, the pivot functions r are chosen to be log-linear functions. Defining the feature function φ:                   Θ and assuming the space Θ is equipped with the dot-product operation, the following class of pivot functions is suitably employed: 
         [0000]        r ( t ,θ)= e   &lt;θ,φ(t)&gt; ,θεΘ  (8)
 
         [0000]    For log-linear functions r as defined in Equation (8), the bound Ī is log-convex and differentiable in θ. This can be shown as follows. The factors log I ƒ  (θ;p) and log I g (θ; q) are log-sum-exp functions (logarithm of the integral of an exponential function) and therefore are convex in θ. Hence log Ī(θ)=log I ƒ (θ;p)+log I g (θ; q) is a sum of two functions which are convex in θ, so that Ī(θ) is log-convex. Because of this log-convex property, gradient descent algorithms are less subject to local minima. In practice, this means that complex initialization procedures are not required. In addition, the optimization is not restricted to generic gradient descent algorithms, but rather other optimization methods, tailored to specific problems can alternatively be employed. Furthermore, various available theoretical tools pertaining to convex optimization may be employed to analyze convergence properties. For example, it is expected that the objective function of Equation (7) with a log-convex pivot function as per Equation (8) can be solved in polynomial time using standard gradient descent algorithms. 
         [0023]    Optimization of Equation (7) with a pivot of the form r(t,θ):=e θ     T     φ(t)  using a gradient descent algorithm may, for example, employ the following. Equation (7) may be written as log I h (θ;p):=log(         h(t) p e pθ     T     φ(t) dv(t)), and its gradient ∇ θ  log I h (θ;p) as 
         [0000]    
       
         
           
             
               
                 
                   ∇ 
                   θ 
                 
                  
                 log 
               
                
               
                   
               
                
               
                 
                   I 
                   h 
                 
                  
                 
                   ( 
                   
                     θ 
                     ; 
                     p 
                   
                   ) 
                 
               
             
             := 
             
               
                 p 
                 
                   
                     I 
                     h 
                   
                    
                   
                     ( 
                     
                       θ 
                       ; 
                       p 
                     
                     ) 
                   
                 
               
                
               
                 
                   ∫ 
                    
                 
                  
                 
                   
                     φ 
                      
                     
                       ( 
                       t 
                       ) 
                     
                   
                    
                   
                     
                       h 
                        
                       
                         ( 
                         t 
                         ) 
                       
                     
                     p 
                   
                    
                   
                      
                     
                       p 
                        
                       
                           
                       
                        
                       
                         θ 
                         T 
                       
                        
                       
                         φ 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                    
                   
                      
                     
                       v 
                        
                       
                         ( 
                         t 
                         ) 
                       
                     
                   
                 
               
             
           
         
       
     
         [0000]    which is p           h,θ,p [φ(t)]. Similarly, the gradient ∇ p  log I h (θ;p) may be written as 
         [0000]    
       
         
           
             
               
                 1 
                 
                   
                     I 
                     h 
                   
                    
                   
                     ( 
                     
                       θ 
                       ; 
                       p 
                     
                     ) 
                   
                 
               
                
               
                 θ 
                 T 
               
                
               
                 
                   ∫ 
                    
                 
                  
                 
                   
                     φ 
                      
                     
                       ( 
                       t 
                       ) 
                     
                   
                    
                   
                     
                       h 
                        
                       
                         ( 
                         t 
                         ) 
                       
                     
                     p 
                   
                    
                   
                      
                     
                       p 
                        
                       
                           
                       
                        
                       
                         θ 
                         T 
                       
                        
                       
                         φ 
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                   
                    
                   
                      
                     
                       v 
                        
                       
                         ( 
                         t 
                         ) 
                       
                     
                   
                 
               
             
             + 
             
               
                 1 
                 
                   
                     I 
                     h 
                   
                    
                   
                     ( 
                     
                       θ 
                       ; 
                       p 
                     
                     ) 
                   
                 
               
                
               
                 ∫ 
                  
               
             
           
         
       
     
         [0000]    log(h(t))h(t) p e pθ     T     φ(t) dv(t) which is θ T             h,θ,p [φ(t)]+           h,θ,p [log h(t)]. 
         [0024]    With reference to  FIG. 1 , a variational inference system  10  performs variational inference to compute an integral  12  (which is the integral Z given in Equation (1)) in which the integrand is a product of two functions ƒ  14  and g  16  by applying the variational Hölder&#39;s inequality  20  (which is Equation (2)). An upper bound minimization engine  22  optimizes an objective function  24  (which is the objective function given in Equation (7)) respective to pivot function parameters  26  (which are suitably the parameters θ of the family of variational functions          ={r(•,θ),θεθ}) and a variational Hölder&#39;s inequality parameter  28  (which is the scalar p in the illustrative system of  FIG. 1 ; alternatively, the parameter may be q since p and q are related by 
         [0000]    
       
         
           
             
               
                 
                   
                     1 
                     p 
                   
                   + 
                   
                     1 
                     q 
                   
                 
                 = 
                 1 
               
               ) 
             
             . 
           
         
       
     
         [0025]    The upper bound minimization engine  22  employs a suitable optimization algorithm, such as a gradient descent algorithm, to optimize the objective function  24  respective to the pivot function parameters  26  and Hölder&#39;s inequality parameter  28 . Other optimization algorithms may be used. By way of an additional illustrative example, in some embodiments a modified gradient descent is employed in which only the pivot function parameters  26  are subject to optimization by the gradient descent algorithm while the Hölder&#39;s inequality parameter  28  can be optimized using a grid approach (e.g., repeating the gradient descent optimization of pivot function parameters θ for each of a plurality of discrete values of p and choosing the value of p yielding the best gradient descent result). 
         [0026]    When employing log-linear functions r as defined in Equation (8), or another pivot function family for which the bound Ī is log-convex, it is ensured that a global upper bound minimum can be identified. Nonetheless, it is to be understood that as used herein the term “minimize”, “optimize” and similar phraseology encompasses iterative optimization techniques that are configured to settle on a local minimum that is not the global minimum, and/or that are configured to stop before reaching the precise global minimum. For example, the upper bound minimization engine  22  may employ an iterative optimization algorithm with stopping criteria that terminate the iterative process when the iteration-to-iteration change in the objective function value becomes less than a stopping threshold. 
         [0027]    The output of the upper bound minimization engine  22  is a set of optimized parameters  30  including optimized pivot function parameters θ opt  and optimized Hölder&#39;s inequality parameters p opt  and q opt . Equation (2) is then applied using these optimized values to compute the integral  32  according to 
         [0000]    
       
         
           
             Z 
             ≅ 
             
               
                 
                   ( 
                   
                     
                       ∫ 
                        
                     
                      
                     
                       
                         
                           f 
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                         
                           p 
                           opt 
                         
                       
                        
                       
                         
                           
                             r 
                             opt 
                           
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                         
                           p 
                           opt 
                         
                       
                        
                       
                          
                         
                           v 
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                       
                     
                   
                   ) 
                 
                 
                   1 
                   
                     p 
                     opt 
                   
                 
               
                
               
                 
                   
                     ( 
                     
                       
                         ∫ 
                          
                       
                        
                       
                         
                           
                             g 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                           
                             q 
                             opt 
                           
                         
                          
                         
                           
                             
                               r 
                               opt 
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                           
                             - 
                             
                               q 
                               opt 
                             
                           
                         
                          
                         
                            
                           
                             v 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                       
                     
                     ) 
                   
                   
                     1 
                     
                       q 
                       opt 
                     
                   
                 
                 . 
               
             
           
         
       
     
         [0028]    In addition to computing the integral of the distribution ƒ(t) g  (t) over the space           (that is, computing Z:=         ƒ(t)g(t)dv(t)), the foregoing approach is also optionally applied to compute the normalized distribution ƒ(t)g(t) itself. This is based on the recognition herein that obtaining an accurate upper bound to the log-partition is closely related to the fact that the approach provides a good approximation of the distribution and its moments. Hence, minimizing the variational Hölder inequality (Equation (2)) is also an efficient inference algorithm. To see this, the exact solution of Equation (5) is revisited. Recall that for r=cr* with c being a positive scalar and r* as defined in Equation (5), the variational Hölder&#39;s inequality of Equation (2) becomes an equality. In this case, the product ƒg can be approximated by ƒ p r p  or g q r −q  as: 
         [0000]      ƒ g=ƒ   p   r   p  or ƒ g=g   q   r   −q   (9)
 
         [0000]    To see this, it is observed that Equation (5) leads to 
         [0000]    
       
         
           
             r 
             = 
             
               
                 
                   g 
                   
                     1 
                     p 
                   
                 
                  
                 
                   f 
                   
                     - 
                     
                       1 
                       q 
                     
                   
                 
                  
                 
                     
                 
                  
                 for 
                  
                 
                     
                 
                  
                 r 
               
               = 
               
                 
                   cr 
                   * 
                 
                 . 
               
             
           
         
       
     
         [0000]    By raising this equation to power p the expression 
         [0000]    
       
         
           
             
               r 
               p 
             
             = 
             
               
                 gf 
                 
                   - 
                   
                     p 
                     q 
                   
                 
               
               = 
               
                 
                   ( 
                   gf 
                   ) 
                 
                 × 
                 
                   f 
                   
                     
                       - 
                       
                         p 
                         q 
                       
                     
                     - 
                     1 
                   
                 
               
             
           
         
       
     
         [0000]    is obtained, so that 
         [0000]    
       
         
           
             fg 
             = 
             
               
                 
                   r 
                   p 
                 
                  
                 
                   f 
                   
                     1 
                     + 
                     
                       p 
                       q 
                     
                   
                 
               
               = 
               
                 
                   r 
                   p 
                 
                  
                 
                   f 
                   p 
                 
               
             
           
         
       
     
         [0000]    because 
         [0000]    
       
         
           
             
               1 
               + 
               
                 p 
                 q 
               
             
             = 
             
               
                 1 
                 + 
                 
                   p 
                    
                   
                     ( 
                     
                       1 
                       - 
                       
                         1 
                         p 
                       
                     
                     ) 
                   
                 
               
               = 
               
                 p 
                 . 
               
             
           
         
       
     
       Similarly, 
       [0029]    
       
         
           
             
               
                 g 
                 q 
               
                
               
                 r 
                 
                   - 
                   q 
                 
               
             
             = 
             
               
                 fg 
                 
                   q 
                   - 
                   
                     q 
                     p 
                   
                 
               
               = 
               
                 
                   fg 
                   
                     q 
                      
                     
                       ( 
                       
                         1 
                         - 
                         
                           1 
                           p 
                         
                       
                       ) 
                     
                   
                 
                 = 
                 
                   fg 
                   . 
                 
               
             
           
         
       
     
         [0000]    While this holds exactly only for r=cr* for which the inequality of Equation (2) is an equality, the relations of Equation (9) hold at least approximately where the bound is tight but not exact, for example when using the optimized bound provided by the optimized parameters  30 . Thus, with reference again to  FIG. 1 , a normalized distribution  34 , that is, the normalized distribution ƒ(t)g(t), is suitably computed using Equation (9) to yield ƒ(t)g (t)≅ƒ p     opt   (t)r p     opt   (t) or alternatively ƒ(t)g(t)≅g q     opt   (t)r −q     opt   (t). 
         [0030]    The illustrative system of  FIG. 1  is suitably embodied by a computer  40  or other electronic data processing device programmed to perform an embodiment of the disclosed variational inference algorithm, and/or by a non-transitory storage medium storing instructions executable by the illustrative computer  40  or other electronic data processing device to perform the disclosed variational inference algorithm. The non-transitory storage medium may, for example, comprise a hard disk or other magnetic storage medium, an optical disk or other optical storage medium, a random access memory (RAM), read-only memory (ROM), flash memory or other electronic storage medium, various combinations thereof, or so forth. The variational inference algorithm may be logically configured in various ways, such as being configured as a library integration function receiving as input the functions  14 ,  16  and optionally other user-configurable inputs such as the choice of pivot function family  26 , a selection input that selects the output as the integral  32  or the normalized distribution  34 , or so forth. The variational inference algorithm may alternatively be logically configured as an integral component of a process utilizing the integral, such as being the integral evaluation component of a Bayesian inference program or library function. 
         [0031]    Some illustrative applications of the integral evaluation system of  FIG. 1  and variants thereof are next described. 
         [0032]    One illustrative application is evaluation of the product of univariate factors with a Gaussian. Using notations previously set forth, the function ƒ(t)=Π i=1   n ƒ i (t i ) is defined, where each function ƒ i :                              is univariate. The function g(t) is also defined, with 
         [0000]    
       
         
           
             
               
                  
                 
                   - 
                   
                     1 
                     2 
                   
                 
               
                
               
                 t 
                 T 
               
                
               At 
             
             + 
             
               
                 b 
                 T 
               
                
               t 
             
           
         
       
     
         [0000]    where A is a definite symmetric n×n matrix and bε           n . The Lebesgue measure is used for the measure v. The integral to be evaluated is: 
         [0000]    
       
         
           
             
               
                 
                   I 
                   := 
                   
                     
                       ∫ 
                       
                         ℝ 
                         n 
                       
                     
                      
                     
                       
                         ∏ 
                         
                           i 
                           = 
                           1 
                         
                         n 
                       
                        
                       
                           
                       
                        
                       
                         
                           
                             f 
                             i 
                           
                            
                           
                             ( 
                             
                               t 
                               i 
                             
                             ) 
                           
                         
                          
                         
                            
                           
                             
                               
                                 - 
                                 
                                   1 
                                   2 
                                 
                               
                                
                               
                                 t 
                                 T 
                               
                                
                               At 
                             
                             + 
                             
                               b 
                               T 
                             
                           
                         
                          
                         
                            
                           t 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
         [0000]    This type of integral is used in certain machine learning tasks, and typically corresponds to the marginal data probability—that is, the evidence—of a linear regression model with known variance and sparse priors, n being the number of variables. Up to an affine change of variable to obtain orthogonal univariate factors, this integral corresponds also to the data evidence in a generalized linear model with Gaussian prior, where n is the number of independent observations. The functions ƒ and g alone are readily integrated, and remain readily integrated when multiplied by a Gaussian potential with diagonal covariance matrices, so that the following variational family is suitable for constructing the variational inference problem: 
         [0000]    
       
         
           
             
               
                 
                   = 
                   
                     { 
                     
                       
                         
                           r 
                            
                           
                             ( 
                             
                               t 
                               ; 
                               θ 
                             
                             ) 
                           
                         
                         := 
                         
                            
                           
                             
                               
                                 - 
                                 
                                   1 
                                   2 
                                 
                               
                                
                               
                                 t 
                                 T 
                               
                                
                               
                                 diag 
                                  
                                 
                                   ( 
                                   
                                     θ 
                                     1 
                                   
                                   ) 
                                 
                               
                                
                               t 
                             
                             + 
                             
                               
                                 
                                   θ 
                                   ~ 
                                 
                                 2 
                                 T 
                               
                                
                               t 
                             
                           
                         
                       
                       , 
                       
                         θ 
                         = 
                         
                           
                             ( 
                             
                               
                                 θ 
                                 1 
                                 T 
                               
                               , 
                               
                                 θ 
                                 2 
                                 T 
                               
                             
                             ) 
                           
                           T 
                         
                       
                       , 
                       
                         
                           θ 
                           1 
                         
                         ∈ 
                         
                           ℝ 
                           n 
                         
                       
                       , 
                       
                         
                           θ 
                           2 
                         
                         ∈ 
                         
                           ℝ 
                           n 
                         
                       
                     
                     } 
                   
                 
               
               
                 
                   ( 
                   11 
                   ) 
                 
               
             
           
         
       
     
         [0000]    A approximation to the integral of Equation (10) is obtained by minimizing the upper bound Ī given in Equation (7) using the system of  FIG. 1 . 
         [0033]    Integration of the orthogonal univariate function is first considered. Here the first term I ƒ  required to compute the bound can be obtained efficiently in terms of univariate integrals: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       I 
                       f 
                     
                      
                     
                       ( 
                       
                         θ 
                         ; 
                         p 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       
                         ∏ 
                         
                           i 
                           = 
                           1 
                         
                         n 
                       
                        
                       
                         
                           ∫ 
                           ℝ 
                           
                               
                           
                         
                          
                         
                             
                         
                          
                         
                           
                             
                               f 
                               i 
                               p 
                             
                              
                             
                               ( 
                               
                                 t 
                                 i 
                               
                               ) 
                             
                           
                            
                           
                              
                             
                               p 
                               ( 
                               
                                 
                                   - 
                                   
                                     
                                       
                                         u 
                                         
                                           1 
                                            
                                           i 
                                         
                                       
                                        
                                       
                                         t 
                                         i 
                                         2 
                                       
                                     
                                     2 
                                   
                                 
                                 + 
                                 
                                   
                                     u 
                                     
                                       2 
                                        
                                       i 
                                     
                                   
                                    
                                   
                                     t 
                                     i 
                                   
                                 
                               
                               ) 
                             
                           
                            
                           
                              
                             
                               t 
                               i 
                             
                           
                         
                       
                     
                     = 
                     
                       
                         ∏ 
                         
                           i 
                           = 
                           1 
                         
                         n 
                       
                        
                       
                           
                       
                        
                       
                         
                           U 
                           
                             [ 
                             
                               f 
                               i 
                             
                             ] 
                           
                         
                          
                         
                           ( 
                           
                             
                               u 
                               
                                 1 
                                  
                                 i 
                               
                             
                             , 
                             
                               u 
                               
                                 2 
                                  
                                 i 
                               
                             
                             , 
                             p 
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   12 
                   ) 
                 
               
             
           
         
       
     
         [0000]    where the univariate integrals U [h] :           2 ×[01]                    are defined as: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       U 
                       
                         [ 
                         h 
                         ] 
                       
                     
                      
                     
                       ( 
                       
                         α 
                         , 
                         β 
                         , 
                         p 
                       
                       ) 
                     
                   
                   := 
                   
                     
                       ∫ 
                       ℝ 
                       
                           
                       
                     
                      
                     
                       
                         
                           ( 
                           
                             
                               h 
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                              
                             
                                
                               
                                 
                                   
                                     - 
                                     
                                       1 
                                       2 
                                     
                                   
                                    
                                   α 
                                    
                                   
                                       
                                   
                                    
                                   
                                     t 
                                     2 
                                   
                                 
                                 + 
                                 
                                   β 
                                    
                                   
                                       
                                   
                                    
                                   t 
                                 
                               
                             
                           
                           ) 
                         
                         p 
                       
                        
                       
                           
                       
                        
                       
                          
                         
                           t 
                           i 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   13 
                   ) 
                 
               
             
           
         
       
     
         [0000]    Here, h is an arbitrary univariate function                              . These integrals can be efficiently computed using quadrature integration (e.g. recursive adaptive Simpson quadrature), but in some practical applications the same functions ƒ i  in Equation (12) are used for many factors. A considerable computational efficiency enhancement can be obtained by designing integrals dedicated to some functions (in practice, using pre-computed functions with linear interpolation is 10 to 100 times faster than running a new quadrature every time). One commonly encountered example is the step function: ƒ i (x)=           {x≧0}  for all iε{1, . . . , n}. For this function and using Gaussian pivot functions as specified in Equation (11), a closed form expression is obtained in terms of normal cdf function Φ: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       U 
                       
                         [ 
                         
                            
                           
                             { 
                             
                               · 
                               
                                 ≥ 
                                 0 
                               
                             
                             } 
                           
                         
                         ] 
                       
                     
                      
                     
                       ( 
                       
                         α 
                         , 
                         β 
                         , 
                         p 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       
                         
                           2 
                            
                           π 
                         
                         
                           p 
                            
                           
                               
                           
                            
                           α 
                         
                       
                     
                      
                     
                       Φ 
                        
                       
                         ( 
                         
                           β 
                            
                           
                             
                               p 
                               α 
                             
                           
                         
                         ) 
                       
                     
                      
                     
                        
                       
                         
                           p 
                            
                           
                               
                           
                            
                           
                             β 
                             2 
                           
                         
                         
                           2 
                            
                           α 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   14 
                   ) 
                 
               
             
           
         
       
     
         [0000]    where Φ is the cumulative density function of a centered normalized Gaussian random variable. 
         [0034]    Concerning the other factor I g , its log-quadratic form corresponds to a standard Gaussian integral: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       I 
                       g 
                     
                      
                     
                       ( 
                       
                         θ 
                         ; 
                         q 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ∫ 
                       
                         ℝ 
                         n 
                       
                       
                           
                       
                     
                      
                     
                       
                          
                         
                           
                             
                               - 
                               
                                 q 
                                 2 
                               
                             
                              
                             
                               
                                 t 
                                 T 
                               
                                
                               
                                 ( 
                                 
                                   A 
                                   - 
                                   
                                     diag 
                                      
                                     
                                       ( 
                                       
                                         θ 
                                         1 
                                       
                                       ) 
                                     
                                   
                                 
                                 ) 
                               
                             
                              
                             t 
                           
                           + 
                           
                             
                               
                                 q 
                                  
                                 
                                   ( 
                                   
                                     b 
                                     - 
                                     
                                       θ 
                                       2 
                                     
                                   
                                   ) 
                                 
                               
                               T 
                             
                              
                             t 
                           
                         
                       
                        
                       
                           
                       
                        
                       
                          
                         t 
                       
                     
                   
                 
               
               
                 
                   ( 
                   15 
                   ) 
                 
               
             
           
         
       
     
         [0000]    which can be recast as: 
         [0000]        I   g (θ; q )= N ( q ( A −diag(θ 1 )), q ( b−θ   2 ))  (16)
 
         [0000]    where N(A,b) is involves a matrix inversion and a determinant computation: 
         [0000]    
       
         
           
             
               
                 
                   
                     N 
                      
                     
                       ( 
                       
                         A 
                         , 
                         b 
                       
                       ) 
                     
                   
                   := 
                   
                     
                       
                         ∫ 
                         
                           ℝ 
                           n 
                         
                         
                             
                         
                       
                        
                       
                         
                            
                           
                             
                               
                                 - 
                                 
                                   1 
                                   2 
                                 
                               
                                
                               
                                 t 
                                 T 
                               
                                
                               At 
                             
                             + 
                             
                               
                                 b 
                                 T 
                               
                                
                               t 
                             
                           
                         
                          
                         
                             
                         
                          
                         
                            
                           t 
                         
                       
                     
                     = 
                     
                       
                         
                           
                             ( 
                             
                               2 
                                
                               π 
                             
                             ) 
                           
                           
                             n 
                             2 
                           
                         
                         
                           
                              
                             A 
                              
                           
                           
                             1 
                             2 
                           
                         
                       
                        
                       
                          
                         
                           
                             1 
                             2 
                           
                            
                           
                             b 
                             T 
                           
                            
                           
                             A 
                             
                               - 
                               1 
                             
                           
                            
                           b 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   17 
                   ) 
                 
               
             
           
         
       
     
         [0035]    Truncated multivariate Gaussian integration is next considered. Most of the truncated multivariate Gaussian integration problems with linear truncations can be put under the canonical form of Equation (10) where truncations are orthogonal, i.e. ƒ i (x)=           {x≧0}  for all iε{1, . . . , n}. Integrating truncated correlated Gaussian is a known open problem for which only approximation techniques are presently available. To be used in a learning framework, upper and lower bounds to the integral of Equation (10) are often useful. The disclosed variational Hölder&#39;s inequality approach disclosed herein is well suited to provide the upper bound. 
         [0036]    With reference to  FIGS. 2 and 3 , examples of truncated Gaussian integration are shown. Each row of  FIGS. 2 and 3  represents a different correlation/truncation setting. The left column shows the target function ƒg. The middle column shows its first tractable approximation ƒ p r p  (for  FIG. 2  and  FIG. 3 ) which is the product of orthogonal univariate function. The right column shows its second tractable approximation g q r −q  (for  FIG. 2  and  FIG. 3 ) which is the correlated Gaussian distribution. Symbols ‘x’ and ‘+’ are the exact and approximate means, respectively. 
         [0037]    Sparse probit regression is next considered. Probit regression is a Generalized Linear Model (GLM) for binary observations. In Bayesian statistics, probit regression is often preferred to logistic regression as there is a natural interpretation of the model as a multivariate truncated Gaussian variable. In fact it corresponds exactly to a truncated multivariate Gaussian integration. It is shown here that good performance for this problem can be obtained using the variational Hölder approach disclosed herein. In brief, the upper bounding technique is applied to the evidence integral to obtain tractable approximations. 
         [0038]    Let          =(X,y), where Xε           m×d  is the data matrix, i.e. X ij  is the value of the j-th covariate for the i-th data point, and yε{−1,1} n  the vector of binary output variables. We define M as the signed data matrix, such that M ij =y i X ij . The likelihood function is p(         |w)=Π i=1   m p(y i |X i1 , . . . , w)=Φ(Mw) where Φ:         ™         [0,1] is the product of normal cdfs: Φ(s)=Π i=1   m ∫ −∞   s     i             (u i |0,1)du i =Π i=1   m ∫ −∞   0           (v i −s i |0,1)dv i =                     {v≧0}           (v|s, I n )dv, and where          (u|μ,Σ) is the multivariate Gaussian probability density function (pdf) with mean μ and covariance matrix E. Arbitrary uncorrelated priors ρ j (w j ) on the coefficients w j  are considered here. The marginal data probability (evidence) is obtained by averaging this likelihood over the prior distribution: 
         [0000]        p (         )=           p (         | w ) dp ( w )=         Φ( Mw ) dp ( w )  (18)
 
         [0000]    which can be written as: 
         [0000]    
       
         
           
             
               
                 
                   
                     p 
                      
                     
                       ( 
                        
                       ) 
                     
                   
                   = 
                   
                     
                       ∫ 
                       
                         ℝ 
                         d 
                       
                       
                           
                       
                     
                      
                     
                       
                         ∫ 
                         
                           ℝ 
                           m 
                         
                         
                             
                         
                       
                        
                       
                         
                            
                           
                             { 
                             
                               v 
                               ≥ 
                               0 
                             
                             } 
                           
                         
                          
                         
                           
                             ∏ 
                             
                               j 
                               = 
                               1 
                             
                             d 
                           
                            
                           
                               
                           
                            
                           
                             
                               
                                 
                                   p 
                                   j 
                                 
                                  
                                 
                                   ( 
                                   
                                     w 
                                     j 
                                   
                                   ) 
                                 
                               
                               
                                  
                                 
                                   ( 
                                   
                                     
                                       
                                         w 
                                         j 
                                       
                                       | 
                                       0 
                                     
                                     , 
                                     1 
                                   
                                   ) 
                                 
                               
                             
                              
                              
                             
                               ( 
                               
                                 
                                   v 
                                   | 
                                   Mw 
                                 
                                 , 
                                 
                                   I 
                                   m 
                                 
                               
                               ) 
                             
                              
                              
                             
                               ( 
                               
                                 
                                   w 
                                   | 
                                   
                                     0 
                                     d 
                                   
                                 
                                 , 
                                 
                                   I 
                                   d 
                                 
                               
                               ) 
                             
                              
                             
                                 
                             
                              
                             
                                
                               v 
                             
                              
                             
                                 
                             
                              
                             
                                
                               w 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   19 
                   ) 
                 
               
             
           
         
       
     
         [0000]    or more succinctly as: 
         [0000]    
       
         
           
             
               
                 
                   
                     p 
                      
                     
                       ( 
                        
                       ) 
                     
                   
                   = 
                   
                     
                       ∫ 
                       
                         ℝ 
                         
                           d 
                           + 
                           m 
                         
                       
                       
                           
                       
                     
                      
                     
                       
                         ∏ 
                         
                           i 
                           = 
                           1 
                         
                         
                           d 
                           + 
                           m 
                         
                       
                        
                       
                           
                       
                        
                       
                         
                           
                             f 
                             i 
                           
                            
                           
                             ( 
                             
                               t 
                               i 
                             
                             ) 
                           
                         
                          
                         
                            
                           
                             
                               - 
                               
                                 1 
                                 2 
                               
                             
                              
                             
                               t 
                               T 
                             
                              
                             At 
                           
                         
                          
                         
                             
                         
                          
                         
                            
                           v 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   20 
                   ) 
                 
               
             
           
         
       
     
         [0000]    where: 
         [0000]    
       
         
           
             
               
                 
                   A 
                   = 
                   
                     
                       
                         [ 
                         
                           
                             
                               
                                 I 
                                 d 
                               
                             
                             
                               
                                 M 
                                 T 
                               
                             
                           
                           
                             
                               M 
                             
                             
                               
                                 
                                   I 
                                   n 
                                 
                                 + 
                                 
                                   MM 
                                   T 
                                 
                               
                             
                           
                         
                         ] 
                       
                        
                       
                           
                       
                        
                       and 
                        
                       
                           
                       
                        
                       
                         
                           f 
                           i 
                         
                          
                         
                           ( 
                           
                             t 
                             i 
                           
                           ) 
                         
                       
                     
                     = 
                     
                       { 
                       
                         
                           
                             
                               
                                 
                                   p 
                                   j 
                                 
                                  
                                 
                                   ( 
                                   
                                     w 
                                     j 
                                   
                                   ) 
                                 
                               
                               
                                  
                                 
                                   ( 
                                   
                                     
                                       
                                         w 
                                         j 
                                       
                                       | 
                                       0 
                                     
                                     , 
                                     1 
                                   
                                   ) 
                                 
                               
                             
                           
                           
                             
                               
                                 if 
                                  
                                 
                                     
                                 
                                  
                                 i 
                               
                               ≤ 
                               d 
                             
                           
                         
                         
                           
                             
                                
                               
                                 { 
                                 
                                   
                                     t 
                                     i 
                                   
                                   ≥ 
                                   0 
                                 
                                 } 
                               
                             
                           
                           
                             otherwise 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   21 
                   ) 
                 
               
             
           
         
       
     
         [0000]    The matrices I n  and 0 n×d  denote the identity matrix and zero matrix of size n×n and n×d, respectively. Equation (20) is based on the concatenation t:=[w T , v T ] T  and corresponds exactly the same form as Equation (10), i.e. a correlated Gaussian pdf multiplied by orthogonal n=d+m univariate functions. 
         [0039]    The system of  FIG. 1  evaluates the integral  12  of the form Z:=         ƒ(t)g(t)dv(t) in which the integrand is a product of two functions ƒ(t) and g(t). However, this can be readily generalized to an integrand which is the product of more than two factors. The extension is based on the generalized Hölder&#39;s inequality which is direct extension of the standard Hölder&#39;s inequality of Equation (1). Assume that ƒ 1 , . . . , ƒ K  are non-negative integrable functions and v is a known probability distribution (the base measure). The integral to be evaluated is Z:=         Π k=1   K (t)dv(t) where K≧2. In this context, the following generalized Hölder&#39;s inequality holds: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       ∫ 
                        
                       
                           
                       
                     
                      
                     
                       
                         ∏ 
                         
                           k 
                           = 
                           1 
                         
                         K 
                       
                        
                       
                           
                       
                        
                       
                         
                           
                             f 
                             k 
                           
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                          
                         
                             
                         
                          
                         
                            
                           
                             v 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                       
                     
                   
                   ≤ 
                   
                     
                       ∏ 
                       
                         k 
                         = 
                         1 
                       
                       K 
                     
                      
                     
                         
                     
                      
                     
                       
                         ( 
                         
                           
                             ∫ 
                              
                             
                                 
                             
                           
                            
                           
                             
                               
                                 f 
                                 k 
                                 
                                   1 
                                   
                                     ρ 
                                     k 
                                   
                                 
                               
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                              
                             
                                 
                             
                              
                             
                                
                               
                                 v 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                             
                           
                         
                         ) 
                       
                       
                         ρ 
                         k 
                       
                     
                   
                 
               
               
                 
                   ( 
                   22 
                   ) 
                 
               
             
           
         
       
     
         [0000]    for any K-dimensional vector ρεΔ K-1  where Δ K-1 ={ρ; ρ k ≧0, Σ k=1     K   ρ k =1} is the (K−1)-dimensional simplex. The generalized Hölder&#39;s inequality of Equation (22) is readily demonstrated by applying the standard Hölder inequality of Equation (1) recursively. By setting 
         [0000]    
       
         
           
             
               K 
               = 
               2 
             
             , 
             
               p 
               = 
               
                 
                   
                     1 
                     
                       ρ 
                       1 
                     
                   
                    
                   
                       
                   
                    
                   and 
                    
                   
                       
                   
                    
                   q 
                 
                 = 
                 
                   1 
                   
                     ρ 
                     2 
                   
                 
               
             
             , 
           
         
       
     
         [0000]    Equation (1) is recovered. 
         [0040]    The introduction of a pivot function into the generalized Hölder inequality of Equation (22) is developed as follows. Define a non-negative pivot function q(t)=Π i=1   K q k (t). Then, the following results holds: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       ∫ 
                        
                       
                           
                       
                     
                      
                     
                       
                         ∏ 
                         
                           
                             k 
                              
                             
                                 
                             
                              
                             ′ 
                           
                           = 
                           1 
                         
                         K 
                       
                        
                       
                           
                       
                        
                       
                         
                           
                             f 
                             k 
                           
                            
                           
                             ( 
                             t 
                             ) 
                           
                         
                          
                         
                             
                         
                          
                         
                            
                           
                             v 
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                       
                     
                   
                   ≤ 
                   
                     
                       ∏ 
                       
                         k 
                         = 
                         1 
                       
                       K 
                     
                      
                     
                         
                     
                      
                     
                       
                         ( 
                         
                           
                             ∫ 
                              
                             
                                 
                             
                           
                            
                           
                             
                               
                                 ( 
                                 
                                   
                                     f 
                                     
                                       k 
                                        
                                       
                                           
                                       
                                        
                                       
                                         ( 
                                         t 
                                         ) 
                                       
                                     
                                   
                                   
                                     
                                       q 
                                       k 
                                     
                                      
                                     
                                       ( 
                                       t 
                                       ) 
                                     
                                   
                                 
                                 ) 
                               
                               
                                 1 
                                 
                                   ρ 
                                   k 
                                 
                               
                             
                              
                             
                               
                                 ∏ 
                                 
                                   
                                     k 
                                      
                                     
                                         
                                     
                                      
                                     ′ 
                                   
                                   = 
                                   1 
                                 
                                 K 
                               
                                
                               
                                   
                               
                                
                               
                                 
                                   
                                     q 
                                     
                                       k 
                                        
                                       
                                           
                                       
                                        
                                       ′ 
                                     
                                   
                                    
                                   
                                     ( 
                                     t 
                                     ) 
                                   
                                 
                                  
                                 
                                     
                                 
                                  
                                 
                                    
                                   
                                     v 
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                 
                               
                             
                           
                         
                         ) 
                       
                       
                         ρ 
                         k 
                       
                     
                   
                 
               
               
                 
                   ( 
                   23 
                   ) 
                 
               
             
           
         
       
     
         [0000]    This result can be demonstrated as follows. Substituting 
         [0000]    
       
         
           
             
               f 
               k 
             
             
               q 
               k 
             
           
         
       
     
         [0000]    for ƒ k  and Π k′=1   K q k ,(t)dv(t) for dv(t) in Equation (22) leaves the left integral unchanged but gives the required right hand side of the equation. This result is also a variational Hölder inequality, since it is a direct generalization of Equation (2) for more than two factors. The function q(t) is actually a factored approximation rather than a formal pivot function, similarly to the difference between binary logistic regression (with odd-ratios) and multinomial logistic regression (with unnormalized scores). Nonetheless, the function q(t) serves the same functionality (e.g. providing a tighter bound) as does the function r(t) in Equation (2), and the function q(t) is defined as a pivot function herein. 
         [0041]    The pivot function q(t) provides a tight bound as follows. In the following, K≧2 is assumed. (The special case of K=2 is also covered by Equation (2) and its sequel). Analogous to the Equation (5) and its surrounding discussion, if q k (t)=c k ƒ k (t) for any tε          where c k  are non-negative constants then the inequality in Equation (23) is an equality. 
         [0042]    In Equation (23), the integral over the product is transformed into a product of integrals, each of them being potentially easier to approximate. Equation (23) can be written in a slightly different for by isolating the terms ƒ k  and q k , and taking the logarithm: 
         [0000]    
       
         
           
             
               
                 
                   
                     log 
                      
                     
                       
                         ∫ 
                          
                         
                             
                         
                       
                        
                       
                         
                           ∏ 
                           
                             k 
                             = 
                             1 
                           
                           K 
                         
                          
                         
                             
                         
                          
                         
                           
                             
                               f 
                               
                                 k 
                                  
                                 
                                     
                                 
                               
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                            
                           
                               
                           
                            
                           
                              
                             
                               v 
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                   ≤ 
                   
                     
                       ∑ 
                       
                         k 
                         = 
                         1 
                       
                       K 
                     
                      
                     
                         
                     
                      
                     
                       
                         ρ 
                         k 
                       
                        
                       log 
                        
                       
                         
                           ∫ 
                            
                           
                               
                           
                         
                          
                         
                           
                             
                               ( 
                               
                                 
                                   f 
                                   
                                     k 
                                      
                                     
                                         
                                     
                                      
                                     
                                       ( 
                                       t 
                                       ) 
                                     
                                   
                                 
                                 
                                   
                                     q 
                                     k 
                                   
                                    
                                   
                                     ( 
                                     t 
                                     ) 
                                   
                                 
                               
                               ) 
                             
                             
                               1 
                               
                                 ρ 
                                 k 
                               
                             
                           
                            
                           
                             
                               ∏ 
                               
                                 
                                   k 
                                    
                                   
                                       
                                   
                                    
                                   ′ 
                                 
                                 = 
                                 1 
                               
                               K 
                             
                              
                             
                                 
                             
                              
                             
                               
                                 
                                   q 
                                   
                                     k 
                                      
                                     
                                         
                                     
                                      
                                     ′ 
                                   
                                 
                                  
                                 
                                   ( 
                                   t 
                                   ) 
                                 
                               
                                
                               
                                   
                               
                                
                               
                                 
                                    
                                   
                                     v 
                                      
                                     
                                       ( 
                                       t 
                                       ) 
                                     
                                   
                                 
                                 . 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   24 
                   ) 
                 
               
             
           
         
       
     
         [0000]    It is seen that this bound can be viewed as a Reverse Information Inequality by comparison directly with the standard information inequality, further assuming that q is a proper distribution function, i.e. that it sums to one: 
         [0000]    
       
         
           
             
               
                 
                   
                     log 
                      
                     
                       
                         ∫ 
                          
                         
                             
                         
                       
                        
                       
                         
                           ∏ 
                           
                             k 
                             = 
                             1 
                           
                           K 
                         
                          
                         
                             
                         
                          
                         
                           
                             
                               f 
                               
                                 k 
                                  
                                 
                                     
                                 
                               
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                            
                           
                               
                           
                            
                           
                              
                             
                               v 
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                   ≥ 
                   
                     
                       
                         ∑ 
                         
                           k 
                           = 
                           1 
                         
                         K 
                       
                        
                       
                         
                           ∫ 
                            
                           
                               
                           
                         
                          
                         
                           
                             
                               q 
                               
                                 k 
                                  
                                 
                                     
                                 
                                  
                                 ′ 
                               
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                            
                           log 
                            
                           
                               
                           
                            
                           
                             
                               f 
                               k 
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                            
                           
                               
                           
                            
                           
                              
                             
                               v 
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                           
                         
                       
                     
                     + 
                     
                        
                       
                         ( 
                         q 
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   25 
                   ) 
                 
               
             
           
         
       
     
         [0000]    where          (q)=−Σ k×1   K  log ƒ         Σ k′=1   K q k ,(t)log q k (t)dv(t) is the entropy of q. 
         [0043]    In analogy to the K=2 embodiment shown in  FIG. 1 , the result of minimizing the upper bound of Equation (23) respective to the pivot function q(t) is an optimized pivot function q opt (t) consisting of optimized elements q 1     opt   (t), . . . , q K     opt   (t) and the optimized K-dimensional vector ρ opt  consisting of optimized components ρ 1     opt   , . . . , ρ K     opt   . The integral is then evaluated as 
         [0000]    
       
         
           
             Z 
             ≅ 
             
               
                 ∏ 
                 
                   k 
                   = 
                   1 
                 
                 K 
               
                
               
                   
               
                
               
                 
                   ( 
                   
                     
                       ∫ 
                        
                       
                           
                       
                     
                      
                     
                       
                         
                           ( 
                           
                             
                               
                                 f 
                                 k 
                               
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                             
                               
                                 q 
                                 k 
                               
                                
                               
                                 ( 
                                 t 
                                 ) 
                               
                             
                           
                           ) 
                         
                         
                           1 
                           
                             ρ 
                             k 
                           
                         
                       
                        
                       
                         
                           ∏ 
                           
                             
                               k 
                                
                               
                                   
                               
                                
                               ′ 
                             
                             = 
                             1 
                           
                           K 
                         
                          
                         
                             
                         
                          
                         
                           
                             
                               q 
                               
                                 k 
                                  
                                 
                                     
                                 
                                  
                                 ′ 
                               
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                            
                           
                               
                           
                            
                           
                              
                             
                               v 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                   ) 
                 
                 
                   ρ 
                   k 
                 
               
             
           
         
       
     
         [0000]    with q(t)=q opt (t) and ρ=ρ opt . 
         [0044]    The disclosed variational inference approaches are fundamentally different from other inference techniques commonly used in Bayesian inference. It has the advantage of being scalable, i.e. it leads to a convex problem that can be efficiently optimized, thus reducing the computational resources required to perform the integration. There are diverse applications in data analytics where Bayesian inference is used to advantages, including by way of example: factorization methods, as used in customer care to profile users and agents, or in education analytics, where the performances of the student are measured based on a lot of random exam assignment. Another area of application is transportation analytics to recover the trajectory of a user based on partial location information (such as the location of the ticket validations): the recovery of the user home location cannot be perfect, but a distribution over the possible locations can be inferred. 
         [0045]    The disclosed variational integral approximations are based on minimization of an upper bound to a log-partition function. The disclosed variational inference problem is convex if the variational function is log-linear, which has practical and theoretical advantages. Application to handling sparse regression in GLMs, e.g. using probit regression (but the extension to other losses is straightforward) is described as an illustrative application. Advantageously, a convex objective is maintained even if the sparse factors exhibit heavy tails. 
         [0046]    It will be appreciated that various of the above-disclosed and other features and functions, or alternatives thereof, may be desirably combined into many other different systems or applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.