Patent Publication Number: US-10332016-B2

Title: Comparison of feature vectors of data using similarity function

Description:
FIELD OF THE INVENTION 
     The invention concerns a method for comparing data obtained from a sensor or interface to determine a rate of similarity between the data. In particular the invention concerns a data comparison method via machine learning. 
     STATE OF THE ART 
     Numerous tasks implemented in the field of computer vision (or digital vision) for example require the comparison of complex data such as images to obtain a similarity score between such data. 
     For example, in the field of biometric authentication, face images of individuals are compared to determine whether the images have been obtained from the same person. 
     To treat this type of problem it is known to carry out an extraction of features from the data to be compared, the extraction of features converting the data to be compared into feature vectors, and subsequently to compute a similarity function between the feature vectors. 
     The computed similarity function generally comprises parameters that are a priori unknown. These parameters are determined and progressively optimised by machine learning. To do so, a processing unit conducts data comparison operations on a set of data known from a database, compares the results given by the similarity function with a real result and optimises the parameters of the similarity function accordingly for more reliable results. 
     For example from the publication by D. Chen, X. Cao L. Wang, F. Wen and J. Sun,  Bayesian Face Revisited: A Joint Formulation , in ECCV, 2012, a learning method is known for a similarity function between data wherein the data are modelled by summation of two independent Gaussian variables: the mean of the class to which one datum belongs and the variation of the datum relative to the mean. 
     For example, if the data are images of faces, the class corresponds to the identity of the subject, and therefore the variation relative to the mean of the class corresponds to all the changes which may occur between a mean face image of the subject and an image taken under different circumstances:
         Lighting, shadows on the image,   Facial pose in the image,   Face expression,   Local blur, etc.       

     However, an improvement in the performance level of comparison resulting from machine learning is limited by the fact that data of varying quality are taken into account in the database. As a result the determined similarity function shows deteriorated performance and hence deteriorated quality of comparison. The proposed comparison method is therefore not entirely reliable. 
     DISCLOSURE OF THE INVENTION 
     It is the objective of the invention to propose a data comparison method having improved performance compared with the prior art. 
     In this respect the subject of the invention is a method to compare two computer data items, obtained from a sensor or interface, carried out by processing means of a processing unit, the method comprising the computing of a similarity function between two feature vectors of the data to be compared,
 
characterized in that each feature vector of a datum is modelled as the summation of Gaussian variables the said variables comprising:
         a mean of a class to which the vector belongs,   an intrinsic deviation, and   observation noise of the vector
 
each feature vector being associated with a quality vector comprising information on the observation noise of the feature vector,
 
and in that the similarity function is computed from the feature vectors and from the associated quality vectors.
 
Advantageously but optionally, the method of the invention may further comprise at least one of the following characteristics:
   the similarity function is the logarithm of the ratio between the probability density P(x,y|H sim ,S ε     x   ,S ε     y   ) of the feature vectors with the vectors belonging to one same class, and the probability density P (x,y|H dis ,S ε     x   ,S ε     y   ) of the feature vectors with the vectors belonging to two different classes.   The similarity function is also computed as a function of the covariance matrices of the components of the feature vectors, and the covariance matrix of the observation noise of each feature vector is obtained as a function of the associated quality vector.   The method further comprises the implementation of a learning algorithm to determine the covariance matrices of the means of the classes to which the vectors belong and vector deviations relative to the class means.   The learning algorithm is an algorithm of expectation—maximization type;   The similarity function is given by the formula:
 
 LR ( x,y|S   ε     x     ,S   ε     y   ) = x   T ( A− ( S   μ   +S   ω   +S   ε     x   ) −1 ) x+y   T ( C− ( S   μ   +S   ω   +S   ε     y   ) −1 ) y+ 2 x   T   By − log| S   μ   +S   ω   +S   ε     x   |−log| A|+ constant
    where:
 
 A= ( S   μ   +S   ω   +S   ε     x     −S   μ ( S   μ   +S   ω   +S   ε     y   ) −1   S   μ ) −1  
 
 B=−AS   μ ( S   μ   +S   ω   +S   ε     y   ) −1  
 
 C= ( S   μ   +S   ω   +S   ε     y   ) −1  ( I+S   μ   AS   μ ( S   μ   +S   ω   +S   ε     y   ) −1 )
    and where S μ  is the covariance matrix of the means of the classes (inter-class covariance matrix), S ω  is the covariance matrix of the deviations relative to a mean (intra-class covariance matrix), and S εx  and S εy  are the covariance matrices of the observation noises of vectors x and y respectively;   The computer data derived from sensors or interfaces are data representing physical objects or physical magnitudes;   The computer data derived from sensors or interfaces are images, and the feature vectors are obtained by applying at least one filter to the images;   The components of the quality vector are generated as a function of type of datum and type of features forming the feature vector;   The method further comprises the comparison of the result of the computed similarity function with a threshold to determine whether the data belong to a common class.
 
A further subject of the invention is a computer programme product comprising code instructions to implement the method according to the preceding description when executed by processing means of a processing unit.
 
A further subject of the invention is a system comprising:
   a database comprising a plurality of so-called labelled data;   a data acquisition unit; and   a processing unit comprising processing means adapted to construct two feature vectors and two associated quality vectors from two data items, said processing unit also being adapted to compare the data by implementing the comparison method according to the preceding description.       

     The proposed method allows data quality to be taken into account when computing the similarity function between data. This makes it possible to use variable weighting between data of good quality and more uncertain data. 
     For example when the method of the invention is applied to a comparison of images, the shadow or blur regions of an image are not taken into account by the similarity function with as much weighting as the clearly visible, clearly distinct regions. 
     Increased performance of data comparison is thereby obtained. 
     Additionally, machine learning allows optimisation of the similarity function parameters and hence improved performance of the comparison method. 
    
    
     
       DESCRIPTION OF THE FIGURES 
       Other characteristics, objectives and advantages of the invention will become apparent from the following non-limiting description given solely for illustration and is to be read in connection with the appended drawings in which: 
         FIG. 1  illustrates an example of a system adapted to implement a comparison method; 
         FIG. 2  illustrates the main steps of a data comparison method according to one embodiment of the invention; 
     
    
    
     DETAILED DESCRIPTION OF AT LEAST ONE EMBODIMENT OF THE INVENTION 
     With reference to  FIG. 1 , a system  1  is illustrated comprising a processing unit  10 , comprising processing means  11  to implement the computer data comparison method described below. 
     The processing unit  10  may be an integrated circuit for example and the processing means may be a processor. 
     Advantageously the system  1  further comprises an optionally remote database  20  storing in memory a plurality of data used by the processing unit  10  to carry out machine learning as described below. 
     Finally, the system  1  comprises a data acquisition unit  30 , or if the data acquisition unit  30  is independent of the system it comprises an interface (not illustrated) adapted to communicate with such a unit. In this manner the system  1  is able to receive and process data b, in particular for comparison thereof using the method described below. 
     Depending on the type of data to be compared in the method described below, the data acquisition unit may be of any type e.g. an optical sensor (photographic camera, video camera, scanner), acoustic sensor, fingerprint sensor, movement sensor etc. It may also be a Man-Machine interface (keypad, tablet with touch-screen interface) to record data entered by an operator such as a text, figure, etc. 
     The computer data b are obtained by the acquisition unit  30 , and are therefore derived from a sensor or interface e.g. a Man-Machine interface. They may be data representing a physical object e.g. an image, a schematic, a recording, a description, or representing a physical magnitude (electric, mechanical, thermal, acoustic, etc.), for example data recorded by a sensor. 
     The processing means  11  of the processing unit are advantageously configured to perform the data comparison method described below by executing a suitable programme. 
     To implement this method, the processing means  11  also advantageously comprise a feature extracting module  12  adapted to generate—from an input computer datum b communicated by a data acquisition unit  30 —an extraction of features to generate a feature vector x associated with the datum and a quality vector qx of the datum associated with the feature vector. 
     The quality vector qx may be a vector of same size as the feature vector and each element thereof indicates a quality of the information contained in the corresponding element of the feature vector x. Alternatively, the quality vector qx may be of any size. The generation thereof is dependent on the type of datum b. 
     For example, feature extraction can be performed by applying to datum b one or more filters designed for this purpose, optionally followed by processing of the filtering result (e.g. computed histogram, etc.). 
     The generation of the quality vector is dependent on the type of datum b and type of features of the feature vector x—i.e. component elements of vector x. Each element of the quality vector takes into account intrinsic datum-related information associated with the particular features of the feature vector. 
     For example, in the field of signal processing or image processing, when the datum is an image or acquisition of a representative signal acquired by a sensor, it is frequent to use as feature vector x a frequency representation (e.g. Fourier transform) or spatial-frequency representation (e.g. wavelet transform) of the data. Each component of the feature vector then only depends on some frequency bands. 
     In such cases, the high frequency components of the datum may prove to be more discriminating than low frequency components, but also more sensitive to phenomena such as the presence of noise or lack of signal resolution. 
     The amount of noise in the datum can be determined by analysing its energy spectrum if the datum is a signal acquired by a sensor, or its intrinsic resolution if the datum is an image. For example the article by Pfenning and Kirchner is known for determination of the resolution of an image: «Spectral Methods to Determine the Exact Scaling Factor of Resampled Digital Images», ISCCP, 2012. 
     The quality vector qx generated as a function of the feature vector x and of the intrinsic quality of the datum can then be constructed as follows:
         High quality is attributed to the components of the feature vector that are sensitive to the low frequency components of the datum;   High quality is attributed to the components of the feature vector that are sensitive to the high frequency components of the datum and have low noise level and/or high resolution;   Low quality is attributed to the components of the feature vector that are sensitive to the high frequency components and have a high noise level and/or low resolution.
 
The attributed quality values and thresholds for noise level or resolution can be determined experimentally to optimise the performance of the comparison method on a validation basis.
       

     According to another example, the datum is a face image. 
     According to this example, a feature vector can be obtained as shown in the article by Chen et al., «Blessing of Dimensionality: High-dimensional Feature and Its Efficient Compression for Face Verification», VCPR, 2013, by concatenating local descriptors extracted in the vicinity of certain semantic points of the face (e.g. tip of nose, corners of the mouth, eyes, etc.). 
     This representation has the advantage of being more robust against variations in pose than methods which extract descriptors on a regular grid. 
     However the extraction of these features comprises a step to detect these points. Throughout this step a detector can be used which in addition to providing the most probable position of each point of the face in the image, also provides information translating the confidence level of detection accuracy. 
     A measurement is known for example from the article by Rapp et al., «Blessing of Dimensionality: «Multiple kernel learning SVM and statistical validation for facial landmark detection», Automatic Face &amp; Gesture Recognition, 2011, which measures the distance to the separating hyperplane when using a detector based on Support Vector Machines (SVM). 
     Another example is given in the article by Dantone et al. «Real-time Facial Feature Detection using Conditional Regression Forests”, CVPR, 2012, wherein a measurement of confidence is given by a number of votes determined by a detector using regression trees. 
     This confidence information can be used to create a quality associated with each component of the feature vector by attributing thereto the quality of detection of the facial semantic point to which it corresponds. 
     According to a further example, when the face image is a face image generated from an image which is not a front image of the face e.g. by applying the method described in application N° FR 2 998 402, the quality vector may be a confidence index, this index being relatively higher for the points of the face occurring in the original image and relatively lower for the points of the face not occurring in the original image and reconstructed via extrapolation. 
     More generally, when the datum is an image the quality vector can be obtained by local measurement of blur. 
     Alternatively, the feature-extracting module is a module of the acquisition unit  30  enabling the acquisition unit to communicate directly with the processing means  11  a feature vector and an associated quality vector. 
     Data Comparison Method 
     With reference to  FIG. 2  a description is now given of the data comparison method implemented by the processing means  11  of the processing unit. 
     This method comprises the comparison of two data items by calculating a similarity function  100  between two feature vectors x and y of same size obtained from the data respectively, and by performing machine learning  200  of the parameters of the similarity function on a database. 
     In this method each feature vector is modelled as the summation of three independent Gaussian variables:
 
 x=μ+ω+ε 
 
where:
         μ is the mean of a class to which vector x belongs;   ω is the intrinsic deviation of vector x from the mean; and   ε is observation noise.       

     A class is a set of feature vectors considered to be similar. Two feature vectors are considered similar if their comparison by the similarity function produces a higher result than a threshold, this threshold being determined empirically. 
     For example, if the data are face images, a class advantageously corresponds to an individual. By comparing two feature vectors of several data, the data are considered to be similar if they originate from the same individual. 
     To return to the model previously described, two feature vectors belonging to one same class therefore have an identical value μ, but different values of ω and ε. 
     If the feature vectors belong to different classes, the three variables are fully independent. 
     It is considered that these three variables follow a multivariate normal distribution centred at 0, and the respective covariance matrixes are written S μ , S ω , and S ε . S μ  is called an inter-class covariance matrix, S ω  an intra-class covariance matrix and S ε  an observation noise covariance matrix. 
     S μ , S ω  are unknowns common to all the feature vectors. 
     S ε  on the other hand is known since it is obtained from the quality vector associated with the feature vector, by the feature extracting module. It is of same size as the associated feature vector. 
     For example, assuming that the observation noises do not correlate with one another, S ε  can be well approximated by a diagonal matrix. 
     The elements of this diagonal matrix, then corresponding to the variance of the components of the quality vector, can be obtained from this vector. 
     For example, variance can be imposed by applying to the components of the quality vector qx a sigmoid function of type f(q x )=1/e aqx+b . The coefficients a and b can be chosen to associate a determined variance level with a quality level. 
     For example, a high quality can be associated with zero variance, a very low quality can be associated with maximum variance, the intermediate variances corresponding to intermediate qualities. 
     In general, since the quality vector and the feature vector depend on datum type, the transfer function which converts a quality vector to a noise covariance matrix is specific to the associated quality vector and feature vector. 
     In the remainder hereof S εx  denotes the covariance matrix of the background noise of vector x obtained from the quality vector qx, and S εy  is the covariance matrix of the background noise of vector y obtained from the quality vector qy. 
     H sim  denotes the hypothesis that two feature vectors belong to one same class i.e. the corresponding data are considered to be similar, and H dis  denotes the reverse hypothesis that the feature vectors belong to different classes and the corresponding data are considered to be dissimilar. 
     The joint probability of generating x and y, knowing their respective covariance matrixes of background noise and considering the hypothesis H sim , is written P(x,y|H sim ,S ε     x   ,S ε     y   ). This probability follows a Gaussian law for which the centre is zero and the covariance matrix is S sim :P(x,y|H sim ,S ε     x   ,S ε     y   )=N([x T y T ] T |0, S sim ). 
     The joint probability of generating x and y, knowing their respective covariance matrixes of background noise and considering the hypothesis H dis , is written P(x,y|H dis ,S ε     x   ,S ε     y   ). This probability follows a Gaussian law for which the centre is zero and the covariance matrix is S dis :P(x,y|H dis ,S ε     x   ,S ε     y   )=N([x T y T ] T |0, S dis ). 
     The matrixes and S sim  and S dis  are defined as follows: 
     
       
         
           
             
               S 
               sim 
             
             = 
             
               [ 
               
                 
                   
                     
                       
                         S 
                         μ 
                       
                       + 
                       
                         S 
                         ω 
                       
                       + 
                       
                         S 
                         
                           ɛ 
                           x 
                         
                       
                     
                   
                   
                     
                       S 
                       μ 
                     
                   
                 
                 
                   
                     
                       S 
                       μ 
                     
                   
                   
                     
                       
                         S 
                         μ 
                       
                       + 
                       
                         S 
                         ω 
                       
                       + 
                       
                         S 
                         
                           ɛ 
                           y 
                         
                       
                     
                   
                 
               
               ] 
             
           
         
       
       
         
           
             
               S 
               dis 
             
             = 
             
               [ 
               
                 
                   
                     
                       
                         S 
                         μ 
                       
                       + 
                       
                         S 
                         ω 
                       
                       + 
                       
                         S 
                         
                           ɛ 
                           x 
                         
                       
                     
                   
                   
                     
                       
                           
                       
                       ⁢ 
                       0 
                     
                   
                 
                 
                   
                     
                       
                           
                       
                       ⁢ 
                       0 
                     
                   
                   
                     
                       
                         S 
                         μ 
                       
                       + 
                       
                         S 
                         ω 
                       
                       + 
                       
                         S 
                         
                           ɛ 
                           y 
                         
                       
                     
                   
                 
               
               ] 
             
           
         
       
     
     The probability density of P(x,y|H sim ,S ε     x   ,S ε     y   ) in a manner known per se is 
                 1         (     2   ⁢   π     )       N   /   2       ⁢            S   sim            1   2           ⁢     e       -     1   2       ⁢     x   T     ⁢     S   sim     -   1       ⁢   x         ,         
where |S sim | is the determinant of S sim , and N is the dimension of a feature vector.
 
     The same expression applies mutatis mutandis to the density probability P(x,y|H dis ,S ε     x   ,S ε     y   ). 
     The computed similarity function to compare the two data corresponding to the vectors x and y is the logarithm of the ratio between the probability density of the feature vectors with the vectors belonging to one same class, and the probability density of the feature vectors with the vectors belonging to two different classes. 
     The similarity function is therefore expressed as follows: 
     
       
         
           
             
               LR 
               ⁢ 
               
                   
               
               ⁢ 
               
                 ( 
                 
                   x 
                   , 
                   
                     y 
                     | 
                     
                       S 
                       
                         ɛ 
                         x 
                       
                     
                   
                   , 
                   
                     S 
                     
                       ɛ 
                       y 
                     
                   
                 
                 ) 
               
             
             = 
             
               log 
               ⁡ 
               
                 ( 
                 
                   
                     P 
                     ⁡ 
                     
                       ( 
                       
                         x 
                         , 
                         
                           y 
                           | 
                           
                             H 
                             sim 
                           
                         
                         , 
                         
                           S 
                           
                             ɛ 
                             x 
                           
                         
                         , 
                         
                           S 
                           
                             ɛ 
                             y 
                           
                         
                       
                       ) 
                     
                   
                   
                     P 
                     ⁡ 
                     
                       ( 
                       
                         x 
                         , 
                         
                           y 
                           | 
                           
                             H 
                             dis 
                           
                         
                         , 
                         
                           S 
                           
                             ɛ 
                             x 
                           
                         
                         , 
                         
                           S 
                           
                             ɛ 
                             y 
                           
                         
                       
                       ) 
                     
                   
                 
                 ) 
               
             
           
         
       
     
     When using the expression of probability density indicated above, and when developing the function using the block inversion formula to invert the matrixes S sim  and S dis , the similarity function obtained is expressed as follows:
 
 LR ( x,y|S   ε     x     ,S   ε     y   ) = x   T ( A− ( S   μ   +S   ω   +S   ε     x   ) −1 ) x+y   T ( C− ( S   μ   +S   ω   +S   ε     y   ) −1 ) y+ 2 x   T   By − log| S   μ   +S   ω   +S   ε     x   |−log| A|+ constant
 
In this expression, A, B and C are terms resulting from block inversion of S sim  and are respectively expressed as follows:
 
 A= ( S   μ   +S   ω   +S   ε     x     −S   μ ( S   μ   +S   ω   +S   ε     y   ) −1   S   μ ) −1  
 
 B=−AS   μ ( S   μ   +S   ω   +S   ε     y   ) −1  
 
 C= ( S   μ   +S   ω   +S   ε     y   ) −1 ( I+S   μ   AS   μ ( S   μ   +S   ω   +S   ε     y   ) −1 )
 
The constant is not dependent on x,y, S ε     x    or S ε     y    and can therefore be ignored.
 
     It is therefore found that the similarity function LR takes into account the covariance matrixes S ε     x    and S ε     y    of the observation noises of x and y, and hence takes into account the quality vector associated with each feature vector. 
     The result of comparison is therefore impacted by the quality—or confidence—associated with a feature vector, which allows lesser weighting of a feature considered to be of poor or uncertain quality and greater weighting of a feature of good quality or having greater confidence. 
     As will be seen in the remainder hereof, this similarity function is also parameterised by machine learning. By taking into account the quality associated with a feature vector, the impact of a datum of poor quality on the parameterising of the function can be minimised. 
     The comparison method is therefore more reliable. 
     Returning to  FIG. 2 , the result of comparison is then compared at a step  110  with a determined threshold. 
     This threshold is advantageously determined empirically by applying a large number of comparisons to known feature vectors in a database (that are known to belong or not belong to one same class). 
     If the result of the similarity function applied to x and y is higher than the determined threshold, the corresponding data are considered to be similar. Otherwise the data are considered to be dissimilar. 
     The expression of the similarity function LR indicated previously shows that this function is parameterised by the covariance matrixes S μ , S ω , which are unknown. 
     Therefore the method comprises a step  200  to determine said matrixes by machine learning. 
     This method is advantageously conducted using an expectation-maximization algorithm (EM) and is performed on a set of data stored in the database  20 , this data being called «labelled» i.e. the respective classes to which they belong are known. 
     A class to which a number m c  of feature vectors belongs is denoted c, and X c =[x c,1   T , . . . , x c,m     c     T ] T  denotes the concatenation of the feature vectors of the class and 
               S     ɛ     c   ,   1         ,   …   ⁢           ,     S     ɛ     c   ,     m   c                 
denote their respective covariance matrixes of observation noise.
 
     The latent variables Z c =[μ c ,ω c,1   T , . . . ,ω c,m     c     T ] T  are defined for each class c in which each μ c  is the class mean; there is therefore only one, and each ω c,i  is the deviation of a feature vector in the class from this mean (there is therefore one per feature vector). 
     The parameter to be estimated by the EM algorithm is Θ={S μ ,S ω }. 
     The expectation-Maximisation algorithm is an iterative algorithm comprising a first step  210  to estimate the parameters of distribution of the latent variables P(Z c |X c , Θ ), where  Θ  is the preceding estimation of the parameters. 
     On initialisation of the method,  Θ  is a first empirical estimation of the parameters. 
     The initialisation of parameter S μ  is advantageously obtained by calculating, for each class c, the empirical mean of the class and then determining a covariance matrix of the mean values. 
     The initialisation of parameter S ω  can be obtained by calculating, for each class, the covariance matrix of the feature vectors from which the class mean is subtracted (i.e. feature vector differences relative to the mean) and then calculating the mean covariance matrix on all classes. 
     The algorithm next comprises a maximization step  220  following Θ of the expected logarithmic likelihood on the latent variables Z c : 
     
       
         
           
             
               Q 
               ⁡ 
               
                 ( 
                 
                   Θ 
                   , 
                   
                     Θ 
                     _ 
                   
                 
                 ) 
               
             
             = 
             
               
                 ∑ 
                 c 
               
               ⁢ 
               
                   
               
               ⁢ 
               
                 ∫ 
                 
                   
                     P 
                     ⁡ 
                     
                       ( 
                       
                         
                           
                             Z 
                             c 
                           
                           | 
                           
                             X 
                             c 
                           
                         
                         , 
                         
                           Θ 
                           _ 
                         
                       
                       ) 
                     
                   
                   ⁢ 
                   log 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     P 
                     ⁡ 
                     
                       ( 
                       
                         
                           X 
                           c 
                         
                         , 
                         
                           
                             Z 
                             c 
                           
                           | 
                           
                             Θ 
                             _ 
                           
                         
                       
                       ) 
                     
                   
                   ⁢ 
                   
                     dZ 
                     c 
                   
                 
               
             
           
         
       
     
     For the proper conducting of this step and to minimise computing time, consideration is given to the fact that the latent variables ω c,i  are conditionally independent of μ c  fixed by factorising P(Z c |X c , S   μ , S   ω ) as follows: 
     
       
         
           
             
               P 
               ⁡ 
               
                 ( 
                 
                   
                     
                       Z 
                       c 
                     
                     | 
                     
                       X 
                       c 
                     
                   
                   , 
                   
                     
                       S 
                       _ 
                     
                     μ 
                   
                   , 
                   
                     
                       S 
                       _ 
                     
                     ω 
                   
                 
                 ) 
               
             
             = 
             
               
                 P 
                 ⁡ 
                 
                   ( 
                   
                     
                       
                         μ 
                         c 
                       
                       | 
                       
                         X 
                         c 
                       
                     
                     , 
                     
                       
                         S 
                         _ 
                       
                       μ 
                     
                     , 
                     
                       
                         S 
                         _ 
                       
                       ω 
                     
                   
                   ) 
                 
               
               ⁢ 
               
                 
                   ∏ 
                   
                     i 
                     = 
                     1 
                   
                   
                     m 
                     c 
                   
                 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   P 
                   ⁡ 
                   
                     ( 
                     
                       
                         
                           ω 
                           
                             c 
                             , 
                             i 
                           
                         
                         | 
                         
                           x 
                           
                             c 
                             , 
                             i 
                           
                         
                       
                       , 
                       
                         μ 
                         c 
                       
                       , 
                       
                         
                           S 
                           _ 
                         
                         μ 
                       
                       , 
                       
                         
                           S 
                           _ 
                         
                         ω 
                       
                     
                     ) 
                   
                 
               
             
           
         
       
     
     Optimisation of Q(Θ,  Θ ) at step  220  requires computing the probability distributions of the parameters P(μ c |X c , S   μ , S   ω ) and P(ω c,i |x c,i ,μ c , S   μ , S   ω ). These computations are detailed below: 
                     P   ⁡     (         μ   c     |     X   c       ,       S   _     μ     ,       S   _     ω       )       ∝       P   ⁡     (         X   c     |     μ   c       ,       S   _     ω       )       ⁢     P   ⁡     (       μ   c     |       S   _     μ       )         ∝       ∏     i   =   1       m   c       ⁢           ⁢       𝒩   ⁡     (         x     c   ,   i       |     μ   c       ,         S   _     ω     +     S     ɛ     c   ,   i             )       ⁢     𝒩   ⁡     (         μ   c     |   0     ,       S   _     μ       )           ∝     e       -     1   2       ⁢     (         μ   c   T     ⁢       S   _     μ     -   1       ⁢     μ   c       +       ∑     i   =   1       m   c       ⁢           ⁢         (       μ   c     -     x     c   ,   i         )     T     ⁢       (         S   _     ω     +     S     ɛ     c   ,   i           )       -   1       ⁢     (       μ   c     -     x     c   ,   i         )           )         ∝     e       -     1   2       ⁢     (         (       μ   c     -     b     μ   c         )     T     ⁢       T     μ   c       -   1       ⁡     (       μ   c     -     b     μ   c         )         )                 (   1   )                 Where   ⁢     :       ⁢     
     ⁢       T     μ   c       =       (         S   _     μ     -   1       +       ∑     i   =   1       m   c       ⁢           ⁢       (         S   _     ω     +     S     ɛ     c   ,   i           )       -   1           )       -   1         ⁢     
     ⁢   And   ⁢     
     ⁢       b     μ   c       =       T     μ   c       ⁢       ∑     i   =   1       m   c       ⁢           ⁢         (         S   _     ω     +     S     ϵ     c   ,   i           )       -   1       ⁢     x     c   ,   i                                       
The combining of equation (1) and the fact that P(μ c |X c , S   μ , S   ω ) is a distribution of probabilities implies that P(μ c |X c , S   μ , S   ω )=N(μ c |b μ     c   ,T μ     c   ). Additionally:
 
 P (ω c,i   |X   c   , S     μ   , S     ω )=∫ P (μ c   |X   c   , S     μ   , S     ω )  P (ω c,i   |x   c,i ,μ c     S     ω ) dμ   c   ∝∫N (μ c   |b   μ     c     ,T   μ     c   )  N (ω c,i |0,   S     ω )  N (ω c,i   |x   c,i −μ c   ,S   ε     c,i   ) dμ   c   ∝∫N (μ c   |b   μ     c     ,T   μ     c   )  N (ω c,i   |R   c,i   S   ε     c,i     −1 ( x   c,i −μ c ),  R   c,i ) dμ   c  
 
Where  R   c,i =( S   ε     c,i     −1   + S     ω   −1 ) −1   ∝∫N (μ c   |b   μ     c     ,T   μ     c   )  N (μ c   |x   c,i   −R   c,i   −1   S   ε     c,i   ω c,i   ,R   c,i   −1   S   ε     c,i     R   c,i   −1 ) dμ   c  
 
               P   ⁡     (         ω     c   ,   i       |     X   c       ,       S   _     μ     ,       S   _     ω       )       =       𝒩   ⁡     (         ω     c   ,   i       |       R     c   ,   i       ⁢       S     ɛ     c   ,   i         -   1       ⁡     (       x     c   ,   i       -     b     μ   c         )           ,         R     c   ,   i       ⁢     S     ɛ     c   ,   i         -   1       ⁢     T     μ   c       ⁢     S     ɛ     c   ,   i         -   1       ⁢     R     c   ,   i         +     R     c   ,   i           )       =     𝒩   ⁡     (         ω     c   ,   i       |     b     ω     c   ,   i           ,     T     w     c   ,   i           )               
Where:
 
 T   w     c,i     =R   c,i   S   ε     c,i     −1   T   μc   S   ε     c,i     −1   R   c,i   +R   c,i  
 
And
 
 b   ω     c,i     =R   c,i   S   ε     c,i     −1 ( x   c,i   −b   μ     c   )
 
Step  220  therefore entails maximising relative to S μ  and S w :
 
     
       
         
           
             
               
                 Q 
                 ~ 
               
               ⁡ 
               
                 ( 
                 
                   Θ 
                   , 
                   
                     Θ 
                     _ 
                   
                 
                 ) 
               
             
             ⁢ 
             
               
                 ∑ 
                 
                   
                     c 
                     = 
                     1 
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     i 
                     = 
                     1 
                   
                 
                 
                   
                     nb 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     class 
                   
                   , 
                   
                     
 
                   
                   ⁢ 
                   
                     m 
                     c 
                   
                 
               
               ⁢ 
               
                   
               
               ⁢ 
               
                 ∫ 
                 
                   
                     P 
                     ⁡ 
                     
                       ( 
                       
                         
                           
                             Z 
                             c 
                           
                           | 
                           
                             X 
                             c 
                           
                         
                         , 
                         
                           
                             S 
                             = 
                           
                           μ 
                         
                         , 
                         
                           
                             S 
                             = 
                           
                           ω 
                         
                       
                       ) 
                     
                   
                   ⁢ 
                   
                     ( 
                     
                       
                         log 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           P 
                           ⁡ 
                           
                             ( 
                             
                               
                                 
                                   x 
                                   
                                     c 
                                     , 
                                     i 
                                   
                                 
                                 | 
                                 
                                   μ 
                                   c 
                                 
                               
                               , 
                               
                                 ω 
                                 
                                   c 
                                   , 
                                   i 
                                 
                               
                             
                             ) 
                           
                         
                       
                       + 
                       
                         log 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           P 
                           ⁡ 
                           
                             ( 
                             
                               
                                 μ 
                                 c 
                               
                               | 
                               
                                 S 
                                 μ 
                               
                             
                             ) 
                           
                         
                       
                       + 
                       
                         log 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           P 
                           ⁡ 
                           
                             ( 
                             
                               
                                 ω 
                                 
                                   c 
                                   , 
                                   i 
                                 
                               
                               | 
                               
                                 S 
                                 ω 
                               
                             
                             ) 
                           
                         
                       
                     
                     ) 
                   
                   ⁢ 
                   
                     dZ 
                     c 
                   
                 
               
             
           
         
       
     
     This is obtained by computing the gradients and solving 
     
       
         
           
             
               
                 ∂ 
                 
                   
                     Q 
                     ~ 
                   
                   ⁡ 
                   
                     ( 
                     
                       Θ 
                       , 
                       
                         Θ 
                         _ 
                       
                     
                     ) 
                   
                 
               
               
                 ∂ 
                 
                   S 
                   μ 
                 
               
             
             = 
             
               
                 0 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 and 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   
                     ∂ 
                     
                       
                         Q 
                         ~ 
                       
                       ⁡ 
                       
                         ( 
                         
                           Θ 
                           , 
                           
                             Θ 
                             _ 
                           
                         
                         ) 
                       
                     
                   
                   
                     ∂ 
                     
                       S 
                       ω 
                     
                   
                 
               
               = 
               0. 
             
           
         
       
     
     
       
         
           
             
               
                 ∂ 
                 
                   
                     Q 
                     ~ 
                   
                   ⁡ 
                   
                     ( 
                     
                       Θ 
                       , 
                       
                         Θ 
                         _ 
                       
                     
                     ) 
                   
                 
               
               
                 ∂ 
                 
                   S 
                   μ 
                 
               
             
             = 
             
               
                 
                   ∑ 
                   
                     c 
                     = 
                     1 
                   
                   
                     nb 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     classes 
                   
                 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 
                   ∫ 
                   
                     
                       P 
                       ⁡ 
                       
                         ( 
                         
                           
                             
                               Z 
                               c 
                             
                             | 
                             
                               X 
                               c 
                             
                           
                           , 
                           
                             Θ 
                             _ 
                           
                         
                         ) 
                       
                     
                     ⁢ 
                     
                       
                         
                           ∂ 
                           log 
                         
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           P 
                           ⁡ 
                           
                             ( 
                             
                               
                                 μ 
                                 c 
                               
                               | 
                               
                                 S 
                                 μ 
                               
                             
                             ) 
                           
                         
                       
                       
                         ∂ 
                         
                           S 
                           μ 
                         
                       
                     
                     ⁢ 
                     
                       dZ 
                       c 
                     
                   
                 
               
               = 
               
                 
                   
                     
                       ∑ 
                       
                         c 
                         = 
                         1 
                       
                       
                         nb 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         classes 
                       
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       ∫ 
                       
                         
                           P 
                           ⁡ 
                           
                             ( 
                             
                               
                                 
                                   μ 
                                   c 
                                 
                                 | 
                                 
                                   X 
                                   c 
                                 
                               
                               , 
                               
                                 Θ 
                                 _ 
                               
                             
                             ) 
                           
                         
                         ⁢ 
                         
                           
                             
                               ∂ 
                               log 
                             
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             
                               P 
                               ⁡ 
                               
                                 ( 
                                 
                                   
                                     μ 
                                     c 
                                   
                                   | 
                                   
                                     S 
                                     μ 
                                   
                                 
                                 ) 
                               
                             
                           
                           
                             ∂ 
                             
                               S 
                               μ 
                             
                           
                         
                         ⁢ 
                         d 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           μ 
                           c 
                         
                       
                     
                   
                   ∝ 
                   
                     
                       - 
                       
                         1 
                         2 
                       
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           c 
                           = 
                           1 
                         
                         
                           nb 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           classes 
                         
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         ∫ 
                         
                           
                             P 
                             ⁡ 
                             
                               ( 
                               
                                 
                                   
                                     μ 
                                     c 
                                   
                                   | 
                                   
                                     X 
                                     c 
                                   
                                 
                                 , 
                                 
                                   Θ 
                                   _ 
                                 
                               
                               ) 
                             
                           
                           ⁢ 
                           
                             ( 
                             
                               
                                 S 
                                 μ 
                                 
                                   - 
                                   1 
                                 
                               
                               - 
                               
                                 
                                   S 
                                   μ 
                                   
                                     - 
                                     1 
                                   
                                 
                                 ⁢ 
                                 
                                   μ 
                                   c 
                                 
                                 ⁢ 
                                 
                                   μ 
                                   c 
                                   T 
                                 
                                 ⁢ 
                                 
                                   S 
                                   μ 
                                   
                                     - 
                                     1 
                                   
                                 
                               
                             
                             ) 
                           
                           ⁢ 
                           d 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             μ 
                             c 
                           
                         
                       
                     
                   
                   ∝ 
                   
                     
                       - 
                       
                         1 
                         2 
                       
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           nbclasses 
                           × 
                           
                             S 
                             μ 
                             
                               - 
                               1 
                             
                           
                         
                         - 
                         
                           
                             S 
                             μ 
                             
                               - 
                               1 
                             
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 c 
                                 = 
                                 1 
                               
                               
                                 nb 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 classes 
                               
                             
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             
                               ∫ 
                               
                                 
                                   P 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       
                                         
                                           μ 
                                           c 
                                         
                                         | 
                                         
                                           X 
                                           c 
                                         
                                       
                                       , 
                                       
                                         Θ 
                                         _ 
                                       
                                     
                                     ) 
                                   
                                 
                                 ⁢ 
                                 
                                   μ 
                                   c 
                                 
                                 ⁢ 
                                 
                                   μ 
                                   c 
                                   T 
                                 
                                 ⁢ 
                                 d 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   μ 
                                   c 
                                 
                                 ⁢ 
                                 
                                   S 
                                   μ 
                                   
                                     - 
                                     1 
                                   
                                 
                               
                             
                           
                         
                       
                       ) 
                     
                   
                   ∝ 
                   
                     
                       - 
                       
                         1 
                         2 
                       
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           nbclasses 
                           × 
                           
                             S 
                             μ 
                             
                               - 
                               1 
                             
                           
                         
                         - 
                         
                           
                             S 
                             μ 
                             
                               - 
                               1 
                             
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 c 
                                 = 
                                 1 
                               
                               
                                 nb 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 classes 
                               
                             
                             ⁢ 
                             
                               
                                 ( 
                                 
                                   
                                     T 
                                     
                                       μ 
                                       c 
                                     
                                   
                                   + 
                                   
                                     
                                       b 
                                       
                                         μ 
                                         c 
                                       
                                     
                                     ⁢ 
                                     
                                       b 
                                       
                                         μ 
                                         c 
                                       
                                       T 
                                     
                                   
                                 
                                 ) 
                               
                               ⁢ 
                               
                                 S 
                                 μ 
                                 
                                   - 
                                   1 
                                 
                               
                             
                           
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       
                         ∂ 
                         
                           
                             Q 
                             ~ 
                           
                           ⁡ 
                           
                             ( 
                             
                               Θ 
                               , 
                               
                                 Θ 
                                 _ 
                               
                             
                             ) 
                           
                         
                       
                       
                         ∂ 
                         
                           S 
                           μ 
                         
                       
                     
                   
                 
                 = 
                 
                   
                     0 
                     ⇒ 
                     
                       S 
                       μ 
                     
                   
                   = 
                   
                     
                       1 
                       
                         nb 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         classes 
                       
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           c 
                           = 
                           1 
                         
                         
                           nb 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           classes 
                         
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             T 
                             
                               μ 
                               c 
                             
                           
                           + 
                           
                             
                               b 
                               
                                 μ 
                                 c 
                               
                             
                             ⁢ 
                             
                               b 
                               
                                 μ 
                                 c 
                               
                               T 
                             
                           
                         
                         ) 
                       
                     
                   
                 
               
             
           
         
       
       
         
           
             Additionally 
             ⁢ 
             
               : 
             
           
         
       
       
         
           
             
               
                 ∂ 
                 
                   
                     Q 
                     ~ 
                   
                   ⁡ 
                   
                     ( 
                     
                       Θ 
                       , 
                       
                         Θ 
                         _ 
                       
                     
                     ) 
                   
                 
               
               
                 ∂ 
                 
                   S 
                   ω 
                 
               
             
             = 
             
               
                 
                   ∑ 
                   
                     
                       c 
                       = 
                       1 
                     
                     ⁢ 
                     
                       
 
                     
                     ⁢ 
                     
                       i 
                       = 
                       1 
                     
                   
                   
                     
                       nb 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       class 
                     
                     , 
                     
                       
 
                     
                     ⁢ 
                     
                       m 
                       c 
                     
                   
                 
                 ⁢ 
                 
                   ∫ 
                   
                     
                       P 
                       ⁡ 
                       
                         ( 
                         
                           
                             
                               Z 
                               c 
                             
                             | 
                             
                               X 
                               c 
                             
                           
                           , 
                           
                             Θ 
                             _ 
                           
                         
                         ) 
                       
                     
                     ⁢ 
                     
                       
                         
                           ∂ 
                           log 
                         
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           P 
                           ⁡ 
                           
                             ( 
                             
                               
                                 ω 
                                 
                                   c 
                                   , 
                                   i 
                                 
                               
                               | 
                               
                                 S 
                                 ω 
                               
                             
                             ) 
                           
                         
                       
                       
                         ∂ 
                         
                           S 
                           ω 
                         
                       
                     
                     ⁢ 
                     
                       dZ 
                       c 
                     
                   
                 
               
               = 
               
                   
               
               ⁢ 
               
                 
                   
                     
                       ∑ 
                       
                         
                           c 
                           = 
                           1 
                         
                         ⁢ 
                         
                           
 
                         
                         ⁢ 
                         
                           i 
                           = 
                           1 
                         
                       
                       
                         
                           nb 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           class 
                         
                         , 
                         
                           
 
                         
                         ⁢ 
                         
                           m 
                           c 
                         
                       
                     
                     ⁢ 
                     
                       ∫ 
                       
                         ∫ 
                         
                           
                             P 
                             ⁡ 
                             
                               ( 
                               
                                 
                                   μ 
                                   c 
                                 
                                 | 
                                 
                                   
                                     X 
                                     
                                       c 
                                       , 
                                     
                                   
                                   ⁢ 
                                   
                                     
                                       S 
                                       _ 
                                     
                                     u 
                                   
                                 
                               
                               ) 
                             
                           
                           ⁢ 
                           
                             P 
                             ⁡ 
                             
                               ( 
                               
                                 
                                   
                                     ω 
                                     
                                       c 
                                       , 
                                       i 
                                     
                                   
                                   | 
                                   
                                     x 
                                     
                                       c 
                                       , 
                                       i 
                                     
                                   
                                 
                                 , 
                                 
                                   μ 
                                   c 
                                 
                                 , 
                                 
                                   
                                     S 
                                     _ 
                                   
                                   ω 
                                 
                               
                               ) 
                             
                           
                           ⁢ 
                           
                             
                               
                                 ∂ 
                                 log 
                               
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 P 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     
                                       ω 
                                       
                                         c 
                                         , 
                                         i 
                                       
                                     
                                     | 
                                     
                                       S 
                                       ω 
                                     
                                   
                                   ) 
                                 
                               
                             
                             
                               ∂ 
                               
                                 S 
                                 ω 
                               
                             
                           
                           ⁢ 
                           d 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             ω 
                             
                               c 
                               , 
                               i 
                             
                           
                           ⁢ 
                           
                             ∫ 
                             
                               ⋯ 
                               ⁢ 
                               
                                 ∫ 
                                 
                                   
                                     ∏ 
                                     
                                       
                                         j 
                                         = 
                                         1 
                                       
                                       ⁢ 
                                       
                                         
 
                                       
                                       ⁢ 
                                       
                                         j 
                                         ≠ 
                                         i 
                                       
                                     
                                     
                                       m 
                                       c 
                                     
                                   
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   
                                     
                                       P 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           
                                             
                                               ω 
                                               
                                                 c 
                                                 , 
                                                 j 
                                               
                                             
                                             | 
                                             
                                               x 
                                               
                                                 c 
                                                 , 
                                                 j 
                                               
                                             
                                           
                                           , 
                                           
                                             μ 
                                             c 
                                           
                                           , 
                                           
                                             
                                               S 
                                               _ 
                                             
                                             ω 
                                           
                                         
                                         ) 
                                       
                                     
                                     ⁢ 
                                     d 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     
                                       ω 
                                       
                                         c 
                                         , 
                                         j 
                                       
                                     
                                     ⁢ 
                                     d 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     
                                       μ 
                                       c 
                                     
                                     ⁢ 
                                     
                                       
                                         ∑ 
                                         
                                           
                                             c 
                                             = 
                                             1 
                                           
                                           ⁢ 
                                           
                                             
 
                                           
                                           ⁢ 
                                           
                                             i 
                                             = 
                                             1 
                                           
                                         
                                         
                                           
                                             nb 
                                             ⁢ 
                                             
                                                 
                                             
                                             ⁢ 
                                             class 
                                           
                                           , 
                                           
                                             
 
                                           
                                           ⁢ 
                                           
                                             m 
                                             c 
                                           
                                         
                                       
                                       ⁢ 
                                       
                                         ∫ 
                                         
                                           ∫ 
                                           
                                             
                                               P 
                                               ( 
                                               
                                                 
                                                   
                                                     μ 
                                                     c 
                                                   
                                                   | 
                                                   
                                                     X 
                                                     c 
                                                   
                                                 
                                                 , 
                                                 
                                                   
                                                     S 
                                                     _ 
                                                   
                                                   μ 
                                                 
                                               
                                               ) 
                                             
                                             ⁢ 
                                             
                                               P 
                                               ⁡ 
                                               
                                                 ( 
                                                 
                                                   
                                                     
                                                       ω 
                                                       
                                                         c 
                                                         , 
                                                         i 
                                                       
                                                     
                                                     | 
                                                     
                                                       x 
                                                       
                                                         c 
                                                         , 
                                                         i 
                                                       
                                                     
                                                   
                                                   , 
                                                   
                                                     μ 
                                                     c 
                                                   
                                                   , 
                                                   
                                                     
                                                       S 
                                                       _ 
                                                     
                                                     ω 
                                                   
                                                 
                                                 ) 
                                               
                                             
                                             ⁢ 
                                             d 
                                             ⁢ 
                                             
                                                 
                                             
                                             ⁢ 
                                             
                                               μ 
                                               c 
                                             
                                             ⁢ 
                                             
                                               
                                                 
                                                   ∂ 
                                                   log 
                                                 
                                                 ⁢ 
                                                 
                                                     
                                                 
                                                 ⁢ 
                                                 
                                                   P 
                                                   ⁡ 
                                                   
                                                     ( 
                                                     
                                                       
                                                         ω 
                                                         
                                                           c 
                                                           , 
                                                           i 
                                                         
                                                       
                                                       | 
                                                       
                                                         S 
                                                         ω 
                                                       
                                                     
                                                     ) 
                                                   
                                                 
                                               
                                               
                                                 ∂ 
                                                 
                                                   S 
                                                   ω 
                                                 
                                               
                                             
                                             ⁢ 
                                             d 
                                             ⁢ 
                                             
                                                 
                                             
                                             ⁢ 
                                             
                                               ω 
                                               
                                                 c 
                                                 , 
                                                 i 
                                               
                                             
                                             ⁢ 
                                             
                                               
                                                 ∑ 
                                                 
                                                   
                                                     c 
                                                     = 
                                                     1 
                                                   
                                                   ⁢ 
                                                   
                                                     
 
                                                   
                                                   ⁢ 
                                                   
                                                     i 
                                                     = 
                                                     1 
                                                   
                                                 
                                                 
                                                   
                                                     nb 
                                                     ⁢ 
                                                     
                                                         
                                                     
                                                     ⁢ 
                                                     class 
                                                   
                                                   , 
                                                   
                                                     
 
                                                   
                                                   ⁢ 
                                                   
                                                     m 
                                                     c 
                                                   
                                                 
                                               
                                               ⁢ 
                                               
                                                 ∫ 
                                                 
                                                   
                                                     P 
                                                     ( 
                                                     
                                                         
                                                     
                                                     ⁢ 
                                                     
                                                       
                                                         
                                                           ω 
                                                           
                                                             c 
                                                             , 
                                                             i 
                                                           
                                                         
                                                         | 
                                                         
                                                           X 
                                                           c 
                                                         
                                                       
                                                       , 
                                                       
                                                         
                                                           S 
                                                           _ 
                                                         
                                                         μ 
                                                       
                                                       , 
                                                       
                                                         
                                                           S 
                                                           _ 
                                                         
                                                         ω 
                                                       
                                                     
                                                     ) 
                                                   
                                                   ⁢ 
                                                   
                                                     
                                                       
                                                         ∂ 
                                                         log 
                                                       
                                                       ⁢ 
                                                       
                                                           
                                                       
                                                       ⁢ 
                                                       
                                                         P 
                                                         ⁡ 
                                                         
                                                           ( 
                                                           
                                                             
                                                             ω 
                                                             
                                                             c 
                                                             , 
                                                             i 
                                                             
                                                             
                                                             | 
                                                             
                                                             S 
                                                             ω 
                                                             
                                                           
                                                           ) 
                                                         
                                                       
                                                     
                                                     
                                                       ∂ 
                                                       
                                                         S 
                                                         ω 
                                                       
                                                     
                                                   
                                                   ⁢ 
                                                   d 
                                                   ⁢ 
                                                   
                                                       
                                                   
                                                   ⁢ 
                                                   
                                                     ω 
                                                     
                                                       c 
                                                       , 
                                                       i 
                                                     
                                                   
                                                 
                                               
                                             
                                           
                                         
                                       
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   ∝ 
                   
                     
                       - 
                       
                         1 
                         2 
                       
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           ∑ 
                           
                             c 
                             = 
                             1 
                           
                           
                             nb 
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             classes 
                           
                         
                         ⁢ 
                         
                           
                             m 
                             c 
                           
                           ⁢ 
                           
                             S 
                             ω 
                             
                               - 
                               1 
                             
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 
                                   c 
                                   = 
                                   1 
                                 
                                 ⁢ 
                                 
                                   
 
                                 
                                 ⁢ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                               
                               
                                 
                                   nb 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   class 
                                 
                                 , 
                                 
                                   
 
                                 
                                 ⁢ 
                                 
                                   m 
                                   c 
                                 
                               
                             
                             ⁢ 
                             
                               ∫ 
                               
                                 
                                   P 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       
                                         
                                           ω 
                                           
                                             c 
                                             , 
                                             i 
                                           
                                         
                                         | 
                                         
                                           X 
                                           c 
                                         
                                       
                                       , 
                                       
                                         
                                           S 
                                           _ 
                                         
                                         μ 
                                       
                                       , 
                                       
                                         
                                           S 
                                           _ 
                                         
                                         ω 
                                       
                                     
                                     ) 
                                   
                                 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   ω 
                                   
                                     c 
                                     , 
                                     i 
                                   
                                 
                                 ⁢ 
                                 
                                   ω 
                                   
                                     c 
                                     , 
                                     i 
                                   
                                   T 
                                 
                                 ⁢ 
                                 d 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   ω 
                                   
                                     c 
                                     , 
                                     i 
                                   
                                 
                                 ⁢ 
                                 
                                   S 
                                   ω 
                                   
                                     - 
                                     1 
                                   
                                 
                               
                             
                           
                         
                       
                       ) 
                     
                   
                   ∝ 
                   
                       
                   
                   ⁢ 
                   
                     
                       - 
                       
                         1 
                         2 
                       
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             ∑ 
                             
                               c 
                               = 
                               1 
                             
                             
                               nb 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               classes 
                             
                           
                           ⁢ 
                           
                             
                               m 
                               c 
                             
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             
                               S 
                               ω 
                               
                                 - 
                                 1 
                               
                             
                           
                         
                         - 
                         
                           
                             S 
                             ω 
                             
                               - 
                               1 
                             
                           
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 
                                   c 
                                   = 
                                   1 
                                 
                                 ⁢ 
                                 
                                   
 
                                 
                                 ⁢ 
                                 
                                   i 
                                   = 
                                   1 
                                 
                               
                               
                                 
                                   nb 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   class 
                                 
                                 , 
                                 
                                   
 
                                 
                                 ⁢ 
                                 
                                   m 
                                   c 
                                 
                               
                             
                             ⁢ 
                             
                               
                                 ( 
                                 
                                   
                                     T 
                                     
                                       ω 
                                       
                                         c 
                                         , 
                                         i 
                                       
                                     
                                   
                                   + 
                                   
                                     
                                       b 
                                       
                                         ω 
                                         
                                           c 
                                           , 
                                           i 
                                         
                                       
                                     
                                     ⁢ 
                                     
                                       b 
                                       
                                         ω 
                                         
                                           c 
                                           , 
                                           i 
                                         
                                       
                                       T 
                                     
                                   
                                 
                                 ) 
                               
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 S 
                                 ω 
                                 
                                   - 
                                   1 
                                 
                               
                             
                           
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       
                         ∂ 
                         
                           
                             Q 
                             ~ 
                           
                           ⁡ 
                           
                             ( 
                             
                               Θ 
                               , 
                               
                                 Θ 
                                 _ 
                               
                             
                             ) 
                           
                         
                       
                       
                         ∂ 
                         
                           S 
                           ω 
                         
                       
                     
                   
                 
                 = 
                 
                   
                     0 
                     ⇒ 
                     
                       S 
                       ω 
                     
                   
                   = 
                   
                     
                       1 
                       
                         
                           ∑ 
                           
                             c 
                             = 
                             1 
                           
                           
                             
                               
                                 
                                   nb 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   classes 
                                 
                               
                             
                             
                               
                                 
                                   m 
                                   c 
                                 
                               
                             
                           
                         
                         ⁢ 
                         
                             
                         
                       
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           
                             c 
                             = 
                             1 
                           
                           ⁢ 
                           
                             
 
                           
                           ⁢ 
                           
                             i 
                             = 
                             1 
                           
                         
                         
                           
                             nb 
                             ⁢ 
                             
                                 
                             
                             ⁢ 
                             class 
                           
                           , 
                           
                             
 
                           
                           ⁢ 
                           
                             m 
                             c 
                           
                         
                       
                       ⁢ 
                       
                         ( 
                         
                           
                             T 
                             
                               ω 
                               
                                 c 
                                 , 
                                 i 
                               
                             
                           
                           + 
                           
                             
                               b 
                               
                                 ω 
                                 
                                   c 
                                   , 
                                   i 
                                 
                               
                             
                             ⁢ 
                             
                               b 
                               
                                 ω 
                                 
                                   c 
                                   , 
                                   i 
                                 
                               
                               T 
                             
                           
                         
                         ) 
                       
                     
                   
                 
               
             
           
         
       
     
     The expectation-maximization algorithm is performed iteratively by successively computing at step  210  the variables T μ     c   , b μ     c   , T ω     c,i    and b ω     c,i   , and by adapting the values of S μ  and S ω —and hence of  Θ —at step  220  until convergence. At each new iteration of step  210 , the new value of  Θ  obtained at preceding step  220  is reused.