Supervised learning with incomplete data via sparse representations

1 Introduction . In many machine learning applications sometimes the measurements are noisy or affected by artifacts , resulting in incomplete data samples . Examples of this situation include : self-driving vehicle or robot where objects in the view field can be partially occluded ; recommendation systems built from the information gathered by different users where not all the users have fully completed their forms ; or medical datasets where typically not all tests can be performed on all patients . A classical approach of supervised learning with missing entries is to apply entry imputation as a preprocessing step , followed by standard training based on these reconstructions ( Little & Rubin , 2014 ) . For example , some techniques compute missing entries using statistical methods , such as the “ mean ” , “ regression ” and “ multiple ” imputation methods ( Little & Rubin , 2014 ) . Other completion methods are based on machine learning ideas by estimating missing entries through K-nearest neighbor ( Batista & Monard , 2002 ) , Self Organization Maps ( SOM ) ( Fessant & Midenet , 2002 ) , multilayer or recurrent neural networks ( Yoon & Lee , 1999 ; Bengio & Gingras , 1995 ) , tensor completion algorithms ( Solé-Casals et al. , 2018 ) and others ( García-Laencina et al. , 2009 ) . In ( Huang et al. , 2018 ) , a technique for completing the matrix of features was proposed by imposing low-rankness and incorporating the label information into the completion process . A different approach is to avoid direct imputation of lost inputs and rely on a probabilistic model of input data , learning model parameters through the Expectation Maximization ( EM ) algorithm and building a Bayesian classification ( Ghahramani & Jordan , 1993 ; Little & Rubin , 2014 ) . However , the latter “ model-based ” approach has the disadvantage that it requires a good probabilistic data model , which is usually not available , especially for real-world applications such as those involving natural images . On the other hand , during the last few years in the signal processing community , there has been a rapid development of the Compressed Sensing ( CS ) theory and sparse coding algorithms that exploit the redundancy of natural signals ( Candes et al. , 2006 ; Donoho , 2006 ; Eldar & Kutyniok , 2012 ; Gregor & LeCun , 2010 ) . A particular case of CS is the signal completion problem , i.e . the reconstruction of signals from incomplete measurements . Several algorithms for matrix and tensor ( multidimensional signals ) completion have been recently proposed relying on different low-dimensional manifold models that are fit to the available entries , allowing for an extrapolation of the unobserved entries . This category of models ranges from sparse representation of vectors ( Aharon et al. , 2006 ; Fadili et al. , 2008 ) and tensors ( Caiafa & Cichocki , 2012 ) to low-rank matrix ( Candes & Tao , 2010 ) and tensor decompositions ( Liu et al. , 2012 ; Yuan et al. , 2018 ) . The success of sparse representations in many signal processing applications motivated researchers to use dictionary learning techniques for data classification ( Mairal et al. , 2008 ) , either using class-specific dictionaries ( Ramirez et al. , 2010 ; Sprechmann & Sapiro , 2010 ) , or using a single one for all classes ( Tosic & Frossard , 2011 ) . In this paper , the goal is to develop a method for training a classifier using incomplete data samples with their labels and to identify the conditions under which such a classifier performs as good as the ideal classifier , i.e . the one that could be obtained from complete observations . We propose to use the sparse representation model for data vectors , and analyze the limitations of the sequential approach , i.e . imputation followed by training ( section 2 ) . In section 3 we introduce the simultaneous classification and coding approach , which consists of incorporating a sparse data representation model into a cost function optimized for training the classifier and , at the same time , finding the best sparse representation of the observed data . We provide theoretical conditions under which we can guarantee that the obtained classifier is as good as the ideal classifier . In section 4 , a computational method is presented to train a classifier on incomplete data ; in section 5 , extensive experimental results are presented using synthetic and well known benchmark datasets illustrating the effectiveness of our proposed algorithms ; and finally , in section 6 , the main conclusions are outlined . 1.1 Preliminary definitions and problem formulation . We assume a supervised learning scenario with vector samples and their labels { xi , yi } , i = 1 , 2 , . . . I , xi ∈ RN and yi ∈ { 0 , 1 , . . . , C − 1 } ( C classes ) , so that we only have access to subsets of entries in each data vector along with their labels , which is denoted by { x̃i , yi } , x̃i ∈ RM with M < N . The success of CS theory is based on the fact that sparse coding of natural signals is nearly ubiquitous . It is found , for example , in the way that neurons encode sensory information ( Olshausen & Field , 1996 ; 1997 ; Lee et al. , 2007 ) . We define the set of all K-sparse vectors ΣLK = { s ∈ RL such that ‖s‖0 ≤ K } ( containing at most K non-zero entries ) and assume that data vectors xi admit K-sparse representations over an unknown dictionary D ∈ RN×L ( L ≥ N ) , i.e . xi = Dsi , with si ∈ ΣLK . The columns di ∈ RN of a dictionary are called “ atoms ” because we can express any data vector as a combination of at most K atoms . Without losing generality , to simplify the mathematical treatment , we shuffle the entries of the full vector xi to have the observed values placed in the upper M positions , so that we can write : xi = [ x̃i zi ] , with zi ∈ R ( N−M ) . ( 1 ) However , our results are valid for any pattern of missing entries for each data vector . Accordingly , we define Di ∈ RN×L as the shuffled version of D , such that it stacks in the top M rows those corresponding to the observed entries : Di = [ D̃i Ei ] , with D̃i ∈ RM×L and Ei ∈ R ( N−M ) ×L . ( 2 ) Some important uniqueness properties of sparse vector reconstructions are characterized by the Restricted Isometry Property ( RIP ) of a matrix , which is defined as follows : a matrix A ∈ RN×L satisfies the RIP of order K if there exists δK ∈ [ 0 , 1 ) such that ( 1−δK ) ‖s‖22 ≤ ‖As‖22 ≤ ( 1+δK ) ‖s‖22 , holds for all s ∈ ΣLK . RIP was introduced in the context of CS by Candes and Tao ( Candès & Tao , 2005 ) and characterizes matrices which are nearly orthonormal when operating on sparse vectors . Let us assume that a perfect classifier pΘ ( y | x ) in a two classes scenario ( yi ∈ { 0 , 1 } ) , e.g . a logistic regression or deep neural network , can be trained on the complete dataset { xi , yi } , such that 0.5 < pΘ ( yi = 1 | xi ) ≤ 1.0 and 0 ≤ pΘ ( yi = 0 | xi ) ≤ 0.5 , ∀i = 1 , 2 , . . . , I , where Θ is the set of trained parameters . We want to develop a method to estimate the classifier parameters Θ̂ using only the incomplete data { x̃i , yi } and to identify the conditions under which such classifier is compatible with the ideal classifier . In the following section we will show that , when data imputation based on sparsity followed by training is used , some limitations are clearly identified . 2 The problem of training after data imputation . First , we will provide some intuition about the limitation of the sparsity-based imputation or “ sequential ” approach through a toy example . We consider the classification of hand-written digit images belonging to two classes : the numbers “ 3 ” and “ 8 ” . We will assume that they admit 2-sparse representations over a dictionary and only the right halves of these digits are observed . Let us consider two example vectors xi and xj belonging to classes “ 3 ” and “ 8 ” , respectively , as shown in Figure 1 ( a-d ) . Here , we are clearly faced with the indiscernible observations problem because our observations of two vectors from different classes are identical ( x̃i = x̃j ) . It is obvious that at least two possible 2-sparse representations for the observed data exist , let ’ s say x̃i = D̃isi = D̃is′i , with si 6= s′i . Since the solution is not unique , we are not able to correctly reconstruct the original data vectors based only on the observations and the sparsity assumption . If the K-sparse representation of the observations x̃i are unique , then xi can be perfectly reconstructed from the incomplete observations and the classifier can be successfully trained using these reconstructions , but unfortunately this is not the case most of the times . In the particular case where the dictionary D is known in advance , there are some conditions on the sampling patterns based on properties of matrices such as coherence , spark or RIP that can guarantee a correct reconstruction ( Eldar & Kutyniok , 2012 ) . However , these conditions are difficult to meet in practice . Moreover , in the more general case where the dictionary D is unknown and needs to be learned from data , it is even more difficult to obtain reconstructions of partially observed data vectors that are well separated . For example , even in the case where complete data vectors are well separated , it could happen that their reconstructions become difficult to separate , leading to suboptimal classifiers . In this paper , motivated by the limitation of the sequential approach , we propose to incorporate the class information of incomplete samples in order to learn simultaneously their sparsest representations and the classifier , such that the classification error of the reconstructions is minimized . 3 The simultaneous classification and coding approach . In the previous section , we showed that sparsity-based imputation methods using only the information on available entries are prone to fail because the non-uniqueness of solutions can make the training of a good classifier an impossible task . It is interesting to note that we could solve this problem by incorporating from the beginning the information of the class to which the incomplete data vectors belong to . Let us consider the toy example of previous section , for which we assumed the classes are linearly separable and each data vector admits a K-sparse representation over a dictionary D ∈ RN×L . A two-dimensional simplified visualization for this example is provided in Figure 1 ( e ) . When we apply a sparsity-based reconstruction algorithm on observations x̃i , and x̃j such that x̃i = x̃j and , assuming for now that dictionary D is known , we search for the K-sparse representation that is compatible with the measurements , i.e . we find s ∈ ΣLK such that x̃i = Dis . In this case , there are at least two equally acceptable solutions si and sj corresponding to vectors xi and xj , respectively . If we were to make the wrong decision of assigning sj to xi and si to xj , then the resulting set of reconstructed vectors would not be linearly separable , which is illustrated in Figure 1 ( f ) , therefore it would be impossible to find a linear classifier . The key observation here is that , in order to assign the proper sparse representation we could also look at the label information yi and yj , and make the solution unique by discarding any K-sparse representation resulting in not separable vector reconstructions . This suggests that we would need to train the classifier and find the proper representation not only as sparse as possible but also providing the best separation of classes . In other words , we should simultaneously learn the optimal classifier and find sparse representation of observations . Next , we theoretically analyze the case of using a logistic regression classifier and demonstrate that , if we are able to train a classifier on incomplete observations such that their sparse representations are well separated by a hyperplane , then the same classifier correctly separates the original ( unobserved ) data vectors .

Summary: The authors propose a scheme for simultaneous dictionary learning and classification based on sparse representation of the data within the learned dictionary. Their goal is achieved via optimization of a three-part training cost function that explicitly models the accuracy and sparsity of the sparse model, simultaneously with usual classification error minimization. An alternating optimization algorithm is used, alternating between sparse representations and other parameters.