Supervised learning with incomplete data via sparse representations

1 Introduction . In many machine learning applications sometimes the measurements are noisy or affected by artifacts , resulting in incomplete data samples . Examples of this situation include : self-driving vehicle or robot where objects in the view field can be partially occluded ; recommendation systems built from the information gathered by different users where not all the users have fully completed their forms ; or medical datasets where typically not all tests can be performed on all patients . A classical approach of supervised learning with missing entries is to apply entry imputation as a preprocessing step , followed by standard training based on these reconstructions ( Little & Rubin , 2014 ) . For example , some techniques compute missing entries using statistical methods , such as the “ mean ” , “ regression ” and “ multiple ” imputation methods ( Little & Rubin , 2014 ) . Other completion methods are based on machine learning ideas by estimating missing entries through K-nearest neighbor ( Batista & Monard , 2002 ) , Self Organization Maps ( SOM ) ( Fessant & Midenet , 2002 ) , multilayer or recurrent neural networks ( Yoon & Lee , 1999 ; Bengio & Gingras , 1995 ) , tensor completion algorithms ( Solé-Casals et al. , 2018 ) and others ( García-Laencina et al. , 2009 ) . In ( Huang et al. , 2018 ) , a technique for completing the matrix of features was proposed by imposing low-rankness and incorporating the label information into the completion process . A different approach is to avoid direct imputation of lost inputs and rely on a probabilistic model of input data , learning model parameters through the Expectation Maximization ( EM ) algorithm and building a Bayesian classification ( Ghahramani & Jordan , 1993 ; Little & Rubin , 2014 ) . However , the latter “ model-based ” approach has the disadvantage that it requires a good probabilistic data model , which is usually not available , especially for real-world applications such as those involving natural images . On the other hand , during the last few years in the signal processing community , there has been a rapid development of the Compressed Sensing ( CS ) theory and sparse coding algorithms that exploit the redundancy of natural signals ( Candes et al. , 2006 ; Donoho , 2006 ; Eldar & Kutyniok , 2012 ; Gregor & LeCun , 2010 ) . A particular case of CS is the signal completion problem , i.e . the reconstruction of signals from incomplete measurements . Several algorithms for matrix and tensor ( multidimensional signals ) completion have been recently proposed relying on different low-dimensional manifold models that are fit to the available entries , allowing for an extrapolation of the unobserved entries . This category of models ranges from sparse representation of vectors ( Aharon et al. , 2006 ; Fadili et al. , 2008 ) and tensors ( Caiafa & Cichocki , 2012 ) to low-rank matrix ( Candes & Tao , 2010 ) and tensor decompositions ( Liu et al. , 2012 ; Yuan et al. , 2018 ) . The success of sparse representations in many signal processing applications motivated researchers to use dictionary learning techniques for data classification ( Mairal et al. , 2008 ) , either using class-specific dictionaries ( Ramirez et al. , 2010 ; Sprechmann & Sapiro , 2010 ) , or using a single one for all classes ( Tosic & Frossard , 2011 ) . In this paper , the goal is to develop a method for training a classifier using incomplete data samples with their labels and to identify the conditions under which such a classifier performs as good as the ideal classifier , i.e . the one that could be obtained from complete observations . We propose to use the sparse representation model for data vectors , and analyze the limitations of the sequential approach , i.e . imputation followed by training ( section 2 ) . In section 3 we introduce the simultaneous classification and coding approach , which consists of incorporating a sparse data representation model into a cost function optimized for training the classifier and , at the same time , finding the best sparse representation of the observed data . We provide theoretical conditions under which we can guarantee that the obtained classifier is as good as the ideal classifier . In section 4 , a computational method is presented to train a classifier on incomplete data ; in section 5 , extensive experimental results are presented using synthetic and well known benchmark datasets illustrating the effectiveness of our proposed algorithms ; and finally , in section 6 , the main conclusions are outlined . 1.1 Preliminary definitions and problem formulation . We assume a supervised learning scenario with vector samples and their labels { xi , yi } , i = 1 , 2 , . . . I , xi ∈ RN and yi ∈ { 0 , 1 , . . . , C − 1 } ( C classes ) , so that we only have access to subsets of entries in each data vector along with their labels , which is denoted by { x̃i , yi } , x̃i ∈ RM with M < N . The success of CS theory is based on the fact that sparse coding of natural signals is nearly ubiquitous . It is found , for example , in the way that neurons encode sensory information ( Olshausen & Field , 1996 ; 1997 ; Lee et al. , 2007 ) . We define the set of all K-sparse vectors ΣLK = { s ∈ RL such that ‖s‖0 ≤ K } ( containing at most K non-zero entries ) and assume that data vectors xi admit K-sparse representations over an unknown dictionary D ∈ RN×L ( L ≥ N ) , i.e . xi = Dsi , with si ∈ ΣLK . The columns di ∈ RN of a dictionary are called “ atoms ” because we can express any data vector as a combination of at most K atoms . Without losing generality , to simplify the mathematical treatment , we shuffle the entries of the full vector xi to have the observed values placed in the upper M positions , so that we can write : xi = [ x̃i zi ] , with zi ∈ R ( N−M ) . ( 1 ) However , our results are valid for any pattern of missing entries for each data vector . Accordingly , we define Di ∈ RN×L as the shuffled version of D , such that it stacks in the top M rows those corresponding to the observed entries : Di = [ D̃i Ei ] , with D̃i ∈ RM×L and Ei ∈ R ( N−M ) ×L . ( 2 ) Some important uniqueness properties of sparse vector reconstructions are characterized by the Restricted Isometry Property ( RIP ) of a matrix , which is defined as follows : a matrix A ∈ RN×L satisfies the RIP of order K if there exists δK ∈ [ 0 , 1 ) such that ( 1−δK ) ‖s‖22 ≤ ‖As‖22 ≤ ( 1+δK ) ‖s‖22 , holds for all s ∈ ΣLK . RIP was introduced in the context of CS by Candes and Tao ( Candès & Tao , 2005 ) and characterizes matrices which are nearly orthonormal when operating on sparse vectors . Let us assume that a perfect classifier pΘ ( y | x ) in a two classes scenario ( yi ∈ { 0 , 1 } ) , e.g . a logistic regression or deep neural network , can be trained on the complete dataset { xi , yi } , such that 0.5 < pΘ ( yi = 1 | xi ) ≤ 1.0 and 0 ≤ pΘ ( yi = 0 | xi ) ≤ 0.5 , ∀i = 1 , 2 , . . . , I , where Θ is the set of trained parameters . We want to develop a method to estimate the classifier parameters Θ̂ using only the incomplete data { x̃i , yi } and to identify the conditions under which such classifier is compatible with the ideal classifier . In the following section we will show that , when data imputation based on sparsity followed by training is used , some limitations are clearly identified . 2 The problem of training after data imputation . First , we will provide some intuition about the limitation of the sparsity-based imputation or “ sequential ” approach through a toy example . We consider the classification of hand-written digit images belonging to two classes : the numbers “ 3 ” and “ 8 ” . We will assume that they admit 2-sparse representations over a dictionary and only the right halves of these digits are observed . Let us consider two example vectors xi and xj belonging to classes “ 3 ” and “ 8 ” , respectively , as shown in Figure 1 ( a-d ) . Here , we are clearly faced with the indiscernible observations problem because our observations of two vectors from different classes are identical ( x̃i = x̃j ) . It is obvious that at least two possible 2-sparse representations for the observed data exist , let ’ s say x̃i = D̃isi = D̃is′i , with si 6= s′i . Since the solution is not unique , we are not able to correctly reconstruct the original data vectors based only on the observations and the sparsity assumption . If the K-sparse representation of the observations x̃i are unique , then xi can be perfectly reconstructed from the incomplete observations and the classifier can be successfully trained using these reconstructions , but unfortunately this is not the case most of the times . In the particular case where the dictionary D is known in advance , there are some conditions on the sampling patterns based on properties of matrices such as coherence , spark or RIP that can guarantee a correct reconstruction ( Eldar & Kutyniok , 2012 ) . However , these conditions are difficult to meet in practice . Moreover , in the more general case where the dictionary D is unknown and needs to be learned from data , it is even more difficult to obtain reconstructions of partially observed data vectors that are well separated . For example , even in the case where complete data vectors are well separated , it could happen that their reconstructions become difficult to separate , leading to suboptimal classifiers . In this paper , motivated by the limitation of the sequential approach , we propose to incorporate the class information of incomplete samples in order to learn simultaneously their sparsest representations and the classifier , such that the classification error of the reconstructions is minimized . 3 The simultaneous classification and coding approach . In the previous section , we showed that sparsity-based imputation methods using only the information on available entries are prone to fail because the non-uniqueness of solutions can make the training of a good classifier an impossible task . It is interesting to note that we could solve this problem by incorporating from the beginning the information of the class to which the incomplete data vectors belong to . Let us consider the toy example of previous section , for which we assumed the classes are linearly separable and each data vector admits a K-sparse representation over a dictionary D ∈ RN×L . A two-dimensional simplified visualization for this example is provided in Figure 1 ( e ) . When we apply a sparsity-based reconstruction algorithm on observations x̃i , and x̃j such that x̃i = x̃j and , assuming for now that dictionary D is known , we search for the K-sparse representation that is compatible with the measurements , i.e . we find s ∈ ΣLK such that x̃i = Dis . In this case , there are at least two equally acceptable solutions si and sj corresponding to vectors xi and xj , respectively . If we were to make the wrong decision of assigning sj to xi and si to xj , then the resulting set of reconstructed vectors would not be linearly separable , which is illustrated in Figure 1 ( f ) , therefore it would be impossible to find a linear classifier . The key observation here is that , in order to assign the proper sparse representation we could also look at the label information yi and yj , and make the solution unique by discarding any K-sparse representation resulting in not separable vector reconstructions . This suggests that we would need to train the classifier and find the proper representation not only as sparse as possible but also providing the best separation of classes . In other words , we should simultaneously learn the optimal classifier and find sparse representation of observations . Next , we theoretically analyze the case of using a logistic regression classifier and demonstrate that , if we are able to train a classifier on incomplete observations such that their sparse representations are well separated by a hyperplane , then the same classifier correctly separates the original ( unobserved ) data vectors .

Summary: This manuscript studies supervised learning with incomplete observation of the features, assuming a low-rank structure in the full set of features. The work tackles the problem with a global cost function that optimizes the classifier on observations reconstructed with a dictionary penalized jointly for sparse coding. Importantly, the dictionary is optimized both for the supervised-learning task (the classifier) and the sparse coding. The manuscript contributes a theoretical argument showing with sparse-recovery arguments that if the data can be decomposed sparsely, the incomplete feature prediction problem can perform as well as the complete one. Finally, it performs a empirical study on simulated data as well as computer-vision problems with random missing pixels as well as occlusions, and using for the classifier logistic regression as well as classic neural network architectures.