Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach

1 Introduction . Learning a mapping from observation x ( a vector of attribute features ) to label y is a fundamental and pervasive problem in ML community , with extensive applications spanning from classification/regression tasks for tabular data to advertisement click prediction [ 23 , 38 , 13 , 37 , 27 ] , item recommendation [ 26 , 52 , 14 , 53 ] , question answering [ 12 , 11 ] , or AI more broadly . Existing approaches focus on a fixed input feature space shared by training and test data . Nevertheless , practical ML systems interact with a dynamic open-world where features are incrementally collected . For example , in recommender/advertisement systems , there often occur new user profile features unseen before for current prediction tasks . Also , with the advances of multi-modal recognition [ 45 ] and federated learning [ 29 ] , it is a requirement for a model trained with partial features to incorporate new features from other fields for decision-making on a target task . A challenge stems from the fact that off-the-shelf neural network models can not deal with new features without re-training on new data . As shown in Fig . 1 ( a ) , a neural network builds a mapping from input features to a hidden representation through a weight matrix in the first layer . Given new features as input , the network needs to be augmented with new weight parameters and can not map the new features to a desirable position in latent space if without re-training . ⇤Junchi Yan is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , model re-training would be tricky and bring up several issues . First , re-training a model with both previous and new features would be highly time-consuming and can not meet the requirement for online systems . Alternatively , one can re-train a trained model only on new features , which will induce risks for over-fitting new data or forgetting previous data . Different from machines , fortunately , humans are often equipped with the ability for extrapolating to unseen features and distill the knowledge in new information for solving a target task without any re-training on new data . The inherent gap between existing ML approaches and human intelligence raises a research question : Can we design a ML model that is trained on one set of features and able to generalize to combine new unseen features for the same task without further training ? To find desirable solutions to this non-trivial question is challenging . Let us resort to how humans think and act in physical world scenarios . Imagine that we are well trained on predicting a person ’ s income with one ’ s age and occupation from historical records . Now we are asked to estimate the income of a new person with his/her age , occ . and education ( a new feature ) . We can modularize the process in brain systems that distill and leverage the knowledge in new observations into four steps , as shown in Fig . 1 ( b ) . 1 ) Perception : the observations are recognized by our senses ; 2 ) Abstraction : the perceived information is aligned with concepts in our cognition ; 3 ) Reasoning : we search similar concepts and observations in our memory to understand and absorb the new knowledge ; 4 ) Decision : with new understanding and abstraction , we make final decisions for the prediction . The above human ’ s thinking process inspires the methodology in our paper where we propose a new learning paradigm for open-world feature extrapolation . Our proposed framework contains two modules : a backbone network , which can be a feedforward neural network , and a graph neural network . 1 ) The backbone network first maps input features to embeddings , which can be seen as perception from observation . 2 ) We then treat observed data matrix ( each row represents a feature vector for one instance ) as a feature-data bipartite graph , which explicitly define proximity and locality structures among features and instances . Then a graph neural network is harnessed for neural message passing over adjacent features and instances in a latent space ( abstraction ) . 3 ) The GNN will inductively compute embeddings for new features based on those of existing ones , mimicking the reasoning process from familiar concepts to new ones in our brain . 4 ) The newly obtained embeddings that capture both semantics and feature-level relations will be used to obtain hidden representations of new data with unseen features and make final decisions . To endow the model with ability for extrapolation to new features , we propose two training algorithms , one by self-supervised learning and one by inductive learning . The proposed model and its learning algorithm can easily scale to large-scale dataset ( with millions of features and instances ) via minibatch training . We also provide a theoretical analysis on the generalization error on test data with new features , which allows us to dissect the impact of training features and algorithms on generalization performance . To verify our approach , we conduct extensive experiments on several real-world datasets , including six classification datasets from biology , engineering and social domains with diverse features , as well as two large-scale datasets for advertisement click prediction . Our model is trained on training data using partial features and tested on test data with a mixture of seen and unseen features . The results demonstrate that 1 ) our approach consistently outperform models not using new features for inference ; 2 ) our approach achieves averagely 29.8 % higher Accuracy than baseline methods using KNN , pooling or local aggregation for feature extrapolation ; 3 ) our approach even exceeds models using incremental training on new features , yielding average 4.1 % higher Accuracy . Our contributions are : 1 ) We formulate open-world feature extrapolation problem and show that it is feasible to extend neural models for extrapolating to new features without re-training ; 2 ) We propose a new graph-learning model and two training algorithms for feature extrapolation problem ; 3 ) Our theoretical analysis shows that the generalization error for data with new features relies on the number of training features and the randomness in training algorithms ; 4 ) We conduct comprehensive experiments and show the effectiveness , applicality and scalability of proposed method . 2 Methodology . We focus on attribute features as input in this paper . An input instance is a vector ri = [ rim ] dm=1 2 Rd where each entry rim denotes a raw feature ( like age , occ. , edu. , etc. ) . If rim is a discrete/categorical raw feature with Rm possible values , its space is an integer set { 0 , 1 , · · · , Rm 1 } . If rim0 is a continuous one , a common practice is to convert it into a discrete feature within space { 0 , 1 , · · · , Rm0 1 } by evenly division [ 24 ] or log transformation [ 23 ] . We call Rm as cardinality for m-th raw feature . An effective way to handle attribute features is via one-hot encoding [ 13 , 24 , 17 ] . For rim with cardinality Rm we convert it into a Rm-dimensional one-hot vector xmi where the unique 1 indexes the value . In this way , one can convert an input ri into a concatenation of one-hot vectors : xi = ⇥ x 1 i , x2 i , · · · , xd i ⇤ , where 1 8m d , xm i 2 { 0 , 1 } Rm is a one-hot vector . ( 1 ) Use xij to denote the j-th entry of xi and we call each xij as feature in this paper . Assume D = P d m=1 Rm denotes the number of features and , as a reminder , d is the number of raw features . We next give a formal definition for open-world feature extrapolation problem in this paper : given training data { ( xi , yi ) } i2Itr where xi 2 Xtr = { 0 , 1 } D , yi 2 Y and Itr is a set of indices , we aim to learn a model that can generalize to test data { xi0 , yi0 } i02Ite where xi0 2 Xte = { 0 , 1 } D , yi0 2 Y and Ite is another set of indices . We term Xtr as training feature space and Xte as test feature space . We assume 1 ) the label space Y is shared by training and test data , and 2 ) Xtr ⇢ Xte , i.e. , test feature space is an extension of training feature space . The feature space expansion stems from two possible causes : 1 ) there appear new raw features incrementally collected from other fields ( i.e. , d increases ) or 2 ) there appear new values out of the known space of existing raw features ( i.e. , Rm increases ) . 2.1 Proposed Model . Our model contains three parts : 1 ) feature representation that builds a bipartite feature-data graph from input data ; 2 ) a backbone network which is essentially a neural network model that predicts the labels when fed with input data ; 3 ) a GNN model that inductively compute features ’ embeddings based on their proximity and local structures to achieve feature extrapolation . Feature Representation with Graphs . We stack the feature vectors of all the training data as a matrix Xtr = [ xi ] i2Itr 2 { 0 , 1 } N⇥D where N = |Itr| . Then we treat each feature and instance as nodes and construct a bipartite graph between them . Formally , we define a node set Ftr [ Itr where Ftr = { fj } Dj=1 with fj the j-th feature and Itr = { oi } Ni=1 with oi the i-th instance in training set . The binary matrix Xtr constitutes an adjacency matrix where the non-zero entries indicate edges connecting two nodes in Ftr and Itr , respectively . The induced feature-data bipartite graph will play an important role in our extrapolation approach . The representation is flexible for variable-size feature set , enabling our model to handle test data xi0 2 { 0 , 1 } D which gives Xte = [ xi0 ] i02Ite . Backbone Networks . We next consider a prediction model h✓ ( · ) as a backbone network that maps data features xi to predicted label ŷi . Without loss of generality , a default choice for h✓ is a feedforward neural network . The first layer serves as an embedding layer which shrinks xi into a Hdimensional hidden vector zi = xiW where W 2 RD⇥H denotes a weight matrix . The subsequent network ( called classifier ) is often a stack of neural layers that predicts label ŷi = FFN ( zi ; ) . We use the notation ŷi = h ( xi ; , W ) to highlight two sets of parameters and ✓ = [ , W ] . Notice that the matrix multiplication in the embedding layer is equivalent to a two-step procedure : 1 ) a lookup of feature embeddings and 2 ) a permutation-invariant aggregation . More specifically , we consider W as a stack of weight vectors W = [ wj ] Dj=1 where wj 2 R1⇥H corresponds to the embedding of feature fj . The non-zero entries in xi will index the corresponding rows of W and induce a set of embeddings { zm i } d m=1 where zmi is the embedding given by xmi ( i.e. , one-hot vector of m-th raw feature ) . Then the hidden vector of i-th instance can be obtained by aggregation , i.e. , zi = P d m=1 z m i which is permutation-invariant w.r.t . the order of feature embeddings in { zm i } d m=1 . A more intuitive illustration is presented in Fig . 2 . The permutation-invariant property opens a way for handling variable-length feature vectors xi0 [ 56 ] . Essentially , on condition that we have embeddings for input features , we can add them up to get a fixed-dimensional hidden representation zi0 as input for the subsequent classifier . Therefore , the problem boils down to learning feature embeddings , especially how to extrapolate for embeddings of new features based on those of existing ones . Remark . Instead of using sum aggregation , some existing architectures consider concatenation of d embedding vectors zm i ’ s , which is essentially equivalent to sum aggregation ( see Appendix A for more details ) . Therefore , the permutation-invariant property holds for widely adopted deep models . GNN for Feature Extrapolation . We proceed to propose a graph neural networks ( GNN ) model for embedding learning with the feature-data graph . Our key insight is that the bipartite graph explicitly embodies features ’ co-occurrence in observed instances , which reflects the proximity among features . Once we conduct message passing for feature embeddings over the graph structures , the embeddings of similar features can be leveraged to compute and update each feature ’ s embedding . The model can learn to extrapolate for new features ’ embeddings using those of existing features with locality structures in a data-driven manner . The message passing over the defined graph representation is inductive w.r.t . variable-sized feature nodes and instance nodes , which enables the model to tackle new feature space with distinct feature sizes and supports . Specifically , we consider the embeddings wj as an initial state w ( 0 ) j for node fj in Ftr . The initial states of instance nodes are set as zero vectors with equal dimension as the feature nodes , i.e . s ( 0 ) i = 0 . The interaction between two sets of nodes { wj } Dj=1 and { si } Ni=1 can be modeled via graph neural networks where the node states in the l-th layer are updated by s ( l ) i = P ( l ) COMB ⇣ s ( l 1 ) i , AGG ( { w ( l 1 ) k } |8k , xik = 1 ) ⌘ , w ( l ) j = P ( l ) COMB ⇣ w ( l 1 ) j , AGG ( { s ( l 1 ) k } |8k , xjk = 1 ) ⌘ , ( 2 ) where P ( l ) 2 RH⇥H is a weight matrix and we do not use non-linearity since it would degrade the performance empirically . For any new feature fj0 in test data , we can set its initial state as a zero vector w ( 0 ) j0 = 0 . The GNN model outputs updated embeddings for feature nodes and we further use them as the feature embeddings in the backbone network . Fig . 3 presents the feedforward computation of proposed model . Formally , with L-layer GNN , the GNN network g gives updated feature embeddings Ŵ = [ w ( L ) j ] D j=1 = g ( W , X ; ! ) and then the backbone network outputs prediction ŷi = h ( xi ; , Ŵ ) where X = Xtr for training , X = Xte for test and ! = { P ( l ) } Ll=1 denotes g ’ s parameters , .

Summary: This paper tackles the unseen features with their extrapolation method from seen features, which is an important problem in machine learning. * The authors first formulate the problem of open-world feature extrapolation that aims to deal with features unobserved at training while observed at inference. * The authors propose to generate embedding for new features with inductive graph neural networks, where the graph-structured is constructed from features and instances of the observed training data. * The authors theoretically analyze the generalization gap of the proposed method, and show that the number of features and the size of subsets for training data are important ingredients.