Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach

1 Introduction . Learning a mapping from observation x ( a vector of attribute features ) to label y is a fundamental and pervasive problem in ML community , with extensive applications spanning from classification/regression tasks for tabular data to advertisement click prediction [ 23 , 38 , 13 , 37 , 27 ] , item recommendation [ 26 , 52 , 14 , 53 ] , question answering [ 12 , 11 ] , or AI more broadly . Existing approaches focus on a fixed input feature space shared by training and test data . Nevertheless , practical ML systems interact with a dynamic open-world where features are incrementally collected . For example , in recommender/advertisement systems , there often occur new user profile features unseen before for current prediction tasks . Also , with the advances of multi-modal recognition [ 45 ] and federated learning [ 29 ] , it is a requirement for a model trained with partial features to incorporate new features from other fields for decision-making on a target task . A challenge stems from the fact that off-the-shelf neural network models can not deal with new features without re-training on new data . As shown in Fig . 1 ( a ) , a neural network builds a mapping from input features to a hidden representation through a weight matrix in the first layer . Given new features as input , the network needs to be augmented with new weight parameters and can not map the new features to a desirable position in latent space if without re-training . ⇤Junchi Yan is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , model re-training would be tricky and bring up several issues . First , re-training a model with both previous and new features would be highly time-consuming and can not meet the requirement for online systems . Alternatively , one can re-train a trained model only on new features , which will induce risks for over-fitting new data or forgetting previous data . Different from machines , fortunately , humans are often equipped with the ability for extrapolating to unseen features and distill the knowledge in new information for solving a target task without any re-training on new data . The inherent gap between existing ML approaches and human intelligence raises a research question : Can we design a ML model that is trained on one set of features and able to generalize to combine new unseen features for the same task without further training ? To find desirable solutions to this non-trivial question is challenging . Let us resort to how humans think and act in physical world scenarios . Imagine that we are well trained on predicting a person ’ s income with one ’ s age and occupation from historical records . Now we are asked to estimate the income of a new person with his/her age , occ . and education ( a new feature ) . We can modularize the process in brain systems that distill and leverage the knowledge in new observations into four steps , as shown in Fig . 1 ( b ) . 1 ) Perception : the observations are recognized by our senses ; 2 ) Abstraction : the perceived information is aligned with concepts in our cognition ; 3 ) Reasoning : we search similar concepts and observations in our memory to understand and absorb the new knowledge ; 4 ) Decision : with new understanding and abstraction , we make final decisions for the prediction . The above human ’ s thinking process inspires the methodology in our paper where we propose a new learning paradigm for open-world feature extrapolation . Our proposed framework contains two modules : a backbone network , which can be a feedforward neural network , and a graph neural network . 1 ) The backbone network first maps input features to embeddings , which can be seen as perception from observation . 2 ) We then treat observed data matrix ( each row represents a feature vector for one instance ) as a feature-data bipartite graph , which explicitly define proximity and locality structures among features and instances . Then a graph neural network is harnessed for neural message passing over adjacent features and instances in a latent space ( abstraction ) . 3 ) The GNN will inductively compute embeddings for new features based on those of existing ones , mimicking the reasoning process from familiar concepts to new ones in our brain . 4 ) The newly obtained embeddings that capture both semantics and feature-level relations will be used to obtain hidden representations of new data with unseen features and make final decisions . To endow the model with ability for extrapolation to new features , we propose two training algorithms , one by self-supervised learning and one by inductive learning . The proposed model and its learning algorithm can easily scale to large-scale dataset ( with millions of features and instances ) via minibatch training . We also provide a theoretical analysis on the generalization error on test data with new features , which allows us to dissect the impact of training features and algorithms on generalization performance . To verify our approach , we conduct extensive experiments on several real-world datasets , including six classification datasets from biology , engineering and social domains with diverse features , as well as two large-scale datasets for advertisement click prediction . Our model is trained on training data using partial features and tested on test data with a mixture of seen and unseen features . The results demonstrate that 1 ) our approach consistently outperform models not using new features for inference ; 2 ) our approach achieves averagely 29.8 % higher Accuracy than baseline methods using KNN , pooling or local aggregation for feature extrapolation ; 3 ) our approach even exceeds models using incremental training on new features , yielding average 4.1 % higher Accuracy . Our contributions are : 1 ) We formulate open-world feature extrapolation problem and show that it is feasible to extend neural models for extrapolating to new features without re-training ; 2 ) We propose a new graph-learning model and two training algorithms for feature extrapolation problem ; 3 ) Our theoretical analysis shows that the generalization error for data with new features relies on the number of training features and the randomness in training algorithms ; 4 ) We conduct comprehensive experiments and show the effectiveness , applicality and scalability of proposed method . 2 Methodology . We focus on attribute features as input in this paper . An input instance is a vector ri = [ rim ] dm=1 2 Rd where each entry rim denotes a raw feature ( like age , occ. , edu. , etc. ) . If rim is a discrete/categorical raw feature with Rm possible values , its space is an integer set { 0 , 1 , · · · , Rm 1 } . If rim0 is a continuous one , a common practice is to convert it into a discrete feature within space { 0 , 1 , · · · , Rm0 1 } by evenly division [ 24 ] or log transformation [ 23 ] . We call Rm as cardinality for m-th raw feature . An effective way to handle attribute features is via one-hot encoding [ 13 , 24 , 17 ] . For rim with cardinality Rm we convert it into a Rm-dimensional one-hot vector xmi where the unique 1 indexes the value . In this way , one can convert an input ri into a concatenation of one-hot vectors : xi = ⇥ x 1 i , x2 i , · · · , xd i ⇤ , where 1 8m d , xm i 2 { 0 , 1 } Rm is a one-hot vector . ( 1 ) Use xij to denote the j-th entry of xi and we call each xij as feature in this paper . Assume D = P d m=1 Rm denotes the number of features and , as a reminder , d is the number of raw features . We next give a formal definition for open-world feature extrapolation problem in this paper : given training data { ( xi , yi ) } i2Itr where xi 2 Xtr = { 0 , 1 } D , yi 2 Y and Itr is a set of indices , we aim to learn a model that can generalize to test data { xi0 , yi0 } i02Ite where xi0 2 Xte = { 0 , 1 } D , yi0 2 Y and Ite is another set of indices . We term Xtr as training feature space and Xte as test feature space . We assume 1 ) the label space Y is shared by training and test data , and 2 ) Xtr ⇢ Xte , i.e. , test feature space is an extension of training feature space . The feature space expansion stems from two possible causes : 1 ) there appear new raw features incrementally collected from other fields ( i.e. , d increases ) or 2 ) there appear new values out of the known space of existing raw features ( i.e. , Rm increases ) . 2.1 Proposed Model . Our model contains three parts : 1 ) feature representation that builds a bipartite feature-data graph from input data ; 2 ) a backbone network which is essentially a neural network model that predicts the labels when fed with input data ; 3 ) a GNN model that inductively compute features ’ embeddings based on their proximity and local structures to achieve feature extrapolation . Feature Representation with Graphs . We stack the feature vectors of all the training data as a matrix Xtr = [ xi ] i2Itr 2 { 0 , 1 } N⇥D where N = |Itr| . Then we treat each feature and instance as nodes and construct a bipartite graph between them . Formally , we define a node set Ftr [ Itr where Ftr = { fj } Dj=1 with fj the j-th feature and Itr = { oi } Ni=1 with oi the i-th instance in training set . The binary matrix Xtr constitutes an adjacency matrix where the non-zero entries indicate edges connecting two nodes in Ftr and Itr , respectively . The induced feature-data bipartite graph will play an important role in our extrapolation approach . The representation is flexible for variable-size feature set , enabling our model to handle test data xi0 2 { 0 , 1 } D which gives Xte = [ xi0 ] i02Ite . Backbone Networks . We next consider a prediction model h✓ ( · ) as a backbone network that maps data features xi to predicted label ŷi . Without loss of generality , a default choice for h✓ is a feedforward neural network . The first layer serves as an embedding layer which shrinks xi into a Hdimensional hidden vector zi = xiW where W 2 RD⇥H denotes a weight matrix . The subsequent network ( called classifier ) is often a stack of neural layers that predicts label ŷi = FFN ( zi ; ) . We use the notation ŷi = h ( xi ; , W ) to highlight two sets of parameters and ✓ = [ , W ] . Notice that the matrix multiplication in the embedding layer is equivalent to a two-step procedure : 1 ) a lookup of feature embeddings and 2 ) a permutation-invariant aggregation . More specifically , we consider W as a stack of weight vectors W = [ wj ] Dj=1 where wj 2 R1⇥H corresponds to the embedding of feature fj . The non-zero entries in xi will index the corresponding rows of W and induce a set of embeddings { zm i } d m=1 where zmi is the embedding given by xmi ( i.e. , one-hot vector of m-th raw feature ) . Then the hidden vector of i-th instance can be obtained by aggregation , i.e. , zi = P d m=1 z m i which is permutation-invariant w.r.t . the order of feature embeddings in { zm i } d m=1 . A more intuitive illustration is presented in Fig . 2 . The permutation-invariant property opens a way for handling variable-length feature vectors xi0 [ 56 ] . Essentially , on condition that we have embeddings for input features , we can add them up to get a fixed-dimensional hidden representation zi0 as input for the subsequent classifier . Therefore , the problem boils down to learning feature embeddings , especially how to extrapolate for embeddings of new features based on those of existing ones . Remark . Instead of using sum aggregation , some existing architectures consider concatenation of d embedding vectors zm i ’ s , which is essentially equivalent to sum aggregation ( see Appendix A for more details ) . Therefore , the permutation-invariant property holds for widely adopted deep models . GNN for Feature Extrapolation . We proceed to propose a graph neural networks ( GNN ) model for embedding learning with the feature-data graph . Our key insight is that the bipartite graph explicitly embodies features ’ co-occurrence in observed instances , which reflects the proximity among features . Once we conduct message passing for feature embeddings over the graph structures , the embeddings of similar features can be leveraged to compute and update each feature ’ s embedding . The model can learn to extrapolate for new features ’ embeddings using those of existing features with locality structures in a data-driven manner . The message passing over the defined graph representation is inductive w.r.t . variable-sized feature nodes and instance nodes , which enables the model to tackle new feature space with distinct feature sizes and supports . Specifically , we consider the embeddings wj as an initial state w ( 0 ) j for node fj in Ftr . The initial states of instance nodes are set as zero vectors with equal dimension as the feature nodes , i.e . s ( 0 ) i = 0 . The interaction between two sets of nodes { wj } Dj=1 and { si } Ni=1 can be modeled via graph neural networks where the node states in the l-th layer are updated by s ( l ) i = P ( l ) COMB ⇣ s ( l 1 ) i , AGG ( { w ( l 1 ) k } |8k , xik = 1 ) ⌘ , w ( l ) j = P ( l ) COMB ⇣ w ( l 1 ) j , AGG ( { s ( l 1 ) k } |8k , xjk = 1 ) ⌘ , ( 2 ) where P ( l ) 2 RH⇥H is a weight matrix and we do not use non-linearity since it would degrade the performance empirically . For any new feature fj0 in test data , we can set its initial state as a zero vector w ( 0 ) j0 = 0 . The GNN model outputs updated embeddings for feature nodes and we further use them as the feature embeddings in the backbone network . Fig . 3 presents the feedforward computation of proposed model . Formally , with L-layer GNN , the GNN network g gives updated feature embeddings Ŵ = [ w ( L ) j ] D j=1 = g ( W , X ; ! ) and then the backbone network outputs prediction ŷi = h ( xi ; , Ŵ ) where X = Xtr for training , X = Xte for test and ! = { P ( l ) } Ll=1 denotes g ’ s parameters , .

Summary: This paper formulates the feature extrapolation problem and proposes a novel framework to tackle it. By definition in the paper, the feature extrapolation problem emerges when, during test time, feature space extends beyond training set either due to the collection of new raw features or feature values going beyond its range. To handle such problems without requiring retraining a model in the occurrence of new features, the authors proposes to jointly train the backbone network, which performs the main classification task, and a GNN, which serves as a middleman to map the raw features to the input of the backbone network. This GNN is trained to find the co-occurrence relationships among feature values and extrapolate the representation for new features. Theoretical analysis was conducted to study the generalization error of the learning algorithm. Empirical study was carried out to demonstrate the effectiveness of the framework in solving tasks from real-world application.