Aux-Drop: Handling Haphazard Inputs in Online Learning Using Auxiliary Dropouts
Authors: Rohit Agarwal
Deepak Gupta
Alexander Horsch
Dilip K. Prasad
Venue: Accepted by TMLR ()

Abstract: Many real-world applications based on online learning produce streaming data that is haphazard in nature, i.e., contains missing features, features becoming obsolete in time, the appearance of new features at later points in time and a lack of clarity on the total number of input features. These challenges make it hard to build a learnable system for such applications, and almost no work exists in deep learning that addresses this issue. In this paper, we present Aux-Drop, an auxiliary dropout regularization strategy for online learning that handles the haphazard input features in an effective manner. Aux-Drop adapts the conventional dropout regularization scheme for the haphazard input feature space ensuring that the final output is minimally impacted by the chaotic appearance of such features. It helps to prevent the co-adaptation of especially the auxiliary and base features, as well as reduces the strong dependence of the output on any of the auxiliary inputs of the model. This helps in better learning for scenarios where certain features disappear in time or when new features are to be modeled. The efficacy of Aux-Drop has been demonstrated through extensive numerical experiments on SOTA benchmarking datasets that include Italy Power Demand, HIGGS, SUSY and multiple UCI datasets. The code is available at https://github.com/Rohit102497/Aux-Drop.

Published in Transactions on Machine Learning Research (05/2023)
Aux-Drop: Handling Haphazard Inputs in Online Learning Using Auxiliary Dropouts
Rohit Agarwal
agorwal. 102497 @ig497 com
Bi Lab, Deportment of Computer Science UT The Aretic University of Norway, Tromss
Dcepak Gupta
Bio-AI Lab, Department of Computer Science UiT The Arctic University of Norway, Tromsø
Alexander Horsch
Bio-Al Lab, Department of Computer Science UT The Arctic University of Norway, Tromss
Dilip K. Prasad
Bio-Al Al Lab, Deportment of Computer Science UT The Arclic Universily of Norway, Tromss
Reviewed on OpenReview: https: // openreview. net/ form?? id= B62
Abstract
Many real-world applications based on online learning produce streaming data that is hap- hazard nature, í.e., contains missing features, features becoming obsolete time, the appearance of new features later points in time and lack of clarity on the total num- ber of imput features. These challenges make it hard build learnable system for such applications, and almost no work exists in deep learning that addresses this issue. In this paper, we present Aux-Drop, an auxiliary dropout regularization strategy for online learn- ing that handles the haphazard input features in an effective manner. Aux-Drop adapts the conventional dropout regularization scheme for the haphazard input feature space en- suring that the final output is minimally impacted by the chaotic appearance of such fea- tures. It helps to prevent the co-adaptation of especially the auxiliary and base features, as well reduces the strong dependence of the oulput on any of the auxiliary inputs of the model. This helps in better learning for scenarios where certain features disappear in time or when new features are to be modelled. The efficacy of Aux-Drop has been demonstrated through exlensive numerical experiments on SOTA benchmarking datasets that include Italy Power Demand, HIGGS, SUSY and multiple UCI datasets. The code is available at http.//o././oittt000///////--/ /
1
Introduction
Many real-life applications produce streaming data that is difficult to model. Moreover, a lot of existing methods assume that the streaming data has time-invariant fixed size and the models trained accord- ingly (Gama 2012 Nguyen et al. 2015). However, this is not always truc and the dimension of inputs can vary over time. The inputs can have missing data, missing features, obsolete features, sudden features and an unknown number of the total features. We define this here as the haphazard inputs. Formally, we define haphazard inputs as streaming data whose dimension varies at every time instance and there is no prior information about data received in the future. The characteristics of haphazard inputs are as follows: (1) Streaming dala Here data arrives sequentially and is modelled using online learning techniques. The model

---

Published in Transactions on Machine Learning Research (05/2023)
Table Comparison of different online deep learning models with respect the characteristics of the haphazard imputs (C1-C6). We showcase the inability of online deep learning methods in handling haphazard inputs even when other techniques like imputation, extrapolation, priori information and Gaussian noise are employed.
Characteristics
Aux-Drop
Online Deep
ODL
ODL Extrapolation
ODL + Prior Information
ODL
Learning
+ Online
Methods
Gaussian
like
Data Imputation
Noise
ODL
Streaming data (C1)
Missing data (C2)
X
X
Missing features (C3)
X
X
Obsolete features (C4)
X
X
V
X
Sudden features (C5)
X
Unknown no. of features (C6)
X
X
predicts an output based on the current data instance and then the actual output is revealed. The model gets trained based on the loss from its prediction and the actual output, and this updated model is used for future prediction (Hoi et al. 2021). (2) Missing data The input features can be missing at any time instance. It can be due to data corruption, malfunctioning sensors, faulty equipment, human errors, etc. [Emmaauel et al. (022)) (3) Missing features It is known that a certain feature will arrive but doesn't 't have any other prior information like its distribution. It is never received at time instance t = 1. (4) Obsolete features - The input features are received at any point time but ceases to cxist after some time instances. (5) Sudden features There is information about the existence of these features when the model is defined. The model might know about this feature suddenly at any point of time after the model deployment. (6) Unknown number of features: At the time of model designing, there is no information about the total number of imput features and at no point in time this information is available.
Some approaches (Beyazit et al. 2019 He et al. 2019) are available to address the challenges of haphazard inputs but there is a dearth deep-learning approaches in this field. Moreover, the characterization of the problem discussed here is not defined properly in the previous works, hence we make an attempt introduce the problem (haphazard inputs) in more formal way in this paper.
The outlined issues of haphazard inputs can be partly tackled by coupling online deep-learning framework with some existing approaches such as feature imputation, extrapolation of information and regularization with Gaussian noise, among others. But not all challenges can be handled by any single framework. This is better explained in Table I where we list the prominent challenges of online learning with streaming data as well as point out the limilations of the existing approaches. Since the data is straminng, it becomes imperative to apply online learning methods like online deep learning (ODL) (Sahoo et al., 2017) and online gradient descent (OGD) (Sahoo et al.) 2017 streaming aspect of the haphazard input. The online imputation model can be used to impute missing data (C2) and can thus be applied in conjunction with any other online learning method but still, it can't address the other characteristics. Extrapolation can be used to address the case of obsolete features (C4). Prior information on the features can be used to project the missing data (C2). Lastly, there can be a naive way of employing the Gaussian noise wherever the data not available. This can address missing data (C2), missing features (C3) and obsolete features (C4), however, it can still not cater to the appearance of new features (C5) as well handle the issue of missing information on the total number of features (C6).
et al. 1996), however, it only handle the
Recently, Agarwal et al. (020 presented Aux-Net, deep learning architecture capable of handling the issucs outlined above. However, Aux-Net employs dedicated layer for each auxiliary feature, which results in a very heavy overall network, and this leads to a significant increase in training time for each additional auxiliary feature being modelled. The high time and space complexity of Aux-Net makes it inefficient and not scalable for larger problems/datasee
Here, auxiliary features refer to those features which are not available consistently in time, rather these are subjected atleast one of the characteristics (C2-C6) outlined in Table I It. is assumed that atleast one feature is always available and is known the base
2

---

Published in Transactions on Machine Learning Research (05/2023)
feature. Let us denote the set of indices of base features received time t by B, = B, then the set of indices of base features received at time ++1 1 is B = IB. Similarly, let us denote the set of indices of auxiliary features received at time & by Ae, then the set of indices of auxiliary features received at time t ++1 1 is given by A++1 S [ U Ae41 where As+1 is the set of indices of new auxiliary features received such that A+t1 0 [,]= = Φ. Here, we follow the assumption that B + o, i.e., there is atleast one base feature.
In this paper, we present Aux-Drop, an auxiliary dropout regularization strategy for an online learning regime that handles the haphazard input features in an accurate as well as efficient manner. Aux-Drop adapts the conventional dropout regularization scheme (Hinton et al. 2012) for the haphazard input feature space ensuring that the final output minimally impacted by the chaotic appearance of such features. It helps to prevent. the co-adaplation of especially the auxiliary and base features, as well as reduces the strong dependence of the output on any of the auxiliary inputs of the model. This helps in better learning for scenarios where certain features disappear in time or when new features are to be modelled. Aux-Drop is simple and lightweight, as well as scalable to even a very large mumber of auxiliary features. We show the working of our model on the Italy Power Demand dataset (Dau et al. 2019), the widely used benchmarking datasets for online learning such as HIGGS (Baldi et al. 2014) and SUSY (Baldi et al. 2014) and 4 different UCI datasets Dua & Graff 2017).
The contributions of this paper can be listed as follows: (1) We formally introduce the problem of haphazard inputs and their characteristics. (2) We propose a dropout-inspired concept called Aux-Drop to handle the haphazard streaming inputs during online learning. It employs selective dropout to drop auxiliary nodes accommodating the haphazard auxiliary features and random dropout to drop other nodes. Together they handle the auxiliary fcatures while preventing co-adaptations of auxiliary and base features. (3) The simplicity of Aux-Drop allows us to couple it with existing deep neural net works with minimal modifications and we demonstrate it through ODL and OGD,
2 Related Work
Online Learning Online learning is approached via multiple concepts in the machine learning domain 2012: Nguyen et al. 2015). Among the various approaches that exist, some popular methods are k-nearest neighbours (Aggarwal et al.) 2006), decision trees (Domingos & Hulten 2000 support vector machines (Tanng el. al. 2007). fuzzy logic (Das et al. 2016: Tyer el al. 2018), bayesian theory (Seidl et al.,
Gama
2009) and neural networks (Liite et al. 2013). Recently, deep learning approaches with different learning mechanisms (Hoi et al.) 2021) are introduced resulting in architectures like online deep learning (ODL) (Sahoo et al. 2017). The ODL has shown tremendous improvement in the learning capability for streaming classification tasks. But all these methods are limited by the assumption of fixed input features.
Haphazard Inputs Different versions and subsets haphazard inputs are present in the literature. Incre- mental learning approaches like ensemble methods (Polikar 2012). Learn++ algorithms (Polikar et al. Mohammed et al. 2006),, Learn++MMF (Polikar et al.) 2010) and + (Elwell & Polikar 2011 handles only missing feature problems and is too expensive in terms of training and storage requirements.
2001
Zhou 2022 presents opern-rroirrommen machine learning which includes emerging new classes (Parmar et al.
2021j, mncremental/deecrmentta features (Hou et al. 2021). changing data distribution (Sehwag et al. 2019 and varied learning objectives (Ding & Zhou 2018). Online learning with streaming features (OLSF) algo-
rithm (Zhang et al. 2016) handles the trapezoidal data streams where both data volume and feature space increase over time. Hou et al.] (2017) introduced the problem of fenture evolvable streams (FEST. where
the set of features changes after a regular time period. Zhang et al. (2020) proposed evolving discrepancy minimization (EDM) for data with evolving feature space and data distribution. Hou et al. (2021) tries to solve an interesting problem where there is an overlap between old and new features by introducing an incomplete overlapping period. Evolving metric learning (EML) (Dong et al. 2021) handles the incremental and decremental features. All the above methods solve specific problems in online learning and are only a subpart of the haphazard inputs. Online learning from varying features (OLVF) (Beyazit et al. 2019 and online learning with capricious data streams (OCDS) (He et al., (2019) can model haphazard inputs but both of them are con-deep-learning approaches and are only tested in small datasets. OCDS trains a learner based a universal feature space that includes the features appearing at each iteration. It reconstructs the
3

---

Published in Transactions on Machine Learning Research (05/2023)
Austmputlayer
AusLayer
(Best viewed color) The Aux-Drop ar- chitecture. The purple-colored box represents any on- line learning-based deep learning approach. The trape- zoid denotes zero more fully connected layers. Both the brown boxes are the same (represented by double- headed arrows) and are known AuxLayer. The Aux- InputLayer is the concatenation of the hidden features from the layer previous to the AuxLayer and the aux- iliary features. The AuxInputLayer and the AuxLayer are fully connected but depending on the unavailabil- ity of auxiliary features, the corresponding nodes are dropped (termed Auxiliary Nodes). The unlocked lock denotes an inherent one-to-one connection between the auxiliary features and the auxiliary nodes.
Figure 1:
Auailary
Audilary Fratuees
Outpu
Non Auxittry
Features
houu
unobsorved instances from observable instances by capturing the relatedness using a graph. Thus, OCDS is based on the dependency between features. Online learning from varying features (OLVF) (Beyazit et al. 2019) tries to handle the varying features by projecting the instance and classifier at any time into a shared feature subspace. It learns to classify the feature spaces and the instances from feature spaces simultaneously. The transformation in different feature spaces leads to loss of information resulting in poorer performance.
Dropout Dropout was proposed by Hinton et al. 2012 to prevent co-adaptations between features. Since then dropout has been used in different settings to handle various problems. Spatial dropout (Tompson let al., 2015) removes adjacent pixels in convolutional feature maps to reduce strong spatial correlation in the feature map activations. Poernomo & Kang (2018) proposed biased dropout and crossmap dropout for better generalization of the model. Biased dropout divides the nodes in layer into two groups, the one with higher activation has a lower dropout rate and the one with lower activation has a higher dropout rate. Crossmap drops or retains the nodes simultaneously in equivalent positions of each feature map to break the correlation between adjacent maps. Stochastic Activation Pruning (SAP) (Dhilonn et al. 2018 preserves the nodes with higher activation magnitude and normalizes the retained node to preserve the dynamic range of activations in each layer. Guided dropout (Keshari et al. 2019) drops the nodes with higher strength in order to improve the performance of low-strength nodes. Group-wise dropout. (Ke et al.) 2020) adjusts the dropout probabilities adaptively using the feature density by analyzing the number of linearly uncorrelated deep features gathered over equally spaced grids in low-dimension feature space. All the discussed dropout strategy is employed to better generalize the model and prevent overfilling but none of these st strategies can be used to handle haphazard inputs. Aux-Drop drops the nodes in the hidden layer based on the unavailability of auxiliary features and dropout rate.
3 Method
3.1
Aux-Drop
The core of Aux-Drop lies in utilizing the concept of dropout to accommodate the ever-changing charac- teristics of haphazard inputs. Dropout drops the nodes randomly from a hidden layer whereas we employ selective dropout along with the random dropout. The proposed Aux-Drop concept handles the base features and auxiliary features synchronously. The input features which are always available are termed base features and the input features which are haphazard inputs are known auxiliary features. The Aux-Drop concept can be applied in any deep learning- baseed based model capable of handling streaming data. A conventional online deep learning model has one input layer which is connected to the first hidden layer and all the input features are passed via this input layer. But in the Aux-Drop selup, we created a division in the passing of input features to the model such that it can utilize all the information from the base features and increment the model learning from the haphazardly available auxiliary features. The base features are directly passod the deep learning model (the purple color box in Figure A hidden layer of this model is designated as an AuxLayen and is represented by the dashed brown rectangular box in Figure 1 The hidden features from
4

---

Published in Transactions on Machine Learning Research (05/2023)
the layer previous to the AuxLayer are concatenated with the incoming auxiliary features and are known as the AuxTnputLayer. The input to the AuxLayer is the AuxTnputlayer and is fully connected. Based on the number of different auxiliary features received, a pool of auxiliary nodes is chosen from the AuxLayer such that there is a correspondence between an auxiliary feature and a specific auxiliary node. Whenever an auxiliary feature is not available, the corresponding auxiliary node is dropped from the AuxLayer. This creates an inherent one-to-one connection between the auxiliary features and the auxiliary nodes. The rest of the nodes in the AuxLayer are termed the Non-Auxiliary nodes. The diagram of the Aux-Drop concept is presented in Figure L
Dropout is applied only in the AuxLayer. The nodes to be dropped include all those nodes from the pool of auxiliary nodes whose corresponding auxiliary features are unavailable, forming the group of selective dropout nodes. The rest of the dropout nodes are chosen randomly from the remaining nodes of AuxLayer. Dropout (Hinton et al.j 2012) was proposed to prevent complex co-adaptations on the training data. We exploit this property of dropout in handling haphazard inputs. Instead of randomly dropping the nodes, we define some auxiliary nodes that certainly need to be dropped. Aux-Drop makes auxiliary features contribute to the deep learning model even when some of the other features are not available making it independent. This prevented complex co-adaptations in which an auxiliary feature is only helpful in the context of several other auxiliary features.
3.2 Mathematical Formulation
The problem statement is defined as finding a mapping f x +Y, where X, Y is streaming data, such that X, = {(X,, Yi), (Xr, Yr)}. The capital letter variables in italics denote a vector here. The input X consists of base features (X and auxiliary features (X+) and can be represented We define ng as the number of base features and n'a as the number of auxiliary features
Problem Statement
as X received at time L. For convenience, we define TA as the maximum number of auxiliary features. Note that, we do not need the information about n' at any point in time in the model. The input feature at time instance is given by X, = features at time t, respectively. Let us denote an imput feature by 2, then the base features at any time t is given by X²_ = [a where B is the set of indices of base features such that B = {1, (1,... b, n.., and b is the index of bth base feature. Similarly, the auxiliary features at any time I is given by X, - (a beja where A, is the set of indices of auxiliary features at time I such that A, C A (1,.......... ". "m and a is the index of th auxiliary feature. The output Y E IRS, where is the total number of classes. Since the problem is based on online learning, at any time t, we have access to only the input fcatures X, and once the model is trained on X, and a prediction is made, we get the output labels Y,
where X² and X are the base features and the auxiliary
AuxLayer AuxTayer handles the haphazard auxiliary inputs by employing the dropout. Any 7th layer of the model is mathematically given by L; (W;; S Ma], where W;, S; and M, denotes the weights connection between the nodes of (i - 1)th and 7th hidden layer, the bias of the 7th layer nodes, and the set of nodes in the 5th layer respectively. If the layer is the 1st layer or the AuxLayer then for W,, the (i (i-1) 1)th layer would be the base features or the AuxLayerInput, respectively (see Figure D Thus, if 2th hidden layer is chosen as the AuxLayer, then AuxLayer can be mathematically given by L2 = (Wa; M&2, where M2, consists of auxiliary nodes (M&) and non-auxiliary nodes (M2). For each auxiliary feature, there is an auxiliary node, i.c., |M = [A]. where |. represents the cardinality of set. The set of auxiliary nodes depends upon the number of new auxiliary features received at any time 1. Thus, whenever a new auxiliary feature arrives, we introduce a new node with a full connection with AuxInputLayer and an inherent one-to-one connection with the auxiliary feature (represented by the unlocked lock in Figure D in the AuxLayer and include it in the set of auxiliary nodes. For simplification from here on, we will consider n as the maximum number of auxiliary features, and thus [M4 = n/aaz. Thus, the number of nodes in the set of non-auxiliary nodes is given by [M&] = [M,| - naaa. The AuxlnputLayer is the input to the AuxLayer and at time and it is given by
1 = (XA, H.-1e
(1)
where H is the output of the (=- - 1=-1) 1)th layer at time t.
5

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm 1 Aux-Drop algorithm
Require: A deep learning-based online learning model OL, dropout d, the AuxLayer
Create Aux-Drop(OL.) from OL as done in Figure]
while time do
Receive ,
Pass X to Aux-Drop(OL) and get the hidden features H2-1,t
Get 1 by eq.
Get M.,, by eq.
Get W2,,, Sa,, by freezing weights, bias affected by unavailable auxiliary features, and dropped nodes Create I,, by eq.
Get the prediction Y, of the model Aux-Drop(OL) Receive the actual label Y; Compute the loss from Y, and Y Update the weights and biases of Aux-Drop(OL) based on the computed loss end while
Auxiliary Dropout Let the dropout value be d, then the number of nodes to be dropped is given by M,] d, and the set of dropout nodes represented by MF. We always choose the value of d sufficiently large such that the number of dropout nodes is always greater than the number of auxiliary nodes. The auxiliary dropout component consists of selective dropout on the auxiliary nodes based on the unavailable auxiliary features and random dropout on the leftover nodes from the AuxTayer. The selective dropout and random dropouts are represented by MD- and MD:, respectively. The selective dropout includes all those nodes from the auxiliary nodes whose correspouding auxiliary features are unavailable and is given by
M.. = M - M
(2)
where M represents the set auxiliary nodes whose corresponding auxiliary features are available and U V represents the relative complement, i.c., clements that belong to U and not to V. Consider pe the probability of dropout of each jth node in the AuxLayer. Then because of the selective dropout, Pj =1 Vj E M.... Random dropout is achieved using the Bernoulli distribution. Our problem statement is closer match with the Bernoulli distribution because it is a discrete probability distribution that models the probability of a binary outcome for a single trial which is the basis of selective dropout. Other graphical models like the Random Markov field will fit our problem statement only we take the case when the set of auxiliary features probability can be modelled as a joint distribution like in the computer vision as there is high dependence between adjacent pixcls. |M.|- |M nodes are dropped randomly from the leftover nodes of AuxLayer (MIg M.)). Thus the dropout probability (p&) of the leftover nodes and the random dropouts (M.+) are given by
M,] d M
Pk
IM, MP
Wk M. -M
(3)
M
1: & M. MD. A = 1, m Bernoulli(px)
Thus, the set of nodes dropped from the AuxLayer is given by
M. = -MM + ?
(4)
Algorithm Here, we explain the working of the Aux-Drop. We choose deep learning based model capable of handling streaming data and name it OL. A dropout value d is set and a hidden layer z is chosen as the AuxLayer. We modify OL as done in Figure and term Aux-Drop(OII At the time t, we receive the base features X.3 and the auxiliary features X.A. The base features are passed Aux-Drop(OI.). We compute the hidden features of all the hidden layers before the AuxLayer. Based on H.-1,t5 the AuxInputLayer (IA) is constructed using eq. [ Now, we have to create the AuxTayer (Iz,a) based on the auxiliary features (XA)

---

Published in Transactions on Machine Learning Research (05/2023)
received at time t. MP, nodes are determined using eq. and are dropped from the AuxLayer. All the weight connections and bias are frozen which are affected by the unavailable auxiliary features and dropped nodes. Thus, the weights and the bias to the AuxLayer at time are given by Wa and S2,, respectively. The AuxLayer is given by
,.,, = ( 2,65 S2,; M, M..))
The Auxen is passed to the AuxLayer and the successive hidden layers computation is done giving final prediction Y. Finally, the ground truth Y, is revealed and the loss is computed between Y, and Y. The weights and biases of the Orx-Drop(OI.) are then updated based on this loss. The algorithm of the Aux-Drop is presented in Algorithm
3.3 Handling Haphazard Inputs
The auxiliary features are haphazard imputs that exhibit all the six characteristics presented in Table I We present a situation with all the characteristies of haphazard inputs and the changes in the Aux-Drop architecture with respect to different characteristics. Consider there are two output features from the hidden layer previous to the AuxLayer. At the time t - 1, two auxiliary features are available. We present only the upper half of the model which handles the auxiliary features. The architecture presented in Figure 2 (a) represents the model connection after instance
The dropout value is set as 0.7. (1) An Auziliary Feature is Missing: Figure 2 (b) presents the change in the architecture when an auxiliary feature is missing. The inherent one-to-one corresponding node is dropped and all the connections that come with it. Also, all the connections from these auxiliary features to all the other nodes are also frozen. Since the dropout value is 0.7, two nodes need to be dropped. One node is randomly chosen from the remaining 3 nodes and is dropped. (2) Missing data arrives: The auxiliary feature missing in the previous case (time t) arrives at inst instance/ I ++ 1. Thus, all the auxiliary features arrive. Two nodes are randomly chosen from all the nodes in the auxiliary layer and dropped along with all its connections as shown in Figure 2 (c). (3) Obsolete features: At time ++ 2, auxiliary feature becomes obsolete. Note that the model at any point doesn't know that this feature is obsolete. Thus for this situation, the change in architecture is the same as the missing auxiliary feature. The change is shown in Figure 2 (d). (4) Sudden features: At time i + 3, sudden auxiliary feature with no prior information arrives. To handle this, a new auxiliary node and all the connections with it are introduced (shown in the black rectangular box in Figure 2 (c)). The connections from this auxiliary feature to all the nodes AuxLayer are also roduced. The feature that became obsolete at time = 2 will not arrive so the corresponding auxiliary node is dropped. Two more nodes are randomly selected to drop. The architectural change present in Figure (e). (5) Missing feature arrives: At time t 1+ 4, a missing feature arrives whose prior information is unknown. The only attribute known about this feature is that it will arrive. To handle this, either an auxiliary node can be created and assume that this feature is unavailable till it arrives or it can be considered as a sudden feature. In either case, the computation overhead is almost negligible and the model performance will not change. Hence, we can consider it as a sudden feature and the architectural change is the same as Figure (e).
3.4 Discussion
Independent of the Maximum Number of Auxiliary Features For simplification of the Aux-Drop explanation, we set a value naaa as the maximum number of auxiliary features available to the model. As shown in the previous subsection 3.3 sudden features can be easily handled by the model and hence the value of n mar is not needed.
Invariant to the Architecture Aux-Drop can be applied to any deep-learning architecture capable of handling streaming data. We apply Aux-Drop the OGD and ODL framework and represent it Aux- DrGD(OGD) and Aux-Drop(ODL) respectively. Here, we demonstrate the change in the architecture of OGD and ODL framework when Aux-Drop is applied to il (see Figure 3 Aux-Drop(OGr
OGD is a simple deep-learning neural network with multiple hidden layers trained using stochastic gradient descent (Ketkar) & Ketkar 2017). For simplification, we present a 4-layer OGD (top figure in 3()).. To apply Aux-Drop on OGD, let's say, we first choose the 3rd hidden layer as the AuxLayer and fix the dropout value as 0.5. The hidden activation from the 2nd hidden layer is concatenated with the incoming auxiliary features giving an
7

---

Published in Transactions on Machine Learning Research (05/2023)
AuxinputL.ayer
AuxLayer
AuxinputLayer
AuxLayer
Auxiliary Nodes
Auxiliary Nodes
Auxillary Features
Auxiliary Features
Non AaniAuiillaayy Nodes
Non Prn-Auxxillary Nodes
Input from hidden layer base model
Input from layer base model
(a) After Instance t
(b) Instance t An auxiliary feature is missing
AuxinputLayer
AuxLayer
AuxinputLayer
AuxLayer
Auxilaary Features
Auxiliany Nodes
Auxiliary Features
Auxilaary Nodes
Con.Auxiliary Nodes
Non-Auxllary Nodes
Input from a hidden layer of base model
Input from a hidden layer of base model
Instance
Miissing
(d) Instance 2: auxiliary
AuxinputLayer
AuxLayer
Random dropout
Selective dropout
Auxillary Features
Auxiliary Nodes
Unavailable suxiliary feature
Inherent one-to-one connection of selective dropout node
Remaining connections selective dropout node
Input from hidden layer of model
Non Auxilary Nodes
Remaining connections
auxiliary
Connections to random dropout nodes
Inherent one-to-one connection of random dropout node (f) Legend
(e) Instance Sudden feature arrives
(Diagram best viewed in color) We present the changes in the connection between AuxInputLayer and AuxLayer with respect to different characteristics of haphazard inputs. Different color meaning: Here the blue circle denotes random dropout node, and the red circle in the AuxLayer and AuxInputLayer denotes selective dropout node and unavailable auxiliary feature respectively. The colored arrow follows hierarchical way of removing connections. First, the inherent one-to-one connection of the selective dropout node is frozen and it is denoted by the dashed red arrow. Next, the other connections the selective dropout node are frozen and it is represented by the green arrows. Then all the remaining connections from the unavailable auxiliary features frozen and shown by the purple arrows. Next, all the remaining connections to the random dropout node are frozen and are denoted by the blue solid arrows. Finally, the inherent one-to-one connection to the random dropout node is shown by the blue dashed arrow. This is more clear from the legend present in Figure (f). In Figure 2 (e): The black rectangular box depicts the arrival of auxiliary feature and the introduction of auxiliary node with all the relevant connections in the AuxLayer. Moreover, all the connections from the new auxiliary feature to all the nodes in the AuxLayer are also introduced.
Figure 2:
AuxinputLayer. All the connections between the 2nd and 3rd hidden layer is removed. The 3rd hidden layer is made flexible by increasing the number of nodes and providing it the capability adding additional nodes whenever required based on the new auxiliary features. The nodes in the AuxLayer are grouped into two parts: auxiliary nodes and non-auxiliary nodes such that the number of nodes in auxiliary nodes is equal
8

---

Published in Transactions on Machine Learning Research (05/2023)
Imput Fantures
Imput Features
hedge
hedggn
Medge
hedge
Output
Wn
Auxinputlayer
Auxinputl.ayer
Audilary Features
Auxiliury Featurus
AuxLayer
AuxLayer
Features
Base Features
Output
hedge
hedge
hedge
(a) Top: OGD, Down: Aux-Drop(OGD)
(b) Top: ODL, Down: Aux-Drop(ODL)
Figure 3: (Diagram best viewed in color) Two online learning methods (OGD and ODL) and their corresponding Aux-Drop architecture (Aux-Drop(OGI) and OUX-Drop(ODL)) are shown here. (a) Top: OGT) architecture, Down: Aux-Drop(OGD) architecture (b) Top: ODL architecture, Down: Anx-Drop(ODL)) architecture.
to the known number auxiliary features. full connection introduced between the LuxlnputLayer and AuxLayer such that there is one-to-one connection between the auxiliary features and the auxiliary nodes. All the required connections (because of the new nodes added in the AuxLayer) between the AuxLayer and the 4hh hidden layer also introduced. This gives us the Aux-Drop(OGD) architecture (sce down figure in the AuxLayer and necessary changes done based on the auxiliary features received at any time instance t as discussed in subsection 3.3 Aux- Mluz-roo(ODDL A snippet of ODL with 4 hidden layers is shown in Figure (bb) (the top figure). The difference between ODL and OGD is that the output prediction in the OGD architecture is based on the last hidden layer whereas the final prediction in the ODL is based on the weighted output from each hidden layer. The weightage (a) of the prediction from each hidden layer is adapted using the Hedge algorithm (Freund Schapire 1997). Detailed information about the ODL architecture can be found in (Sahoo et al.) 2017). To create Aux- (ix-D)roOODL)) (see down figure in 3 bb), we follow similar process done for Aux- Mix--Dro(OGGD) The only difference is that the prediction from AuxLayer is removed since the number of nodes in AuxTayer is always different.
3 (a)). Selective and random dropout is applied
Haphazard Inputs and Aux-Drop Other Works It is to be noted that haphazard inputs are different than adversarial inputs Goooffelow et al., 2018) since haphazard inputs are not modified modified/changee
(more discussion in Appendix A Seniissupervieed techniques like MARK (Patil et al. 2022) atlempt to fill in the blanks by generating an unaviilable pattern for anomaly detection but it is orthogonal to haphazard inputs problem (see Appendix One can envision an analogy between progressive networks (Rusu et al. 2016) and Aux-Drop but progressive networks cannot deal with haphazard inputs because of storage and computational inefficiency (see Appendix C Domain adaptation (Farahani et al. 2021 assumes that all the

---

Published in Transactions on Machine Learning Research (05/2023)
Table 2: The number of instances and features all the datasets.
Datasct
# Instances
# Features
german
1000
Italy Power Demand
1096
24
svmguide3
1243
21
magic04
19020
10
a8a
32561
123
SUSY
1M
8
HIGGS
1M
21
features are base features (sce Appendix Dj. Open set problems (Scheirer et al. 2012) deal with an unknown number of classes whereas Aux-Drop deals with an unknown number input features (see Appendix E
Assumption of Basc Features Aux-Drop assumes that there atleast one basc feature, i.c., atleast one feature is always available. However, the Aux-Drop model can be adapted in multiple ways remove the assumption of base features follows: (1) The naive way to handle this would be to just assume one auritiory feature a base feature and impute it. This would induce some bias but still, it would be for just one feature. (2) Choosing the position of AuzLayer as the first layer, the Auz-Drop model can be adapted to handle haphazard inputs without any base feature. But we attempt to propose generalized concept of Aux-Drop where the position of AuxLayer is a hyperparameter and can be chosen based on the dataset and application. Hence, the assumption of the base feature is adding a better value proposition.
4 Experiments
Datasets We consider the Italy Power Demand datase (Dau et al. 2019). HIGGS BBlli el al. 2014
SUSY Baldi et al. 2014) and 4 different UCI datasets (germar)) svmguide CCang & Lin 2011
magic0 a8a"] Dua & Graff 2017). The number of instances and the features of cach datasot are listed in Table 2 (1) Italy Power Demand It contains the electrical power demand from Italy for twelve months. The associated task is binary classification task to determine the month of each instance, i.e. distinguish days from Oct to March (inclusive) from April to September. (2) HIGGS It is synthetic data produced using Monte Carlo simulations. It has 28 features out of which the first 21 features are kinematic properties measured by the particle detectors in the accelerator and the last 7 features are the functions of the first 21 features. We use only the first 21 fcatures because it is proved experimentally in the paper (Baldi et al., 2014) that the last 7 features don't contribute to the performance of deep learning model. The binary classification task is to distinguish between a signal process (1) where new theoretical Higgs bosons (HIGGS) are produced, and background process (0) with identical decay products but distinct kinematic features.
(3) SUSY It is also synthetic data produced using Monte Carlo simulations. The first 8 features are kinematic properties measured by the particle detectors in the accelerator. The last 10 features are functions of the first 8 features. Similar to HIGGS, we only use the first 8 features as it is proved experimentally by Baldi et al.] (2014) that the last 10 features do not contribute to training a deep learning model. The associated binary classification task to distinguish between a signal process (1) where new super-symmetric particles (SUSY) are produced and a background process (0) with the same detectable partieles. (4) german It contains the financial information of the customer to classify each customer having good or bad credit risks. The numerical form of data is considered which has 24 features. (5) somgwide3 It is a synthetic dataset of practical svm guide with binary classification task containing 1243 instances and 21 features. (6) magic04 It is simulated data for the registration of high
https://swv.ce. ucr wwtpre/eeddeeeoooooiiiieee. serieeseeeeeeeeeeeeeeeeeeeeeeeee data_2018/
https://archive uci http:///w.....aa///////////////a
http://////aaaaaaaaa/////////aaa html
https://rrchive.ic. http:///aaalll.........////a//aa
10

---

Published in Transactions on Machine Learning Research (05/2023)
Probability
Aux-Net
OUX-Drop(ODL)
Table The table contains the average loss on the Italy Power Demand dataset. The first 12 features are base fea- tures and the last 12 features are auxiliary features. The availability of each auxilarr feature is varied by uni- form distribution of probability p. The value of p ranges from .50 to .99. The average loss of Aux-Net for each is reported from the Aux-Net paper (Agarwal et al. 2020). OUX-Drop(ODL) is run 20 times and the mean + stan- dard deviation of the average loss is reported here.
.50
0.6975
0.6031=0.0081
.60
0.6831
0.5839=0.0111
70
0.6788
0.5497=0.000
.00
0.6130
0.5321=0.0071
.90
0.5456
0.5149=0.0119
.95
0.5168
0.5013=0.0108
.99
0.5165
1477880.00100
energy gamma particles in an atmospheric Cherenkov telescope. Monte Carlo is used for the simulation. The binary task is to distinguish a shower image is caused by primary gammas (signal-1) or cosmic rays in the upper almosphere (background-O)) (7) a8a It. is census data from 1994 to predict whether the income exceeds $50k/yr. The pre-processed data to get numerical values has 123 features. The training set of 32561 instances used for experiments in this paper.
Motivation of the Chosen Datasets The motivation to choose these datasets is two-fold: (1) The first is to see how Aux-Drop performs on small (german, Italy Power Demand, svmguide3), medium (magic04, a8a) and large (SUSY, HIGGS) datasets. (2) The number of features in the dataset ranges from 8 (SUSY) to 123 (aSa). This helps in demonstrating the flexibility of Aux-Drop in the varying number of auxiliary features.
Choice of Deep Learning Model We the online deep learning (ODI.) model for few reasons: (a) ODL has shown better performance in the online learning domain and can handle big datasets very efficiently, (b) The only deep learning method available for haphazard inputs (Aux-Net) also use ODL as their base model, hence it gives a fair comparison. We also apply Aux-Drop on the online gradient descent (OGD) framework. We represent them as Aux-Drop(ODL) and Aux- (ux-Drop(OGD)) respectively and collectively as Aux-Drop.
Comparison Models We evaluate the performance of Aux-Drop empirically in multiple scenarios. We compare AuXLDropOODL) with Aux-Net (Agarwal et al. 2020) since it is deep-learning method with ODL base, capable of handling haphazard inputs. We also report Aux- Dux-Drop(ODL) and Aux- Drop on 4 UCT datasets and compare them with OLVF (Beyazit. et al. 2019) and OLSF Zhang et al. (016)). Since most of the datasets used by previous methods were small, we also consider two big datasets to test the effective application and feasibility of Oux-Drop(ODI.) and compare it with ODL. In all the scenarios, the instances are provided one by one the model, and the training and testing are performed in single pass. For cach specific case, the dataset is designed suitably.
4.1 Comparison with Aux-Net
The current literature has only one deep learning model, Aux-Net (Agarwal al. 2020 based on ODL that can handle the situation we present. It is only applied to the Italy Power Demand dataset which is very small dataset. Nevertheless, we compare our model with Aux-Net and prepare the data similar to the Aux-Net paper. We considered the first 12 features of the Italy Power Demand dataset as the base features and the last. 12 features as the auxiliary features. We varied the availability of each auxiliary input independently by a uniform distribution of probability p. In comparison with Aux-Drop, Aux-Net is a very heavy model. Let the number of parameters from the base deep learning architccture be PB. Then the nunber of parameters for Aux-Drop and Aux-Net is given by PD (eq. G and PN (eq. 7 respectively.
PD = PB + NAman Mi,..
(6)
PN PB +NNN AAma NH + N NAmanN NH
(7)
where NH is the number of nodes in the hidden layer. Therefore, the number of parameters in Aux-Net is N NAmaa No Mi... more than the number of parameters in Aux-Drop since Aux-Net dedicates layer
11

---

Published in Transactions on Machine Learning Research (05/2023)
Table 4: Comparison with OLVF on various datasets. Here all the errois reported for OLVF are on p 0.75 (i.e.
Rem = 0.25) and the value is taken from its original paper. Thus, adjust the probability value (p) for haphazard features for Aux-Drop accordingly to match the amount missingness of OLVF. The error is reported as the mean + standard deviation of the 20 experiments performed with random seeds.
Dataset
OLVF
AuX-Drop(ODL)
Aux-Drop(OGG)
p
german
333.419.7
300.414.44
392.
0.73
svmguide3
346.4+11.6
297.2+2.0
297.5=1.5
0.72
magic04
6152.44454.7
5536.7+159.3
5382.8-98.99
0.68
a8a
8993.8++40..3
6710.7+117.8
7313.5+:777.7
0.75
for each auxiliary feature whereas Aux-Drop can handle that feature seamlessly using only one node. In numbers, we assume 200 auxiliary features, 200 hidden nodes and 800 nodes in the AuxTayer then Aux-Net has about 32M more parameters than Aux-Drop.
Aux-Drop settings Aux-Net is compared with Aux-Drop(ODL.) and its settings are kept similar to Aux- Net for fair comparison. Aux-Drop(ODLL) is trained with 11 hidden layers, considcring the 3rd hidden layer as AuxLayer. Each layer has 50 nodes and the AuxLayer has 100 nodes. The smoothing rate is set as 0.2, the discount rate is fixed at 0.99 and the dropout is chosen as 0.3. We use cross-entropy loss. The learning rate is 0.3. Since the number of instances is less, a higher learning rate helps the model converge faster.
Result We report the average loss by taking the mean of the total loss of each instance over the whole dataset. Aux-Drop(ODD) is run 20 times with random seeds for all the different values of p and the average loss + standard deviation is calculated. The result is shown in Table 3 Aux- Aux-Drop(ODi) Net. in all seven different probability scenarios. In the situation, where p 0.9, 0.95, 0.99, the difference in performance between Aux- Aux-Drop(ODL) and Aux-Net is very less. It is because of the less haphazardness in the data, the efficiency of the auxiliary dropout was not used to its fullest. Whereas, when the haphazard inputs are very high and frequent (p- = 0.8, 0.7, 0.6, 0.5), the difference between Aux-Net and Aux- Drop(ODL) is high. Here, dropout makes the features independent of each other and hence when the features are not available frequently, it doesn't affect the performance of the model. Whereas, Aux-Net has dedicated layers for each auxiliary feature, requiring time to converge whenever the data is infrequent. The change in performance in the case of p = 0.8, 0.7, 0.6 and 0.5 is 13.53%, 14.52%, 19.02% and 13.2% respectively. The amount of haphazardness is highest when p 0.5, implying the effectiveness of Aux-Drop(ODL).
outperforms Aux-
4.2 Comparison with state-of-the-ard OLVF
We consider datasets with enough instances (2 1000) and variability from the OLVF paper to apply our model. We chose 4 different UCT datasets, namely, german, svmguide3, magic04 and a8a to simulate the scenarios of haphazard inputs. We compare our model with OLVF for the scenarios of 0.25 removing ratio (Rem) where it denotes that 25% of the instances are removed. We consider the first 2 features as base features and the remaining features as auxiliary features in all the datasets. For fair comparison, we simulate the same amount of haphazardness as OLVF. For e.g., in the magic04 dataset with 10 features, Rem = 0.25 in OLVF experiments accounts for Rem = 0.32 for 8 auxiliary features in Aux-Drop. The probability p of the availability of each auxiliary input is 1 Rem, therefore, p 0.68 for the magic04 dataset. Similarly, we calculate the value of p for each dataset and round it to two decimal places. The p value in the Aux-Drop for cach datasct is shown in Table 4 We compare OLVF with both OODDD) and AOGD).
Aux-Drop Scttings Both Aux-Drop(ODL.) and Aux-Drop(OGD) have the same setting and hence are collectively referred to as Aux-Drop. Since the nunber of instances is less, we design Aux-Drop with only 6 hidden layers. The third layer is set as the AuxLayer. Each hidden layer has 50 nodes but the number of nodes in AuxLayer different for cach dataset considering the number of features and dropout value. The dropout value set as 0.3. The number of nodes in AuxLayer 100 except for aSa which has 400 nodes in AuxLayer. The number of auxiliary features in a8a is 121 and the dropout value is 0.3, we need about
12

---

Published in Transactions on Machine Learning Research (05/2023)
Table Experiments on the trapezoidal data streams. Aux-Drop(ODL) and Aux-Drop(OGI) are compared with OLSF and OLVF (metrics reported from their original paper) in terms of the average number of errors. All the experiments are performed 20 times and the mean standard deviation is reported.
Dataset
OLSF
OLVF
Aux-Drop(ODL) Aux-Drop(OGG)
german
385.5=210.2
329.249.8
312.218.0
320.9
svmguidc3
361.7+:99.7
351.6-25.9
296.9+1.0
297.0##0.9
magic04
6147.4465.3
5784.0+52.77
6361.25-33196
5635.8:
a8a
9420.41549.9
$649.8-526.7
7850.9=15.9
7848.8#10.3
(121/0.3 ~ 403) nodes. The smoothing rate for Aux-Drop(ODL) is 0.2 and the discount rate is 0.99. The cross-entropy loss is employed to train the model. The learning rate is 0.1 for the smaller datasets, i.e., german and svmguide33 whereas, for the larger datasets, i.., magic04 and a8a, it is set as 0.01.
Result We consider the metric reported by OIVE and calculate the number of errors for each dataset. Aux-Drop is run 20 times randomly with respect to data shuffling, creating haphazard imputs and initializing the model as done in the OLVF manuscript. The mean and the standard deviation of these experiments are reported in Table better than Aux-Drop(OGDD in all datasets except magic04. The worst-performing experiment out of the 20 experiments of Aux-Drop is better than the best-performing experiment of OLVF.
Aux-Drop outperforms OLVF for all the datasets. Aux-Drop(ODI, performs
4.3 Experiments on trapezoidal data streams
We experiment on the trapezoidal data streams and compare them with the OLVF and OLSF (best per- forming) algorithms Zhang al. 2016). The trapezoidal streams are simulated by splitting the data into 10 chunks. The number of features in each successive chunk increases with the data stream. The first chunk has the first 10% of the total features, the second chunk has the first 20% features, and so on. The Aux-Drop setting and the dataset used are similar to the section 4.2 except the number of nodes in AuxLayer for a8a is 600. Only the first two features are considered the base features. Note that first 11 features (10%) are always available in this case. All the models are run 20 times and the mcan + standard deviation is reported.
Result Table 5 shows the performance of Aux-Drop as compared to others. The Aux-Drop outperforms OLVE and OLSF in all the datasets except magic04. The mean error is low for Aux-Drop compared to OLVF and OLSF. The amount of error in OOD.)) and Aux-Drop(OGD) compared to OLSF and OLVF is 16.7% and 9.2% less in the a8a dataset, suggesting that as the amount of data increases, the performance of Aux-Drop massively increases as compared to OLSF and OLVF. Furthermore, the standard deviation of Aux-Drop is low from OLVF and OLSF in all the datasets (except magic04), showing that the Aux-Drop has performed well consistently all 20 experiments.
4.4 Evaluation on big datasets
HIGGS and SUSY dataset is used by ODL to report its metrics. Hence, we found them suitable to test the performance of (ux-DropOODL). We run our experiment for the first 1M instances. We design two exper- iments on this dataset: (1) Experiment on variable probability as done in section 4.1 and (2) Experiment on obsolete and sudden unknown features. Since the HIGGS and SUSY dataset is large, we run all the experiments 5 times (instead of 20) and the average is reported. For comparison, we chose ODL as the base model trained with only base features and refer it ODL(B). For HIGGS and SUSY, consider the first 5 and 2 features as base features and the next 16 and 6 features as auxiliary features, respectively.
AuxDrop Settings In both the cases of HIGGS and SUSY, we train the Oux-Drop(ODL) with 11 hidden layers. The 3rd layer is set as the AuxLayer. The number of neurons in each hidden layer is 50 and in the AuxLayer is 100. The dropout value is 0.3 and the lcarning rate value set at 0.05. The discount rate is fixed at 0.99 and the smoothing rate is 0.2. For a fair comparison, we design ODL with 11 hidden layers, 50 nodes in each hidden layer and the same value of learning rate, discount rate and smoothing rate.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Table 6: Error in HIGGS and SUSY for various probability p. The metric is reported as the mean I standard deviation of the number of errors in 5 runs. The error of ODL(B) trained on only base features for HIGGS and SUSY dataset is 441484,4222 184. and 286198.6 189.4 189.4, respectively. The AAvg value reported in the table is calculated by subtracting the average number of errors of ODL(BB with the average number of errors of Aux-Drop(ODDI), respectively.
10
HIGGS
SUSY
0.8
0.6
t/vg//
0.4
HIGGS
SUSY
0.2
p
Avg+Std
A Avg
Avg=std
A Avg
0.0
.01
140039.4+129.9
1449.8
285068.0--69.3
1110.6
0.01
0.20
0.50
0.50 0.99
Availability of auxiliarv data (D)
.05
440045.4+250.55
1437.8
283:63.011:05.5
2735.6
10
139752.0=118.5
1731.2
280752.44:396.2
5446.2
Figure 4: The average error and stan- dard deviation of ODL on the whole fea- tures (base + auxiliary features) for 5 runs for HIGGS and SUSY are 391334.8 and 218622.2. Thus the performance improvement (AAT) of ODL(B) because of the addition the whole 50148.4 and 67576.4 for HIGGS and SUSY, respectively. The fraction improvement in the Aux-Drop(ODL) is calcu- lated by the ratio of AAvg (in Table ( and
20
438775.24-361.7
2708.0
7-777770+-555.7
11291.6
.30
635256.0-665.5
6197.2
269269.8:15499
16928.8
44
332199.81-381.3
9292.4
262713.41632.3
23485.2
.50
426644.8-1666.1
13638.4
29479.2
.60
423002.8
18480.4
250108.0+:29.5
36090.6
auxiliary features
.70
418927.41495
22555.8
243954.21813.7
42244.4
.80
0.2500
28881.6
237211.64664.55
48987.0
.90
405834.61330.3
35648.6
230216.21698.3
55982.4
.95
399234.8=613.7
42248.4
226631 84354.
59566.8
AAI The value of p (in Table O denotes the amount of auxiliary information available.
.99
391787.8-641..8
49695.4
222151.6+18144
64047.0
4.4.1
Experiment variable probability
We vary the availability of each auxiliary feature by a uniform distribution of probability p. Each auxiliary feature is varied by the same value of p, but they are independent of each other. We consider all the situations such as when very little auxiliary data is present (p = 0.01), the haphazardness in the data is maximum (p = 0.5), almost all the auxiliary data is available all the time (p = 0.99), etc.
Results The mean and the standard deviation on HIGSS and SUSY are shown in Table t The mean and standard deviation of the number of errors of ODL(B) for HIGGS is 441483.2 and 184.3 respectively, Whereas, for the SUSY, it is 286198.6 and 189.4, respectively. We consider this value as the base value and also report the metrics for all the values with respect to this in Table 6 The A Avg reported in the table shows the less amount crrors made by Mux-Drop(ODI, utilizing the extra information from auxiliary Teatures and is calculated by subtracting the average number of errors of ODL(B) from the average number of errors of Aux-Drop(ODL)) respectively. Aux-Drop(ODD)) from auxiliary features when p- 0.01 and gives better performance. Moreover, at each increasing p value, the Aux- Aux-DropOODL) performance improves. This is better represented in Figure where the progression of the fraction improvement (AAvg/ AAT) is shown with respect to the availability of auxiliary data (p). The average error of ODL on all available datasets (i.e., all the auxiliary features are always available) is also reported here. For HIGGS, ODL trained on the 21 features gives an error of 391334.8 whereas, for SUSY, it is 218622.2. Based on this, we can say that the performance improvement (A AI) achieved by ODL(B) due to auxiliary data for HIGGS and SUSY is 48.4 and 67576.4, respectively.
is able incorporate even a little amount of data
4.4.2 Obsolete and Sudden Unknown features
We demonstrate the effectiveness of Aux-Drop(ODL) in processing the extra information reccived from auxiliary features in both the SUSY and HIGGS datasets. Here, we design the data in a such way that all of them are sudden features, i.e. there is no information about the existence of these features when the model defined. The model knows about this fcature suddenly time after the model deployment. For the SUSY dataset, the first auxiliary feature starts arriving from 100k till 500k, the next auxiliary feature ranges from 200k till 600k, and so to the 6th auxiliary feature coming from 600k to 1000k instances.
14

---

Published in Transactions on Machine Learning Research (05/2023)
14500
22750
QDL(B)
Aux Drop(ODL)
14000
22500
13500
22250
Number of Errors
13000
22000
12500
12000
21500
11500
ODL(B)
21250
Aux-Drop(ODL)
11000
100
200
300
400
500
600
700
800
900
1000
100
200
300
400
500
600
700
800
900
1000
8.F.1
F.
B.F.2
B.F.3
6.F.2
B.F.4
B.F.5
A.F.1
A.
A.
Features
A.F.2
F
A.F.3
A.F.7
F.
A.F.4
F 10
F 12
A.F.5
13
A.F.6
A. F. 15
16
900
1000
800
1000
Number of Instances (in k) (a) suSY
Number Instances (in
(b) HIGGS
Figure 5: Result of the obsolete and sudden features experiment (a) SUSY and (b) HIGGS dataset for Aux- (ODL. Every experiment run 5 times and average value reported. We calculate the average number of errors in each 50k instance. Thus, the graph starts at 50k and goes till 1000k. B. F. {n} and A. F. {n} represent the th number base features and auxiliary features, respectively. 95% confidence interval (CT) is also shown the graph. The green box represents the time instances when certain auxiliary feature available. The x-axis denotes the number of instances (in k).
Each feature becomes obsolete after arriving for 400k instances. Similarly for the HIGGS dataset, the first auxiliary feature arrives from 50k to 250k instances, the second arrives from 100k to 300k, and so on where every successive auxiliary feature arrives at 50k instances after the previous auxiliary features start arriving and arrive till the next. 200k instances. This is better depicted in the lower part. of Figure ((a) for SUSY and Figure b) for HIGGS.
Results Figure 5 shows the result of obsolete and sudden unknown features for both SUSY and HIGGS. The performance of ODL(B) and Aux-Drop(ODL) is similar for the first 50k instances since both get the same amount of data. But Aux-Drop(ODD) gets the auxiliary information, its performance improves. The maximum amount of auxiliary information received in the SUSY dataset is from 400k to 700k and the best performance is during that period. The minimum is achieved at 600k. Morcover, in the later stages after 900k, when the auxiliary information reduces, the Aux-Drop(ODI, converges to the performance of ODL(B) depicting the agile manner in which Maxx-Drop(OD))
that
handles the haphazard inputs. In the case of HIGGS, AuX-Drop(ODL.) better than ODL as soon as starts getting the auxiliary information. For each period of 50k instances, we get confidence interval (CI) low and CI high. We calculate the 95% CI. The average of all these 20 periods gives CI low of 13027.9 and CT high of 13077.2 for Aux-Drop(ODI.) and
15

---

Published in Transactions on Machine Learning Research (05/2023)
Table Table shows the need for AuxLayer and the use dropout. Notations: RDANDO Random Dropout in AuxLayer No Dropout in Others, RDAL Random Dropout in All Layers, ADARDO Auxiliary Dropout in AuxLayer and Random Dropout Other layers, RDIFL Random Dropout First Layer with features passed directly to the first layer. The probability 0.27, 0.28, 0.32 and 0.25 for german, svmguide3, magic04 and aSa datasets respectively.
Methods RDANDO 318.0=2.8
german
svmguide3
magic04
aSa
297.6=1.9
6123.5+169.1
7853.1216.3
RDAL
319.3=4.4
298.2+3..0
6433.1+143.7
7862:1323.2
ADARDO
318.6=4.2
297.6
6700.4133.1
7852.9115.9
OOx-Drop(ODL.)
317.4+1.9
296.9+1.5
6039.1+190.4
7855.5+16.8
RDIFL
318.5=2.8
297.2=1.7
6528=136.4
7869.9133.2
CI low of 14289.1 and C1 high of 14332.0 for ODL in the SUSY dataset. Whereas for the HIGGS dataset, the average CT (low, high) is (21696.8, 21799.4) for Aux-Drop(ODL.) and (22039.1, 22109.2) for ODL.
5 Ablation Studies
For all the ablation studies, we use AuX-Drop(ODL) framework and apply it to the four UCI dalasets (german, svmguide3, magic04 and a8a). We prepare all the haphazard datasets following the subsection [.2] In order to increase the complexity, Hence consider 1-p value from Table Thus, the probability of the availability of auxiliary features is 0.27, 0.28, 0.32 and 0.25 for german, svmguide3, magic04 and aga datasets respectively. The architecture of Aux-Drop(OOi) (ODL) is also followed from subsection 4.2 until it is specified otherwise. Aux-Drop(ODL) has 6 hidden layers with the 3rd layer as the AuxLayer. In this section, the terms Aux-Drop (ODL) and Aux-Drop are used interchangeably.
perform all the experiments with less availability of auxiliary features.
5.1 Need of AuxLayer
One of the requirements Aux-Drop is the presence of alteast one base feature. So, we design a model where we pass all the inputs (base and auxiliary fcatures) directly the first layer itself without the use of AuxLayer in ODL. Here, we employ Random Dropout in the First layer to handle the haphazard inputs (RDIFL). The performance of RDIFL is shown in the lower part of Table 7 Tt Drop(ODL) outperforms RDIFL by 7.4% in magic004 and is marginally better in other datasets. This is because Aux-Drop utilizes the full information from base layers and increments it with the information from haphazard auxiliary features.
be
that Aux-
5.2 Effective Use of Dropout
We apply the dropout in the AuxLayer with an emphasis on the coupling between the auxiliary feature and auxiliary node by the manner of selectively choosing nodes to drop, based on the unavailability of auxiliary features. But, it is to be noted that, dropout can be applied randomly in the AuxLayer too. Moreover, dropout can also be applied to the other hidden layers as well. We present comparison of all these ways of applying dropout and show empirically that Aux-Drop is the best way to employ dropout. We compare Aux-Drop with its three other variants: (a) RDANDO Random Dropout is applied in the AuxLayer and No Dropout is applied in Other layers, (b) RDAL- Random Dropout is applied in All Layers, and (c) ADARDO Auxiliary Dropout is applied in the AuxLayer and Random Dropout is applied in all the Other layers. The results of all these methods (applied on ODL) are compared with Aux-Drop (ODL) in the UCI machine learning dataset and are shown in the upper half of Table 7 Aux-Drop is better in all the cases except in a8a. The maximum variation in the results is seen the magic04 dataset. The sccond best method is RDANDO which also applies dropout only the AuxLayer. So, the best way is employ auxiliary dropout only in the Auxeayer.
16

---

Published in Transactions on Machine Learning Research (05/2023)
Table Comparison the position of AuxLayer. Pos here stands for the position. The experiment conducted the four UCI datasets. The probability p = 0.27, 0.28, 0.32 and 0.25 for german, svmguide3, magicO1 and aSa datasets respectively.
Pos
german
svinuidee3
magic04
a8a
1
317.7+2.00
298.5::2.9
6049.4-229.5
7842.1=31.6
2
319.1+3.5
298.4+2.8
6054.0=213.3
7730.5=73.5
3
317.4+1.9
296.9+1.5
6039.1+:190.4
7855.4=16.8
4
318.5+2.7
298.9+2.8
6110.7=146.2
7852.9=12.4
5
317.7+2.2
297.4+1.7
6428.7=105.5
7856.7=14.3
5.3 Effect of AuxLayer position in the Model
The position of AuxLayer is a hyperparameter in the Aux-Drop. In all the above methods, fixed the 3rd layer as the AuxLayer. Here, we demonstrate how the model performs with respect to the different positions of the AuxLayer in the 4 UCI datasets. The results are shown in Table g The 3rd layer seems to be the best position except for a8a which gives the best performance for the 2 position. The ratio of base and auxiliary features is 1:60.5 for a8a. Thus, it requires comparatively more layers to process the auxiliary information. The 2nd position outperforms the 1st position in ača because the first hidden layer helps process the base features. Whereas for the other three datasets, the maximum ratio of base features and auxiliary features is 1:11 (for german) and hence comparatively less number of layers enough to capture the auxiliary features.
6 Conclusion
The challenge and application of haphazard inputs are immense and to our knowledge, there are no effective deep learning methods available to handle it. So, we propose a generalized concept called Aux-Drop which can be applied to any deep learning-basee based online architecture. We demonstrate the effectiveness of Aux-Drop in multiple datasets and empirically assert the importance of the Aux-Drop design by applying it ODL and OGD frameworks. The various experiments on big datasets meticulously show the agile manner in which Aux-Drop processes the auxiliary information and converges to the base deep learning architecture during the unavailability of auxiliary features.
Acknowledgement We acknowledge the various funding that supported this project: Researcher Project for Scientific Renewal grant no. 325741 (Dilip K. Prasad) and UiT's thematic funding project VirtualStain with Cristin Project ID 2061348 (Alexander Horsch and Dilip K. Prasad).
References
Rohit Agarwal, Arif Ahmed Sekh, Krishna Agarwal, and Dilip K Prasad. Auxiliary network: Scal- able and agile online learning for dynamic system with inconsistently available inputs. arXiv preprint arXiv:2008. 1 1828, 2020.
Charu C Aggarwal, Jiawei Han, Jianyong Wang, and Philip S Yu. A framework for on-demand classification of evolving data streams. IEEE Transactions on Knowledge and Data Engineering, 18(5):577 589, 2006.
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep lcarning. Nature communications, 5(1):1-9, 2014.
Ege Beyazit, Jeevithan Alagurajah, and Xindong Wu. Online learning from data streams with varying feature spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3232 3239, 2019.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šindić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Machine Learning and Knowledge Discovery Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13, pp. 387-402. Springer, 2013.
17

---

Published in Transactions on Machine Learning Research (05/2023)
Nicolo Cesa-Bianchi Philip M Long, and Manfred K Warmuth. Worst-case quadratic loss bounds for prediction using linear functions and gradient descent. IEEE Transactions on Newral Networks, 7(3): 604 619, 1996.
Chin-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1-27, 2011.
Yining Chen, Haipeng Luo, Tengyu Ma, and Chicheng Zhang, Active online learning with hidden shifting domains. In International Conference on Artificial Intelligence and Statistics, pp. 2053-2061. PMLR, 2021.
Ron Tor Das, Kai Keng Ang, and Chai Quek. ierspop: A novel incremental rough set-based pseudo outer- product with ensemble learning. Applied Soft Computing, 46:170-186, 2016.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chi Michael Ych, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahslana, and Eamonn Keogh. The uer lime series archive. IEEE/CAA Journal of Automatica Sinica, 6(6):1293-133) 13058, 2019.
Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. Stochastic activation pruning for robust adversarial defense. arXiv preprint Xiv:18830 01880.144, 2018.
Yao-Xiang Ding and Zhi-Hua Zhou. Preference based adaptation for learning objectives. Aduances in Neural Information Processing Systems, 31, 2018.
Pedro Domingos and Geoff Hulten. Mining high-speed data streams. In Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 71-80, 2000.
Jiahua Dong, Yang Cong, Gan Sun, Tao Zhang, Xu Tang, and Xiaowei Xu. Evolving metric learning for incremental and decremental features. IEEE Trunsactions Circuits and Systems for Video Technology, 32(4))2290-23022 2021.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://arrcive..tt wcw.:ei..csccc.......ooo. udu.
m1
Ryan Elwell and Robi Polikar. Incremental learning of concept drift in nonstationary environments. 1EEE Transactions on Neural Networks, 22(10):1517-1311 2011.
Tlamelo Emmanuel, Thabiso Maupong, Dimane Mpoeleng, Thabo Semong, Banyatsang, Mphago, and Oteng Tabona. A survey missing data in machine learning. Journal of Big Data, 8(1):1-37, 2021.
Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain adaptation. Advances in Dala Science and Information Engineering: Proceedings from ICDATA 2020 and IKE 2020, pp. 877 894, 2021.
Yoav Freund and Robert E Schapire. A
generalization of on line learning and an application to boosting. Journal of computer and system sciences, 5591):119-139, 1997.
Joao Gama, A survey on learning from data streams: current and future trends. Progress in Artificial Intelligence, 1 (11):45-55, 2012.
Tan Goodfellow, Patrick McDaniel, and Nicolas Papernot. Making machine learning robust against adver- sarial inputs. Communications of the ACM, 61(7):56 66, 2018
lan Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139 144, 2020.
Yi He, Baijun Wu, Di Wu, Ege Beyazit, Sheng Chen, and Xindong Wu. Online learning from capricious data streams: a generative approach. In International Joint Conference on Artificial Intelligence Main track; 2019.
18

---

Published in Transactions on Machine Learning Research (05/2023)
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ily Sutskever, and Ruslan R Salakhutdinov. Im- proving neural nerwsrks by preventing co-adaptation of feature detectors. or Xiv preprinl ar 1iv: 2012.
Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: comprehensive survey. Neuro- computing, 459:249 299:249-289, 289, 2021.
Bo-Jian Hou, Lijun Zhang, and Zui-Hua Zhou. Learning with feature evolvable streams. Advances in Newral Information Processing Systems, 30, 2017.
Bo-Jian Hou, Lijun Zhang, and Zhi-Hua Zhou. Prediction with unpredictable feature evolution. Transactions on Neural Networks and Learning Systems, 2021.
1EEE
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with strong adversary. arXivv preprint arXiv:1511 3XXi::511.0000: 08034, 2015.
Aparna Ramosh Iyer, Dilip K Prasad, and Chai Hiok Quck. Pic-rspop: A brain-inspired psoudo-ineremental ensemble rough set pseudo-outer product fuzzy neural network. Expert Systems with Applications, 95: 172 189,189,00 2018.
Zhiwei Ke, Zhiwei Wen, Weicheng Xie, Yi Wang, and Linlin Shen. Group-wise dynamic dropout based on latent semantic variations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11229 11236, 2020.
Rohit Keshari, Richa Singh, and Mayank Vatsa. Guided dropout. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4065-4072, 2019.
Nikhil Ketkar and Nikhil Ketkar. Stochastic gradient descent. Deep learning with Python: A hands-on introduction, pp. 113 132, 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. or preprint arXiv:1611.01255 2016.
Daniel Leite, Pyramo Costa, and Fernando Gomide. Evolving granular neural networks from fuzzy data streams. Neural Networks, 38:1-16, 2013.
Xuejiao Liu and Xueshuang Xiang. How does arXw:2007.00602,, 2020.
semi-supervised learning work? arXiv preprint
Hussein Syed Mohammed, James Leander, Matthew Marbach, and Robi Polikar. Can adaboost. ml learn incrementall?? a comparison to I under different combination rules. In International Conference on Artificial Neural Networks, pp. 254-263. Springer, 2006.
Hai-Long Nguyen, Yew-Kwong Woon, and Wee- Wee-Keon Keong Ng. A survey on data stream clustering and classifi- cation. Knowledge and information systems, 45(3):535 569, 2015.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions knowledge and data engineering, 22110):3345-135, 2010.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Anauthram Swami. Distillation as a defense adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pp. 582 597. IEEE, 2016.
Jitendra Parmar, Satyendra Singh Chouhan, Vaskar Raychoudhury, and Santosh S Rathore. Open-world machine learning: applications, challenges, and opportunities. ACM Computing Surveys (CSUR), 2021.
Rajendra Patil, Vinay Sachidananda, Hongyi Peng, Akshay Sachdeva, and Mohan Gurusamy. Mark: Fill in the blanks through jointgan based data augmentation for network anomaly detection. Computers 8 Securily, 119:102759, 2022.
19

---

Published in Transactions on Machine Learning Research (05/2023)
Alvin Poernomo and Dae-Ki Kang. Biased dropout and crossmap dropout: learning towards effective dropout regularization in convolutional neural network. Neural nebworks, 104:60-F 67, 2018.
Robi Polikar. Ensemble learning, In Ensemble machine learning, pp. 1-34. Springer, 2012.
Robi Polikar, Lalita Upda, Satish Upda, and Vasant Honavar. Learn++: An incremental learning algorithm for supervised neural networks. IEEE transactions on systems, man, and cybernetics, part (opplications and revieus), (14):99775008 -08, 2001.
Robi Polikar, Joseph DePasquale, Hussein Syed Mohammed, Gavin Brown, and Ludmilla I Kuncheva. Learn++ mf: A random subspace approach for the missing feature problem. Pattern Recognition, 43
(11):3817-3832, 2010.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1666. 14060611, 2016.
Doyen Sahoo, Quang Pham, Jing Lu, and Steven CH Hoi. Online deep learning: Learning deep neural networks on the fly. Xiviv preprint orXiv:1711.0071 2017.
Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkola, and Terrance E Boult. Toward open set recognition. IEEE transactions on paltern analysis and machine intelligence, 35(7)11757-1772, 2012.
Vikash Sehwag, Arjun Nitin Bhagoji, Liwei Song, Chawin Sitawarin, Daniel Cullina, Mung Chiang, and Prateck Mittal. Analyzing the robustness of open-world machinc learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Securily, pp. 105 116, 2019.
Thomas Seidl, Ira Assent, Philipp Kranen, Ralph Krieger, and Jennifer Herrmann. Indexing density models for incremental learning and anytime classification data streams. In Proceedings of the 12th international conference extending database technology: advances in database technology, PD. 311 322, 2009.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object
localization using convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 648 656, 2015.
Ivor W Tsang, Andras Kocsor, and James T Kwok. Simpler core vector machines with enclosing balls. In Proceedings of the 24th international conference on Mochine learning, pp. 911 918, 2007.
Qin Zhang, Peng Zhang, Guodong Long, Wei Ding, Chengqi Zhang, and Xindong Wu. Online learning from trapezoidal data streams. IEEE Transactions Knowledge and Data Engineering. 28(10):2709 2770-273,, 2016.
Zhen- Yu Zhang, Peng Zhao, Yuan Jiang, and Zhi-Hua Zhou. Learning with feature and distribution evolvable streams. In International Conference on Machine Learning, pp. 11317-11827. PMLR, 2020.
Zhi-Hua Zhou. Open-environment machine learning. National Science Review, 9(8):nwac123; 2022.
A Adversarial Inputs
Adversarial inputs are malicious inputs designed to cause the machine-learning model to make mistake. This is achieved by modifying the input data subtly (Goodfellow et al. 2018). Machine learning models are highly vulnerable to small changes in the input at Lest time (Biggio et al. 2013 Kurakin et al. 2016).. Hence dealing
with adversarial inputs require special methods and training procedure like injecting adversarial examples into the training set (Huang et al., (215),, using defensive distillation for network training (Papernot et al., 2016). Whereas, haphazard inputs are not modified, disturbed, or perturbed. All the inputs are used as it is and inputs created at any point in time.
20

---

Published in Transactions on Machine Learning Research (05/2023)
B Semi-supervised techniques
Adaptations of gencrative adversarial networks (GAN)
Goodfellow et al. 2020) have achieved competitive
(Patil et al. 2022) attempt to use GAN to generate data that can exhibit the patterns the unknown test data and this generated data in turn can train the model to deal with adversarial inputs (Goodfellow et al. 2018 and unknown attacks. However, this does not apply to haphazard inputs, since it cannot deal with changing input dimensions and unavailable auxiliary leatures.
results in semi-supervised learning (Liu & Xiang 2020 Approaches like MARK
C Progressive Networks
Progressive networks (Rusu et al., 2016) are designed handle a complex sequence of tasks (Taski, Taskg, Task;) by leveraging transfer learning and avoiding catastrophic forgetting. Here, at Task, deep neural network (D1) is trained in offline manner. When Taskg arrives, copy of D1 is made with random initialization (let's say D2), and Dj is frozen. The input to D2 is now the hidden activations from Dj and the actual imput. This goes on till Task; Whereas in the haphazard inputs problem, there is only one task without any change and the model is trained in an online manner with a stochastic gradient. Hence, progressive networks can not be applied here. One can still think of a connection between progressive networks and Aux-Drop by considering cach new auxiliary feature as task-specific and creating a deep neural network whenever a new feature arrives. In a practical scenario, the number of auxiliary features can go up to thousands (121 in the case of the aSa dataset) which will require 1000 deep neural networks. This is not feasible from both the point of storage and computation.
D Domain Adaptation
Domain adaptation aims to train a model using source data (training data) in such way that the model generalizes well to the target data (test data) too (Farhhani et al. 2021]. Training and test data can be from different distributions, hence it becomes an important part of the model to explicitly handle the data from different domains (distribution). Domain adaptation is a special case of transfor learning (Pan & Yang. 2010). Interestingly, domain adaptation has only been applied in the online learning selting (Chen el al. 2021). But the biggest assumption in domain adaptation that all the features are base features and hence it cannot handle the haphazard input problems.
E
Open Set
Open set problems deal with scenarios where the number of classes (output labels) unknown the training time and new classes appear during testing (Scheirer et al. 2012). The haphazard inputs problem is orthogonal to the open set problem since the number of classes is always known and the number of input features is unknown.
21


Reviews:
[{"review_id": "8l3JiaJeYf", "reviewer": "Reviewer_hDLm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.Online learning is a real problem.\n\n2.This paper is well-written.\n\nWeaknesses:\n\n1.Technical contributions are limited. Now the contribution in the proposed method is engineering contribution rather than technical contribution in the perspective of machine learning.\n\n2.How the proposed method can handle the haphazard inputs is not very clear. \n\n3.For further improving this work, I would suggest clarifying concepts before diving into the technical section. For example, I do not think the concept of Haphazard Inputs  is well-established. Also, this concept can not be found in any paper. It would be better if the authors can convince me this is a well-established concepts in machine learning community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies online learning which produces streaming data. The authors claim that streaming data is haphazard in nature, which means data contains missing features, features becoming obsolete in time, the appearance of new features at later points in time, and a lack of clarity on the total number of input features. The existing haphazard inputs make it hard to build a learnable system. To tackle this challenge, an auxiliary dropout regularization strategy named Aux-Drop is proposed to help in better learning for scenarios where certain features disappear in time or when new features are to be modeled. Experimental results show the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.Technical contributions are limited. Now the contribution in the proposed method is engineering contribution rather than technical contribution in the perspective of machine learning.\n\n2.How the proposed method can handle the haphazard inputs is not very clear. \n\n3.For further improving this work, I would suggest clarifying concepts before diving into the technical section. For example, I do not think the concept of Haphazard Inputs  is well-established. Also, this concept can not be found in any paper. It would be better if the authors can convince me this is a well-established concepts in machine learning community.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0VZWOSu953", "reviewer": "Reviewer_MXSX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The online learning setup that incorporates various challenges from the real world is interesting.\n2. Competitive performance has been demonstrated by evaluating the proposed approach on multiple datasets of different sizes.\n\nWeaknesses (detailed in the section below):\n1. The primary objective of this paper is ambiguous, as it is not clear whether the aim is to introduce a new problem setup or a novel learning algorithm.\n2. The definition of \"haphazard\" lacks precision. It appears the notion of \"haphazard\" is intended to cover any potential problem that can arise in online learning, which makes the problem definition overly generic. The authors should clarify what is not considered as \"haphazard\" to better define the scope of the problem.\n3. Assumptions in empirical evaluation violates the broad scope of the problem definition.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Aux-Drop that aims to address challenges in online learning coined as “haphazard inputs.” The proposed approach is based on dropout and is applied to the concatenated representations across base and auxiliary features. Missing auxiliary features are first dropped before dropout is applied to the remaining features. The proposed approach is evaluated on various datasets and demonstrate to outperform baselines such as OLSF and OLVF.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. The primary objective of this paper is ambiguous, as it is not clear whether the aim is to introduce a new problem setup or a novel learning algorithm. If the primary emphasis of the paper is on a new problem setup, it is crucial to highlight the differences between the proposed setup and existing ones, and provide a clear rationale for why the new setup is desired. On the other hand, if the main focus is on proposed a new algorithm, it is more preferable to begin the introduction by discussing the limitations of existing approaches.\n\n2. The definition of \"haphazard\" lacks precision. It appears the notion of \"haphazard\" is intended to cover any potential problem that can arise in online learning, which makes the problem definition overly generic. The authors should clarify what is not considered as \"haphazard\" to better define the scope of the problem. If the proposed approach is intended to cover any online learning problem, does it include adversarial inputs?\n\n3. Assumptions in empirical evaluation violates the broad scope of the problem definition. The definition of \"haphazard\" includes missing/obsolete/sudden-loss of features. But in the proposed approach and evaluation setup, the base features are always predefined, which contradicts with the flexible problem setup—does the proposed setup include assumptions in prior on which feature is always available?\n\n4. Overall, I suggest the authors better define the problem/scope of this paper and try to prevent over generalizing the proposed approach.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nv05fZUGMN", "reviewer": "Reviewer_7CMD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n- Very interesting paper and a relevant topic\n- The motivation of technical contribution clear, I especially like Table 1\n- Empirical results are based on several datasets, adding credibility\n- The work seems to be sufficiently novel and contribution is non-trivial\n\n\nWEAKNESSES\n- The general motivation is rather weak. There is a lot of emphasis on the self-driving cars, aerospace and real applications. However, the datasets explored in the experimental section do not cover these areas at all and they are simulated. I suggest that the authors rewrite the intro significantly, focusing it specifically on the kinds of problems that they really solve and also make sure that the claims they make are actually supported by the datasets they use in experiments.\n- Four contributions are listed in the paper, but I find that only first two are the actual science contributions, whereas the remaining two points, which I propose to remove, simply describe the properties of the proposed method.\n- Additionally, the second contribution claims that the proposed technique can be seamlessly combined with existing neural networks. However, the experiments only cover the coupling of one neural architecture with the proposed layer. I request that the author provide experiments supporting this claim by demonstrating that it works with at least one more architecture that is differnt from the one they use in the current experimental set.\n- The description of the datasets is very poor in the experimental section. I had to go to the original dataset pages to learn that they are Monte-Carlo simulated and those pages do not contain enough information about the nature of the problem simulated. Please add a motivation subsection in the experiments section clearly describing the datasets and motivating why the use of these datasets is acceptable to support the claims put forward in the paper. The statement that a particular dataset was used in a previous paper is not a good motivation - please take this into account - I will not accept this excuse.\n- Related work section presents a number of relevant methods, but the experimental results compare against only two of them. Why is this the case? Could you please provide more comparisons to make results more convincing?\n- The method is clearly related to group dropout, Ke et al., Group-Wise Dynamic Dropout Based on Latent Semantic Variations, AAAI 2020, reducing the originality of the method. Could you please explain the novelty of your method with respect to dropout techniques utilizing grouping?\n- The paper provides no code, reproducibility is compromised. Could you please release the code?\n\nOverall, I'm optimistic about the paper outlook, however this is subject to authors delivering a thorough revision of the paper based on all the points I raised. Therefore, I do encourage the authors to provide a comprehensive revision of the paper.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the problem of online learning with features that can have time-variable size and distribution. New unknown features may appear, old features may disappear and become obsolete. The problem is handled via a dropout-inspired  AUX layer proposed in the paper.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I formulated some core change requests in the previous section, because they are tightly linked with weaknesses, please do not overlook\n- There is a clear link between the proposed method and the semi-supervised techniques based on filling the blanks, including the Pseudolikelihood (see Section 18.3 in the Deep Learning book by Goodfellow et al.) methods for example. It would be nice to have this link reflected in the literature review and I would encourage authors to also formulate a probabilistic model of their approach to strengthen the theoretical foundation of their paper.\n- Please discuss in detail how your method relates to Progressive networks, domain adaptation approaches and open set problems\n- Please provide reference for Italy Power Demand dataset\n- In Table 3 caption, please correct typo \"Aux-Drop is run 5 fives times\"\n- In Table 3, can you explain why the metric is changing abruptly with increasing $p$? What is the statistical significance of results? How many runs did you use to average the results? If it's only one run, the results presented are not statistically significant. Please run multiple seeds and present average results with confidence intervals. I mean this is logical, especially that you do follow a more rigorous statistical protocol in Tables 4,5. \n- ODL is an integral part of the proposed approach. However, the details of the method are not covered in the paper. Even though this is not original contribution of this paper, I believe it would be beneficial for readability reasons to provide a section that explains the ODL and how exactly it is used in conjunction with the proposed approach.\n- In Figure 4, please provide curves averaged over several runs along with confidence intervals (semi-transparent, same colour as the curve)\n- According to Table 8, the position of Aux layer has very little effect on performance. How can you explain it? What happens when you put it in position 1? Also in previous experiments you use 11 layers, and in this one only 6. Why? What happens if you use 11 layers?\n- I believe Appendices A and B provide important information and should be moved to the main body of the paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studied online learning with "haphazard inputs" that is a new concept defined as "missing features, features becoming obsolete in time, the appearance of new features at later points in time and a lack of clarity on the total number of input features". A dropout-inspired method called Aux-Drop (an auxiliary dropout regularization strategy) was proposed to solve the problem. In the beginning, the reviewers questioned that the contributions were quite engineering but not scientific and the details provided in the paper and the code were not enough to reproduce the results. During the rebuttal period, the authors did a particularly good job --- several parts have been rewritten and the concerns have been addressed. In the end, all reviewers agreed to accept the paper for publication. Moreover, Reviewer 7CMD commented that "this is an interesting paper that meets the technical soundness and reproducibility bars" and recommended the Reproducibility Certification, and I am happy to make the same recommendation. In the camera-ready version, please make sure to incorporate the discussion from the reply "response on assumption of base-features".
Decision: Accept (poster)