# Online Streaming Feature Selection via Conditional Independence
## 6 Conclusions  
We studied the online feature selection problem with streaming features. By employing multi-layer filtering strategies with conditional independence, we proposed an online learning algorithm called _ConInd_ to reduce the dimensionality of streaming features by removing irrelevant and redundant features in real time. The proposed _ConInd_ algorithm can output an approximate _Markov blanket_ in a short running time, with high accuracy even when the streaming features have low redundancy and high relevance. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Algorithms** & \multicolumn{4}{c}{**Average Accuracy for Classifiers in the 14 Datasets (\%)**} & \multicolumn{2}{c}{**Average Accuracy (\%)**} \\ \hline  & Complex & Medium & Simple & Linear & Quadratic & Cubic SVM & \\  & Tree & Tree & Tree & SVM & SVM & 87.59 \\  & 88.54 & 88.74 & 88.80 & 92.76 & 91.78 & 87.59 \\  & \multicolumn{2}{c}{Decision Tree average: 88.69} & \multicolumn{2}{c}{SVM average: 90.71} & \multirow{2}{*}{89.51} \\ \cline{2-2} \cline{6-6} \multirow{-2}{*}{_HITON\_MB_} & \multirow{2}{*}{Fine KNN} & Medium & Cubic & Bagged & Subspace & RUSBoosted \\  & & KNN & KNN & Trees & discriminant & Trees \\  & 89.11 & 90.98 & 90.56 & 88.61 & 91.61 & 85.00 \\  & \multicolumn{2}{c}{KNN average: 90.22} & \multicolumn{2}{c}{ENSEMBLE average: 88.41} \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Average prediction accuracies of _HITON_MB_. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & _HITON_MB_ & _OSFS_ & _ConInd_ \\ \hline wdbc & 5 & 3 & 3 \\ colon & 3 & 3 & 3 \\ lucas0 & 5 & 4 & 4 \\ sylav & 64 & 18 & 24 \\ ionosphere & 6 & 4 & 5 \\ cina0 & 66 & 22 & 30 \\ lucap0 & 79 & 36 & 40 \\ marti1 & 3 & 1 & 1 \\ reged1 & 24 & 13 & 13 \\ lung & 97 & 11 & 35 \\ prostate\_GE & 6 & 3 & 4 \\ leukemia & 7 & 3 & 8 \\ arcene & 8 & 5 & 6 \\ Smk\_can\_187 & 11 & 4 & 9 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: The number of selected features of the three algorithms. Our empirical study demonstrated that: (1) _ConInd_ has significant performance improvements in terms of accuracy prediction compared to _Alpha-investing_ and _OSFS_. The average increase in accuracy prediction was 5.62% higher than that of _Alpha-investing_ and 0.51% higher than that of _OSFS_. (2) _ConInd_ offers perfect efficiency in terms of running time when the datasets have low redundancy and high relevance. The running time was reduced by an average of 53.56% compared to that of _OSFS_. (3) _ConInd_ can retain as many features as possible in the _Markov blanket_, thus the features are not filtered. Meanwhile, although _ConInd_ obviously has high prediction accuracy, many features would also potentially be selected for some datasets. Therefore, it is necessary to conduct thorough theoretical analyses and empirical studies on the following: (1) how to obtain accurate _Markov blankets_; and (2) how to further improve the efficiency of _ConInd_ in terms of running time. In addition, deep learning can be viewed as a high dimensional nonlinear data reduction scheme [40], which represents a pervasive tendency in today's data analysis community. A deep learning model tries to learn the underlying structure of the feature space to learn a better representation of the feature. When the feature space grows over time, the challenges are shown in the following aspects: (1) in an online scenario, deep learning model would need to re-train everything when new features arrive; (2) if relevant features are introduced, uncertainty of the structure of the feature space increases; and (3) learning rate decay needs to be tuned again. Conceptualization, D.Y. and X.W. ; methodology, D.Y. and X.W. ; validation, D.Y. and Z.C. ; formal analysis, D.Y., X.W., Y.H. and X.Y. ; investigation, D.Y., X.W. and S.D. ; resources, X.W. and L.S. ; data curtain, D.Y. and Z.C. ; Writing--original draft preparation, D.Y., Y.H. and X.W. and X.Y. ; and Writing--review and editing, D.Y., C.M. and S.D. This work was supported in part by the US National Science Foundation (NSF) under grant Nos.