# Dependence Representation Learning with Convolutional Neural Networks and 2D Histograms
## 4 Feature Generation
### Improving Feature Generation  
The AutoLearn feature generation process has two limitations: (1) classifying dependence types by setting the dCor threshold to 0.7; and (2) generating new features for almost every feature pair. As shown in Figure 1, we should not use dCor to specify a criterion for classifying dependence types. AutoLearn generates a huge number of new features because it operates on a strict policy of generating new features for every feature pair unless the features in the pair are independent of each other. If the feature selection in step 3 does not select appropriate features, then useless features are output.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Sample Size** & **dCor**[4] & **dCor**[29] & **CorrCNN (CPU)** & **CorrCNN (GPU)** \\ \hline
221 & - & 30.7254 & 0.2203 & **0.1722** \\
222 & - & - & 0.5344 & **0.4868** \\
223 & - & - & 0.8170 & **0.7844** \\
224 & - & - & 1.4447 & **1.3933** \\
25 & - & - & 2.7110 & **2.7036** \\ \hline \hline \end{tabular}
\end{table}
Table 3: \(Cont\).  
To tackle these two problems, we apply dependence representation learning to feature generation. We train a CNN to predict the usefulness of the features generated before actually generating the features. We predict whether features will be useful directly from the 2D histograms. The CNN is trained to predict whether the generated features with LRR or KRR will be useful for improving the classification accuracy. A target function for feature generation, called \(target_{gen}\), is described in Algorithm 2. During the training, the target function actually performs the evaluation by applying the classification algorithm to the generated features. If the generated features improve the classification accuracy, then the target function returns 1, otherwise, it returns 0.  
```
1:Feature pair \((F_{x},F_{y})\), regression algorithm \(Regr\), class label vector \(C\)
2:Learning target \(T\)
3:\(F_{new}\gets generate(F_{x},F_{y},Regr)\)
4:\(score_{orig}\gets evaluate(C,[F_{x},F_{y}])\)
5:\(score_{new}\gets evaluate(C,[F_{x},F_{y},F_{new}])\)
6:if\(score_{new}>score_{orig}\)then
7:return 1
8:else
9:return 0
10:endif
```  
**Algorithm 2** Target function for feature generation \(target_{gen}\).  
The objective function for the feature generation \(\mathcal{L}_{gen}\), is composed of two parts, \(\mathcal{L}_{lgen}\) and \(\mathcal{L}_{kgen}\):  
\[\mathcal{L}_{gen}=\mathcal{L}_{lgen}+\mathcal{L}_{kgen} \tag{10}\]  
\(\mathcal{L}_{lgen}\) is the loss function for predicting the LRR, which is defined by:  
\[\mathcal{L}_{lgen}=H(target_{gen}(F_{x},F_{y},LRR),\sigma(w_{lgen}\cdot Z)), \tag{11}\]  
and \(\mathcal{L}_{kgen}\) is the corresponding function for the KRR:  
\[\mathcal{L}_{kgen}=H(target_{gen}(F_{x},F_{y},KRR),\sigma(w_{kgen}\cdot Z)). \tag{12}\]  
Note that we carry out multi-label classification because both LRR and KRR may be helpful. However, the training is performed using both of \(\mathcal{L}_{gen}\) and \(\mathcal{L}_{cor}\):  
\[\min_{\theta,W}\mathbb{E}_{F_{x},F_{y}\sim\mathbb{F}_{pair}}\big{[}\mathcal{L} _{gen}+\mathcal{L}_{cor}\big{]}. \tag{13}\]  
We add \(\mathcal{L}_{cor}\) because it helps the CNN learn the dependence representations better.  
The prediction probability \(P_{lgen}(F_{x},F_{y})\), which indicates the usefulness of the generated features based on \((F_{x},F_{y})\) and LRR, is defined as:  
\[P_{lgen}(F_{x},F_{y})=\sigma(w_{lgen}\cdot Z), \tag{14}\]  
and the corresponding probability for KRR is defined as:  
\[P_{kgen}(F_{x},F_{y})=\sigma(w_{kgen}\cdot Z). \tag{15}\]If one of the probabilities \(P_{lgen}\) and \(P_{kgen}\) is greater than some threshold \(\alpha\), then new features are generated using the corresponding regression algorithm. If both of the probabilities exceed the threshold, then new features are generated using KRR. No feature is generated for \((F_{x},F_{y})\) if neither probability is greater than the threshold.