# Regularized F-Measure Maximization for Feature  Selection and Classification

Authors: Zhenqiu Liu; Ming Tan; Feng Jiang
Journal: Journal of Biomedicine and Biotechnology
 (2009)

---

## Abstract

Receiver Operating Characteristic (ROC) analysis is a common tool for
assessing the performance of various classifications. It gained much popularity in medical and other fields including biological markers and, diagnostic test. This is particularly due to the fact that in real-world problems
misclassification costs are not known, and thus, ROC curve and related utility
functions such as F-measure can be more meaningful performance measures.
F-measure combines recall and precision into a global measure. In this paper, we propose a novel method through regularized F-measure maximization.
The proposed method assigns different costs to positive and negative samples and does simultaneous feature selection and prediction withL1 penalty. This method is useful especially when data set is highly unbalanced, or the
labels for negative (positive) samples are missing. Our experiments with the
benchmark, methylation, and high dimensional microarray data show that the performance of proposed algorithm is better or equivalent compared with the other popular classifiers in limited experiments.


---

## Body

## 1. Introduction

Receiver Operating Characteristic (ROC) analysis has
received increasing attention in the recent statistics and machine learning
literatures (Pepe [1, 2]; Pepe  and Janes  [3]; Provost and Fawcett  [4];
Lasko et al.  [5]; Kun et al.  [6]). ROC analysis originates in signal
detection theory and is widely used in medical statistics for visualization and
comparison of performance of binary  classifiers. Traditionally, evaluation of a
classifier is done by minimizing an estimation of a generalization error or
some other related measures (Vapnik  [7]). However the accuracy (the rate of
correct classification) of a model does not always work. In fact when the data
are highly unbalanced, accuracy may be misleading, since the all-positive or
all-negative classifiers may achieve very good classification rate. In real
life applications, the situations for which the data sets are unbalanced arise
frequently. Utility functions such as F-measure or AUC provide a better way for
classifier evaluation, since they can assign different error costs for positive
and negative samples.When the goal is to achieve the best performance under
a ROC-based utility functions, it may be better to build classifiers through
directly optimizing the utility functions. In fact, optimizing the
log-likelihood function or the mean-square error does not necessarily imply
good ROC curve performance. Hence, several algorithms have been recently
developed for optimizing the area under ROC curve (AUC) function (Freund et
al.  [8]; Cortes and Mohri  [9]; Rakotomamonjy  [10]), and they have been
proven to work well with different degrees of success. However, there are not
many methods proposed for F-measure maximization. Most approaches to date that
we know of maximize F-measure using SVMs and do so by varying parameters in
standard SVM in an attempt to maximize F-measure as much as possible (Musicant
et al. [11]). While this may result in a “best possible” F-measure for a
standard SVM, there is no evidence that this technique should produce an
F-measure comparable with one from the classifier designed to specifically
optimize F-measure. Jansche [12] proposed an approximation algorithm for
F-measure maximization in the logistic regression framework. His method,
however, gives extremely large values for the estimated parameters and creates
too many steep gradients. It, therefore, either converges very slow or fails to
converge for large datasets.Our aim in this paper is to propose a novel algorithm
that directly optimizes an approximation of the regularized F-measure. The
regularization term can be anL2, L1 or a
combination of L1 and L2 penalty based
on different prior assumptions (Tibshirani [13, 14]; Wang et al. [15]). Due to
the nature of L1 penalty, our
algorithm provides simultaneous feature selection and classification with L1 penalty. The
proposed algorithm can be easily applied to high dimensional microarray data.
One advantage with this method is that it is very efficient when data is highly
unbalanced, since it assigns different costs to the positive and negative
samples.The paper is organized as follows. In Section2 we
introduce the related concept of ROC and F-measure. The algorithm and the brief
proof of its generalization bounds are proposed in Section 3. The computational
experiments and performance evaluation are given in Section 4. Finally the
conclusions and remarks are discussed in Section 5.


## 2. ROC Curves and F-Measure

In binary classification, a classifier attempts to map
the instances into two classes: positive (p) and negative (n). There are four
possible outcomes with the given classifier: true positives (TP), false
positives (FP), true negatives (TN), and false negatives (FN). Table1
summarizes these outcomes with their associated terminology. The number of
positive instances is Np=TP + FN. Similarly Nn=TN + FP is the number
of negative instances.Table 1
Classification outcomes.
PredictedTotal1−1True1TPFNNp−1FPTNNnMpMnFrom these counts the following statistics are
derived:(1)tpr=TPTP+FN,tnr=TNTN+FP,fpr=FPFP+TN,fnr=FNTP+FN,where true positive rate (also
called recall or sensitivity) is denoted by tpr and true
negative rate (specificity) by tnr. False positive rate and false negative rate are
denoted by fpr and fnr, respectively.
Note that tnr=1−fpr, and fnr=1−tpr. We also define the precision Pr=TP/(TP+FP). ROC curves plot the true positive rate versus false
positive rate by varying the threshold which is usually the probability of the
membership to a class, distance to a decision surface, or a score produced by a
decision function. In the ROC space, the upper left corner represents a perfect
classification, while a diagonal line represents random classification. A point
in ROC curve that lies upper left of another point represents a better model.F-measure combines the true positive rate (recall) and
precision Pr into a single
utility function which is defined asγ-weighted
harmonic mean:(2)Fγ=1γ(1/tpr)+(1−γ)(1/Pr),where0≤γ≤1.Fγ can be
expressed with TP, FP, and FN as follows:(3)Fγ=TPTP+γFN+(1−γ)FP or equivalently(4)Fγ=TPγNp+(1−γ)Mp, where Np is the number
of positive samples, and Mp=TP+FP. Clearly 0≤Fγ≤1 and Fγ=1 only when all
the data are classified correctly. Maximizing F-measure is equivalent to
maximizing the weighted sensitivity and specificity. Therefore, maximizing Fγ will indirectly
lead to maximize the area under ROC curve (AUC).To optimizeFγ, we have to define TP, FN, and FP mathematically.
We first introduce an indicator function(5)I(y∈C)={1,ify∈C,0,ify∉C,where C is a set. Let y=f(w,x) be a classifier
with coefficients (weights) w and input
variable x, and let y^ be the
predicted value. Given n samples, D={(x1,y1),…,(xn,yn)}, where xi is a
multidimensional input vector with dimension m and class label yi∈{−1,1}; TP, FN, and FP are given, respectively:(6)TP=∑i=1nI(y^i=1)I(yi=1),FN=∑i=1nI(y^i=−1)I(yi=1),(7)FP=∑i=1nI(y^i=1)I(yi=−1).It is clear that F-measure is a
utility function that applies for the whole data set.


## 3. The Algorithm

Usually given a
classifier with known parametersw, F-measure can be calculated with the test data to
evaluate the performance of the model. The aim of this paper is, however, to
learn a classifier and estimate the corresponding parameters w with a given
training data D and regularized
F-measure maximization. Since Fγ∈[0,1], we have −logFγ∈[0,∞). Statistically Fγ is a
probability that measures the proportion of samples correctly classified. Based
on these observations, we can maximize the logFγ in the maximum
log likelihood framework. Different assumptions for the prior distribution of w will lead to
different penalty terms. Given the coefficient vector w with dimension m, we have L2=(1/2)∑j=1m|wj|2 for the
assumption of Gaussian distribution and L1=∑j=1m|wj| with that of
Laplacian prior. In general, L1 penalty
encourages sparse solutions, while the classifiers with L2 are more
robust. We make TP, FN, and FP depend on w explicitly and
maximize the following penalized F-measure functions:(8)E1(w)=logFγ(TP(w),FN(w),FP(w))−λ∑j=1m|wj|,E2(w)=logFγ(TP(w),FN(w),FP(w))−12λ∑j=1m|wj|2.We have(9)w^=argmaxw^{logFγ(TP(w),FN(w),FP(w))−λ∑j=1m|wj|},w^=argmaxw^{logFγ(TP(w),FN(w),FP(w))−12λ∑j=1m|wj|2}.Note that TP(w), FN(w), and FP(w) are all
integers, and the index function I in (7)
is not differentiable. We first define an S-type function to approximate the
index function I: Let z=wTx be a linear
score function,(10)h(z)={0,z<−1,12(1+z)2,−1≤z≤0,12(2−(1−z)2),0<z≤1,1,z>1.The decision role such that y^(w,x)=1 if z=wTx>0 can be
represented as(11)I(y^=1)=I(z>0)=I(h(wTx)>0.5)≈h(wTx).Figure1 gives some insight about the h(z). Figure 1 shows
that h(z) is a better
approximation of I(z>0) than the
sigmoid function g(z)=1/(1+e−z). The first derivative of h(z) is continuous
and given in (12):(12)h′(z)=dh(z)dz={0,z<−1,1+z,−1≤z≤0,1−z,0<z≤1,0,z>1.Based on (10) and (11),
the approximated version of TP(w) and Mp(w)=TP(w)+FP(w) can be written
as follows:(13)TP(w)=∑i=1yi=1nh(wTxi),Mp(w)=∑i=1nh(wTxi). We can find the first-order derivatives of E1 and E2, respectively, as follows:(14)∂E1(w)∂wj=∂Fγ(w)/∂wjFγ(w)−λsign(wj),∂E2(w)∂wj=∂Fγ(w)/∂wjFγ(w)−λwj,where,(15)∂Fγ(w)∂wj=B∂TP(w)∂wj−B2TP(w)(1−γ)∂Mp(w)∂wj,B=1γNp+(1−γ)Mp(w),∂TP(w)∂wj=∑i=1yi=1nh′(wTx)xij,∂Mp(w)∂wj=∑i=1nh′(wTx)xij.Knowing E1 and E2, and their  derivatives ∇E1=[∂E1/∂wj] and ∇E2=[∂E2/∂wj], we can maximize the penalized function E1 and E2 with gradient
descent-related algorithm such as Broyden-Fletcher-Goldfarb-Shanno- (BFGS-)
related quasi-Newton method (Broyden [16]). The algorithm for E2 maximization is
straight forward as shown in  Algorithm 1. The step-size μ in the
algorithm can be found with line search.Algorithm 1:L2 regularized
F-measure maximization.
1. Givenγ, λ,
a small number ε,
Initialize wt=w0,
and sett=0.2.
While|wt+1−wt|>εwt+1=wt+μ(∇E2),
where μ is the step-size3.t=t+1Figure 1
The plot forh(z), indicator function I(z>0), and Sigmoid g(z)=1/(1+e−z).The regularized F-measure maximization withL1 penalty (E1) is of especial interest because it favors sparse
solutions and can select features automatically. However, maximizing E1 is a little bit
complex since L1 and E1 are not
differentiable at 0. For simplicity, let LF=logFγ(w), we have E1=LF−λ∑j=1m|wj|. The Karush-Kuhn-Tucker (KKT) conditions for
optimality are given as follows:(16)|∂LF∂wj|<λ⇒wj=0,wj≠0⇒|∂LF∂wj|=λ.The KKT conditions tell us that
we have a set Ψ of nonzero
coefficients which corresponds to the variables whose absolute value of
first-order derivative is maximal and equal to λ, and that all
variables with smaller derivatives have zero coefficients at the optimal
penalized solution. Since L1 is
differentiable everywhere except at 0, we can design an algorithm to deal with
the nonzero coefficients only. Algorithm 2
proposes an algorithm that can be
applied to the subspace of nonzero coefficient set denoted by Ψ. The algorithm has a procedure to add or remove
variables from Ψ, when the first-order derivative becomes large and
when a coefficient hits 0, respectively.Algorithm 2:L1 regularized
F-measure maximization.
1. Givenγ, λ, small numbers ε and δ, wt=w0, and sett = 0 and Ψ={j:wj≠0}.2. While|wt+1−wt|>εwt+1=wt+μ((∂LF∂wj)Ψ−λsign(wi)Ψ), where μ isthe step-sizeΨ=Ψ∪{j∉Ψ:|∂LFwj|>λ}Ψ=Ψ∖{j∈Ψ:|wj|<δ}3.t = t + 1


### 3.1. Computational Considerations

Bothγ and λ are free
parameters that need to be chosen. We will choose the best parameter for γ and λ with the area
under ROC curve (AUC). Area under the ROC curve (AUC) is another scalar measure
for classifier comparison. Its value is between (0,1). Larger AUC values indicate better classifier
performance across the full range of possible thresholds. For datasets with
skewed class or cost distribution is unknown as in our applications, AUC is a
better measure than prediction accuracy.Given a binary classification problem withNp positive class
samples and Nn negative class
samples, let f(x) be the score
function to rank a sample x. AUC is the probability that a classifier will rank a
randomly chosen positive instance higher than a randomly chosen negative
instance. Mathematically(17)AUC=∑i=1Np∑j=1NnI(f(xi)>f(yj))NpNn,where I(⋅) is an index
function and I(⋅)=1 if f(xi)>f(yj), otherwise I(⋅)=0. AUC is also called Wilcoxon-Mann-Whitney statistic
(Rakotomamonjy [10]).Note thatlogFγ(w) is generally a
nonconcave function with respect to w; only local maximum is guaranteed. One way to deal
with this difficulty is to employ the multiple-points initialization. Multiple
random points are generated, and our proposed algorithms are used to find the
maximum for each point. The result with the lowest test error is chosen as our
best solution.


## 3.1. Computational Considerations

Bothγ and λ are free
parameters that need to be chosen. We will choose the best parameter for γ and λ with the area
under ROC curve (AUC). Area under the ROC curve (AUC) is another scalar measure
for classifier comparison. Its value is between (0,1). Larger AUC values indicate better classifier
performance across the full range of possible thresholds. For datasets with
skewed class or cost distribution is unknown as in our applications, AUC is a
better measure than prediction accuracy.Given a binary classification problem withNp positive class
samples and Nn negative class
samples, let f(x) be the score
function to rank a sample x. AUC is the probability that a classifier will rank a
randomly chosen positive instance higher than a randomly chosen negative
instance. Mathematically(17)AUC=∑i=1Np∑j=1NnI(f(xi)>f(yj))NpNn,where I(⋅) is an index
function and I(⋅)=1 if f(xi)>f(yj), otherwise I(⋅)=0. AUC is also called Wilcoxon-Mann-Whitney statistic
(Rakotomamonjy [10]).Note thatlogFγ(w) is generally a
nonconcave function with respect to w; only local maximum is guaranteed. One way to deal
with this difficulty is to employ the multiple-points initialization. Multiple
random points are generated, and our proposed algorithms are used to find the
maximum for each point. The result with the lowest test error is chosen as our
best solution.


## 4. Computational Results

### 4.1. Benchmark Data

To evaluate the
performance of the proposed method, experiments were performed on six benchmark
datasets which can be downloaded fromhttp://ida.first.fraunhofer.de/projects/bench/benchmarks.htm. These benchmark datasets have been widely used
in model comparison studies in machine learning. They are all binary
classification problems, and the datasets were randomly divided into train and
test data 100 times to prevent bias and overfitting. The data are normalized
with zero mean and standard deviation. The overview of the datasets is given in
Table 2. The computational results with our algorithms,
logistic regression, and linear support vector machines are given in Figures 2-3.Table 2
Overview of the datasets.
DatasetsNo. of samples (train/test)No. of variablesNo. of experimentsBreast cancer200/779100Diabetis468/3008100Heart170/10013100German700/30020100Thyroid140/755100Titanic150/20513100Test Errors ofL2 F-Measure (FM),
Logistic Regression (LR), and SVM.Test AUC ofL2 F-Measure (FM),
Logistic Regression (LR), and SVM.Figures2-3 show that L2 F-measure
maximization performs better or equivalent compared with logistic regression
and linear support vector machines (SVM) in limited experiments. In fact, the
test errors for all datasets except for Thyroid are competitive with that of
the nonlinear classification methods reported by Ratsch (http://ida.first.fraunhofer.de/projects/bench/benchmarks.htm). The inferior performance of L2 F-measure with
Thyroid data indicates the strong nonlinear factors in that data.


### 4.2. Real Methylation Data

This
methylation data are from 7 CpG regions and 87 lung cancer cell lines (Virmani
et al. [17], Siegmund et al. [18]). 41 lines are from small cell lung cancer
and 46 lines from nonsmall cell lung cancer. The proportion of positive values
for the different regions ranges from 39% to 100% for the small
cell lung cancer and from 65% to 98% for the
nonsmall cell lung cancer. The data are available at http://www-rcf.usc.edu/kims/SupplementaryInfo.html. We utilize the twofold cross validation scheme to
choose the best λ and test our
algorithms. Other cross-validation schemes such as 10-fold cross validation
will lead to similar results but are more computational intensive. We randomly
split the data into two roughly equal-sized subsets and build the classifier
with one subset and test it with the other. To avoid the bias arising from a
particular partition, the procedure is repeated 100 times, each time splitting
the data randomly into two folds and doing the cross validation. The average
computational results with different γs and λ=0.05 are given in
Table 3. Table 3 shows
the selected variables (1: selected; 0: not selected), sensitivity,
specificity, test errors, and AUC values with different γ's. We can see
clearly the sensitivity increases while the specificity decreases as γ increases. When γ=0.9, every example is classified as positive examples.
The best γ will be 0.4
according to AUC but it will be 0.2 based on test error. Therefore, again
there is some inconsistence between two measures. Figure 4 gives some sight
about how to choose λ and the number
of features. Given γ=0.4, the optimal λ=0.04, and those 5 out of 7 CpG regions selected by L1 F-measure
maximization have been proved to be predictive of lung cancer subtype (Siegmund
et al. [18]). The performance of the model is improved roughly 6% in AUC and 3% in test error
with only 5 instead of 7 CpG regions.Table 3
Performance with differentγ's and L1 F-measure
maximization.
γVariables selected (1/0)SensitivitySpecificityTest errorAUC0.111011110.4760.95727.30.8010.211111110.7140.95715.90.8200.301011110.8100.74022.70.8490.411110010.8260.76220.40.8610.511111010.8570.60927.30.8320.611001010.7620.739250.8320.711101100.9040.348380.8470.810111001000.21740.90.8260.91100011100052.30.754Performance with differentλs and number of
variables.


### 4.3. High Dimensional Microarray Data

The colon
microarray data set (Alon et al. [19]) has 2000 features (genes) per sample
and 62 samples which consisted 22 normal and 40 cancer tissues. The task is to
distinguish tumor from normal tissues. The data set was first normalized for
each gene to have zero mean and unit variance. The transformed data was then
used for all the experiments. We employed a same twofold cross validation
scheme to evaluate the model. This computational experiments are repeated 100
times. The AUC was calculated after each cross validation. The computational
results for performance comparison are reported in Table 4.Table 4
Performance with differentγ's and L1 F-measure
maximization (λ=3).
γNo. of variablesFNFPNo. of misclassifiedAUC0.1101133440.5880.2103360.9890.3103360.9890.4103360.9890.5103360.9890.6103360.9890.7102350.9930.8102350.9930.9102570.98811028100.971Table4 gives us some insight that how the model
performance changes with different γ's. Generally
we can see that the false negative (FN) decreases and the false positive (FP)
increases as γ increases. The
only exception is when γ=0.1, both FN and FP have the worst performance. The best
performance is achieved when γ∈[0.7,0.8] according to
both AUC and the number of misclassified samples.The 10 genes selected are given in Table5. The selected
genes allow the separation of cancer from normal samples in the gene expression
map. Some genes were selected because their activities resulted in the
difference in the tissue composition between normal and cancer tissue. Other
genes were selected because they played a role in cancer formation or cell
proliferation. It was not surprise that some genes implicated in other types of
cancer such as breast and prostate cancers were identified in the context of
colon cancer because these tissue types shared similarity. Our method is
supported by the meaningful biological interpretation of selected genes. For
instance, three muscle-related genes (H20709, T92451, and J02854) were selected
from the colon cancer data, reflecting the fact that normal colon tissue had
higher muscle content, whereas colon cancer tissue had lower muscle content
(biased toward epithelial cells), and the selection of x12671 ribosomal protein
agreed with an observation that ribosomal protein genes had lower expression in
normal than in cancer colon tissue.Table 5
10 differentially expressed genes.
Gene IDDescriptionh20709myosin light chain alkali, smooth-muscle
isoform (human)t7102584103 human (human)m76378human cysteine-rich protein (crp) gene,
exons 5 and 6m63391human desmin gene, complete cdsz50753h.sapiens mrna for gcap-ii/uroguanylin
precursorr87126myosin heavy chain, nonmuscle (gallus
gallus)x12671human gene for heterogeneous nuclear
ribonucleoprotein (hnrnp) core protein a1t92451tropomyosin, fibroblast and epithelial
muscle-type (human)j02854myosin regulatory light chain 2, smooth muscle isoform (human); contains
element tar1 repetitive elementm36634human vasoactive intestinal peptide (vip)
mrna, complete cds


## 4.1. Benchmark Data

To evaluate the
performance of the proposed method, experiments were performed on six benchmark
datasets which can be downloaded fromhttp://ida.first.fraunhofer.de/projects/bench/benchmarks.htm. These benchmark datasets have been widely used
in model comparison studies in machine learning. They are all binary
classification problems, and the datasets were randomly divided into train and
test data 100 times to prevent bias and overfitting. The data are normalized
with zero mean and standard deviation. The overview of the datasets is given in
Table 2. The computational results with our algorithms,
logistic regression, and linear support vector machines are given in Figures 2-3.Table 2
Overview of the datasets.
DatasetsNo. of samples (train/test)No. of variablesNo. of experimentsBreast cancer200/779100Diabetis468/3008100Heart170/10013100German700/30020100Thyroid140/755100Titanic150/20513100Test Errors ofL2 F-Measure (FM),
Logistic Regression (LR), and SVM.Test AUC ofL2 F-Measure (FM),
Logistic Regression (LR), and SVM.Figures2-3 show that L2 F-measure
maximization performs better or equivalent compared with logistic regression
and linear support vector machines (SVM) in limited experiments. In fact, the
test errors for all datasets except for Thyroid are competitive with that of
the nonlinear classification methods reported by Ratsch (http://ida.first.fraunhofer.de/projects/bench/benchmarks.htm). The inferior performance of L2 F-measure with
Thyroid data indicates the strong nonlinear factors in that data.


## 4.2. Real Methylation Data

This
methylation data are from 7 CpG regions and 87 lung cancer cell lines (Virmani
et al. [17], Siegmund et al. [18]). 41 lines are from small cell lung cancer
and 46 lines from nonsmall cell lung cancer. The proportion of positive values
for the different regions ranges from 39% to 100% for the small
cell lung cancer and from 65% to 98% for the
nonsmall cell lung cancer. The data are available at http://www-rcf.usc.edu/kims/SupplementaryInfo.html. We utilize the twofold cross validation scheme to
choose the best λ and test our
algorithms. Other cross-validation schemes such as 10-fold cross validation
will lead to similar results but are more computational intensive. We randomly
split the data into two roughly equal-sized subsets and build the classifier
with one subset and test it with the other. To avoid the bias arising from a
particular partition, the procedure is repeated 100 times, each time splitting
the data randomly into two folds and doing the cross validation. The average
computational results with different γs and λ=0.05 are given in
Table 3. Table 3 shows
the selected variables (1: selected; 0: not selected), sensitivity,
specificity, test errors, and AUC values with different γ's. We can see
clearly the sensitivity increases while the specificity decreases as γ increases. When γ=0.9, every example is classified as positive examples.
The best γ will be 0.4
according to AUC but it will be 0.2 based on test error. Therefore, again
there is some inconsistence between two measures. Figure 4 gives some sight
about how to choose λ and the number
of features. Given γ=0.4, the optimal λ=0.04, and those 5 out of 7 CpG regions selected by L1 F-measure
maximization have been proved to be predictive of lung cancer subtype (Siegmund
et al. [18]). The performance of the model is improved roughly 6% in AUC and 3% in test error
with only 5 instead of 7 CpG regions.Table 3
Performance with differentγ's and L1 F-measure
maximization.
γVariables selected (1/0)SensitivitySpecificityTest errorAUC0.111011110.4760.95727.30.8010.211111110.7140.95715.90.8200.301011110.8100.74022.70.8490.411110010.8260.76220.40.8610.511111010.8570.60927.30.8320.611001010.7620.739250.8320.711101100.9040.348380.8470.810111001000.21740.90.8260.91100011100052.30.754Performance with differentλs and number of
variables.


## 4.3. High Dimensional Microarray Data

The colon
microarray data set (Alon et al. [19]) has 2000 features (genes) per sample
and 62 samples which consisted 22 normal and 40 cancer tissues. The task is to
distinguish tumor from normal tissues. The data set was first normalized for
each gene to have zero mean and unit variance. The transformed data was then
used for all the experiments. We employed a same twofold cross validation
scheme to evaluate the model. This computational experiments are repeated 100
times. The AUC was calculated after each cross validation. The computational
results for performance comparison are reported in Table 4.Table 4
Performance with differentγ's and L1 F-measure
maximization (λ=3).
γNo. of variablesFNFPNo. of misclassifiedAUC0.1101133440.5880.2103360.9890.3103360.9890.4103360.9890.5103360.9890.6103360.9890.7102350.9930.8102350.9930.9102570.98811028100.971Table4 gives us some insight that how the model
performance changes with different γ's. Generally
we can see that the false negative (FN) decreases and the false positive (FP)
increases as γ increases. The
only exception is when γ=0.1, both FN and FP have the worst performance. The best
performance is achieved when γ∈[0.7,0.8] according to
both AUC and the number of misclassified samples.The 10 genes selected are given in Table5. The selected
genes allow the separation of cancer from normal samples in the gene expression
map. Some genes were selected because their activities resulted in the
difference in the tissue composition between normal and cancer tissue. Other
genes were selected because they played a role in cancer formation or cell
proliferation. It was not surprise that some genes implicated in other types of
cancer such as breast and prostate cancers were identified in the context of
colon cancer because these tissue types shared similarity. Our method is
supported by the meaningful biological interpretation of selected genes. For
instance, three muscle-related genes (H20709, T92451, and J02854) were selected
from the colon cancer data, reflecting the fact that normal colon tissue had
higher muscle content, whereas colon cancer tissue had lower muscle content
(biased toward epithelial cells), and the selection of x12671 ribosomal protein
agreed with an observation that ribosomal protein genes had lower expression in
normal than in cancer colon tissue.Table 5
10 differentially expressed genes.
Gene IDDescriptionh20709myosin light chain alkali, smooth-muscle
isoform (human)t7102584103 human (human)m76378human cysteine-rich protein (crp) gene,
exons 5 and 6m63391human desmin gene, complete cdsz50753h.sapiens mrna for gcap-ii/uroguanylin
precursorr87126myosin heavy chain, nonmuscle (gallus
gallus)x12671human gene for heterogeneous nuclear
ribonucleoprotein (hnrnp) core protein a1t92451tropomyosin, fibroblast and epithelial
muscle-type (human)j02854myosin regulatory light chain 2, smooth muscle isoform (human); contains
element tar1 repetitive elementm36634human vasoactive intestinal peptide (vip)
mrna, complete cds


## 5. Conclusions and Remarks

We have presented  a novel regularized F-measure
maximization for feature selection and classification. This technique directly
maximizes the tradeoff between specificity and sensitivity. Regularization withL2 and L1 allows the
algorithm to converge quickly and to do simultaneous feature selection and
classification. We found that it has better or equivalent performances when
compared with the other popular classifiers in limited experiments.The proposed  method has the ability to incorporate
nonstandard tradeoffs between sensitivity and specificity with differentγ. It is well suited for dealing with unbalanced data
or data with missing negative (positive) samples. For instance, in the problem
of gene function prediction, the available information is only about positive
samples. In other words, we know which genes have the function of interested,
while it is generally unclear which genes do not have the function. Most
standard classification methods will fail but our method can train the model
with only positive labels by setting γ=1.One difficulty with the regularized F-measure
maximization is the nonconcavity of the error function. We utilized the random
multiple points initialization to find the optimal solutions. More efficient
algorithms for nonconcave optimization will be considered to speed up the
computations. The applications of the proposed method in gene function
predictions and others will be explored in the future.



---

*Source: 617946-2009-04-27.xml*