BACKGROUND
microarray technology is one of the most powerful high-throughput tools in biomedical and biological research.
it has been successfully applied to various studies such as cancer classification  <cit> , drug discovery  <cit> , stress response  <cit> , and cell cycle regulation  <cit> .
microarray data contain missing values due to various technological limitations such as poor hybridization, spotting problems, insufficient resolution, and fabrication errors.
unfortunately, the missing values in microarray data would significantly degrade the performance of downstream analyses such as gene clustering and identification of differentially expressed genes .
therefore, missing value imputation has become an important pre-processing step in microarray data analyses.
one way to deal with the missing values is to repeat the experiments but it is expensive and time consuming.
another way is to discard the genes with missing values but this loses valuable information.
filling missing values with zeros or with the row average is a simple imputation strategy, but it is far from optimal.
therefore, many advanced algorithms have been developed to impute the missing values in microarray data .
the existing algorithms can be divided into four categories  <cit> : global approach, local approach, hybrid approach and knowledge-assisted approach.
global approach algorithms include svd  <cit>  and bpca  <cit> .
local approach algorithms include knn  <cit> , sknn  <cit> , iknn  <cit> , ls  <cit> , lls  <cit> , slls  <cit> , ills  <cit> , shrinkage lls  <cit>  and so on.
hybrid approach algorithms include lincmb  <cit>  and rmi  <cit> .
knowledge-assisted approach algorithms include goimpute  <cit> , pocsimpute  <cit>  and haiimpute  <cit> .
in order to know which algorithm performs best among the dozens of existing ones, an objective and comprehensive performance comparison framework is urgently needed.
to meet the need, we previously developed a performance comparison framework  <cit>  which provides  <dig> testing microarray datasets, three types of performance indices,  <dig> existing algorithms, and  <dig> runs of simulation.
we found that no single algorithm can perform best for all types of microarray data.
the best algorithms are different for different microarray data types  and different performance indices, showing the usefulness of our framework for conducting a comprehensive performance comparison  <cit> .
actually, the most important value of our framework is to give an objective and comprehensive performance evaluation of a new algorithm.
using our framework, bioinformaticians who design new algorithms can easily know their algorithms’ performance and then refine their algorithms if needed.
however, constructing our framework is not an easy task for the interested bioinformaticians.
it involves collecting and processing many microarray raw data from the public domain and using programming languages to implement many existing algorithms and three performance indices.
in order to save bioinformaticians’ efforts and time, we present an easy-to-use web tool named mviaeval  which implements our performance comparison framework.
implementation
twenty benchmark microarray datasets and twelve existing algorithms used for performance comparison
in mviaeval, we collected  <dig> benchmark microarray datasets  of different species and different types .
in addition, we implemented  <dig> existing algorithms including two global approach algorithms and  <dig> local approach algorithms .
do note that we did not include hybrid approach algorithms and knowledge-assisted algorithms because they either are difficult to implement or need extra information from outside data sources which are not always available.table  <dig> the  <dig> benchmark microarray datasets of different types and different species
geo
gds3323
gds3215
homo sapiens
gds3485
mus musculus
gds3476
mus musculus
gds3197
mus musculus
gds3149
mus musculus
gds2107
rattus
norvegicus
gds3464
danio rerio
gds3426
staphylococcus
epidermidis
gds3421
escherichia
coli
gds3360
homo sapiens
gds2863
gds5057
mus musculus
gds5055
mus musculus
gds3428
homo sapiens
gds4484
mus musculus
gds3785
gds3930
rattus
norvegicus
gds4321
escherichia
coli
gds3032
three existing performance indices used for performance evaluation
in mviaeval, we used three existing performance indices for performance evaluation.
first, the inverse of the normalized root mean square error   <cit>  is used to measure the numerical similarity between the imputed matrix  and the original complete matrix.
therefore, the higher the 1/nrmse value is, the better the performance of an imputation algorithm is.
second, the cluster pair proportion   <cit>  is used to measure the similarity of the gene clustering results of the imputed matrix and the complete matrix.
high cpp value means that the imputed matrix  has very similar gene clustering results as the complete matrix does.
therefore, the higher the cpp value is, the better the performance of an imputation algorithm is.
third, the biomarker list concordance index   <cit>  is used to measure the similarity of the differentially expressed genes identification results of the imputed matrix and the complete matrix.
high blci value means that differentially expressed genes identified using the imputed matrix  are very similar to those identified using the complete matrix.
therefore, the higher the blci value is, the better the performance of an imputation algorithm is.
in summary, 1/nrmse measures the numerical similarity, while cpp and blci measure the similarity of downstream analysis results  of the imputed matrix and the complete matrix.
fig.
<dig> shows how the scores of these three performance indices are calculated.fig.
<dig> three performance indices implemented in mviaeval.
mviaeval implements three performance indices, which are a 1/nrmse, b cpp and c blci.
here we provide an example to show how the scores of these three performance indices are calculated
evaluating the performance of an algorithm for a benchmark microarray data matrix using a specific performance index
the simulation procedure for evaluating the performance of an imputation algorithm  for a given complete benchmark microarray data matrix using a performance index  is divided into four steps.
step 1: generate five testing matrices having missing values  with different percentages  from the complete matrix.
step 2: generate five imputed matrices by imputing the missing values in the five testing matrices using knn.
step 3: calculate five cpp scores using the complete matrix and five imputed matrices.
step 4: repeat steps 1– <dig> for b times, where b is the number of simulation runs per missing percentage.
then the final cpp score of knn for the given benchmark microarray data matrix is defined as the average of the 5*b cpp scores.
fig.
<dig> illustrates the whole simulation procedure.fig.
<dig> the simulation procedure for evaluating the performance of an algorithm.
the simulation procedure for evaluating the performance of an imputation algorithm  for a given complete benchmark microarray data matrix using a performance index  is divided into four steps
two existing comprehensive performance scores
in mviaeval, we implemented two existing comprehensive performance scores  <cit>  to provide the overall performance comparison results for the selected benchmark microarray datasets and performance indices.
the first one, termed the overall ranking score , is defined as the sum of the rankings of an algorithm for the selected performance indices and benchmark microarray datasets  <cit> .
the ranking of an algorithm for a specific performance index and a specific benchmark microarray dataset is d if its performance ranks #d among all the compared algorithms.
for instance, the best algorithm has ranking  <dig>  therefore, small ors indicates that an algorithm has good overall performance.
the other comprehensive performance score, termed the overall normalized score , is calculated by the sum of the normalized scores for the benchmark microarray datasets and performance indices  <cit> .
the ons of the algorithm k is calculated like the following: ons=∑i=1i∑j=1jnij=∑i=1i∑j=1jmax,sij,…,sij)) where s
ij and n
ij is the original score and the normalized score of the algorithm k for the selected performance index i and benchmark microarray dataset j, respectively; i is the number of the selected indices; j is the number of the selected benchmark microarray datasets and m is the number of the algorithms being compared.
note that 0 ≤ n
ij ≤  <dig> and n
ij =  <dig> when the algorithm k performs best for the selected performance index i and benchmark microarray dataset j  = max, s
ij, …, s
ij)).
therefore, large ons indicates that an algorithm has good overall performance.
RESULTS
usage
figure  <dig> illustrates the usage of mviaeval.
the easy-to-use web interface allows users to upload the r code of their newly developed algorithm.
subsequently five types of settings of mviaeval need to be set.
first, the test datasets have to be chosen from  <dig> benchmark microarray datasets.
the collected benchmark datasets consist of two types of data:  <dig> non-time series data and  <dig> time series data.
second, the compared algorithms have to be chosen from  <dig> existing algorithms.
the collected existing algorithms consist of two global approach algorithms and  <dig> local approach algorithms.
third, the performance indices have to be chosen from three existing ones .
fourth, the comprehensive performance scores have to be chosen from two existing ones .
fifth, the number of simulation runs have to be specified.
the larger the number of simulation runs is, the more accurate the comprehensive performance comparison result is.
but be cautious that the simulation time increases linearly with the number of simulation runs.
after submission, a comprehensive performance comparison between the user’s algorithm and the selected existing algorithms is executed by mviaeval using the selected benchmark datasets and performance indices.
then a webpage of the comprehensive performance comparison results is generated and the webpage link is sent to the users by e-mails.fig.
<dig> the flowchart of mviaeval.
the flowchart shows how mviaeval conducts a comprehensive performance comparison for a new algorithm
a case study
in mviaeval, the r code of a sample algorithm is provided.
for demonstration purpose, we regard the sample algorithm as the user’s newly developed algorithm and would like to use mviaeval to conduct a comprehensive performance comparison of this new algorithm  to various existing algorithms.
for example, users may upload the r code of the new algorithm and select  two benchmark datasets,   <dig> existing algorithms,  three performance indices,  the overall ranking score as the comprehensive performance score, and   <dig> simulation runs .
after submission, mviaeval outputs the comprehensive comparison results in both tables and figures.
among the  <dig> compared algorithms, the overall performance of the new algorithm ranks six .
actually, mviaeval can provide the performance comparison results in many scenarios .
it can be concluded that the new algorithm is mediocre because its performance is always in the middle of all the  <dig> compared algorithms in different data types , different performance indices  and different comprehensive performance scores .
receiving the comprehensive comparison results from mviaeval, researchers immediately know that there is much room to improve the performance of their new algorithm.fig.
<dig> the input and five settings of mviaeval.
users need to a upload the r code of their new algorithm, b select the test datasets among  <dig> benchmark microarray  datasets, c select the compared algorithms among  <dig> existing algorithms, d select the performance indices from three existing ones, the comprehensive performance scores from two possible choices, and the number of simulation runs
fig.
<dig> the output of mviaeval.
for demonstration purpose, we upload the r code of a sample algorithm as the user’s new algorithm and select two benchmark datasets ,  <dig> existing algorithms, three performance indices, the overall ranking score as the comprehensive performance score, and  <dig> simulation runs.
a the webpage of the comprehensive performance comparison results shows that the overall performance of the user’s algorithm  ranks six among all the  <dig> compared algorithms.
b by clicking “details” in the row of blci for the benchmark dataset gds <dig>  users can see the performance comparison results using only blci score for the benchmark dataset gds <dig>  it can be seen that the user’s algorithm ranks five among the  <dig> compared algorithms using only blci score for the benchmark dataset gds <dig>  the details of blci score for each algorithm can also be found
performance
the performance comparison results of the user’s algorithm  and various existing algorithms using different types of datasets , different performance indices , and different overall performance scores  or overall normalized score ) are shown.
more details could be seen at http://cosbi.ee.ncku.edu.tw/mviaeval/a_case_study
CONCLUSIONS
missing value imputation is an inevitable pre-processing step of microarray data analyses.
this is why the computational imputation of the missing values in microarray data has become a hot research topic.
the newest algorithm is published in year  <dig>  <cit>  and we believe that many new algorithms will be developed in the near future.
using mviaeval, bioinformaticians can easily get a comprehensive and objective performance comparison results of their new algorithm.
therefore, bioinformaticians now can focus on developing new algorithms instead of putting a lot of efforts for conducting a comprehensive and objective performance evaluation of their new algorithm.
in conclusion, mviaeval will definitely be a very useful tool for developing missing value imputation algorithms.
abbreviations
mviaevalmissing value imputation algorithm evaluator
nrmsenormalized root mean square error
cppcluster pair proportion
blcibiomarker list concordance index
orsoverall ranking score
onsoverall normalized score

 microarray technology is one of the most powerful high-throughput tools in biomedical and biological research. it has been successfully applied to various studies such as cancer classification, drug discovery, stress response, and cell cycle regulation. microarray data contain missing values due to various technological limitations such as poor hybridization, spotting problems, insufficient resolution, and fabrication errors. unfortunately, the missing values in microarray data would significantly degrade the performance of downstream analyses such as gene clustering and identification of differentially expressed genes. therefore, missing value imputation has become an important pre-processing step in microarray data analyses. one way to deal with the missing values is to repeat the experiments but it is expensive and time consuming. another way is to discard the genes with missing values but this loses valuable information. filling missing values with zeros or with the row average is a simple imputation strategy, but it is far from optimal. therefore, many advanced algorithms have been developed to impute the missing values in microarray data. the existing algorithms can be divided into four categories: global approach, local approach, hybrid approach and knowledge-assisted approach. global approach algorithms include svd and bpca. local approach algorithms include knn, sknn, iknn, ls, lls, slls, ills, shrinkage lls and so on. hybrid approach algorithms include lincmb and rmi. knowledge-assisted approach algorithms include goimpute, pocsimpute and haiimpute. in order to know which algorithm performs best among the dozens of existing ones, an objective and comprehensive performance comparison framework is urgently needed. to meet the need, we previously developed a performance comparison framework which provides testing microarray datasets, three types of performance indices, existing algorithms, and runs of simulation. we found that no single algorithm can perform best for all types of microarray data. the best algorithms are different for different microarray data types and different performance indices, showing the usefulness of our framework for conducting a comprehensive performance comparison. actually, the most important value of our framework is to give an objective and comprehensive performance evaluation of a new algorithm. using our framework, bioinformaticians who design new algorithms can easily know their algorithms’ performance and then refine their algorithms if needed. however, constructing our framework is not an easy task for the interested bioinformaticians. it involves collecting and processing many microarray raw data from the public domain and using programming languages to implement many existing algorithms and three performance indices. in order to save bioinformaticians’ efforts and time, we present an easy-to-use web tool named mviaeval which implements our performance comparison framework. implementation twenty benchmark microarray datasets and twelve existing algorithms used for performance comparison in mviaeval, we collected benchmark microarray datasets of different species and different types. in addition, we implemented existing algorithms including two global approach algorithms and local approach algorithms. do note that we did not include hybrid approach algorithms and knowledge-assisted algorithms because they either are difficult to implement or need extra information from outside data sources which are not always available.table the benchmark microarray datasets of different types and different species geo gds3323 gds3215 homo sapiens gds3485 mus musculus gds3476 mus musculus gds3197 mus musculus gds3149 mus musculus gds2107 rattus norvegicus gds3464 danio rerio gds3426 staphylococcus epidermidis gds3421 escherichia coli gds3360 homo sapiens gds2863 gds5057 mus musculus gds5055 mus musculus gds3428 homo sapiens gds4484 mus musculus gds3785 gds3930 rattus norvegicus gds4321 escherichia coli gds3032 three existing performance indices used for performance evaluation in mviaeval, we used three existing performance indices for performance evaluation. first, the inverse of the normalized root mean square error is used to measure the numerical similarity between the imputed matrix and the original complete matrix. therefore, the higher the 1/nrmse value is, the better the performance of an imputation algorithm is. second, the cluster pair proportion is used to measure the similarity of the gene clustering results of the imputed matrix and the complete matrix. high cpp value means that the imputed matrix has very similar gene clustering results as the complete matrix does. therefore, the higher the cpp value is, the better the performance of an imputation algorithm is. third, the biomarker list concordance index is used to measure the similarity of the differentially expressed genes identification results of the imputed matrix and the complete matrix. high blci value means that differentially expressed genes identified using the imputed matrix are very similar to those identified using the complete matrix. therefore, the higher the blci value is, the better the performance of an imputation algorithm is. in summary, 1/nrmse measures the numerical similarity, while cpp and blci measure the similarity of downstream analysis results of the imputed matrix and the complete matrix. fig.  shows how the scores of these three performance indices are calculated.fig. three performance indices implemented in mviaeval. mviaeval implements three performance indices, which are a 1/nrmse, b cpp and c blci. here we provide an example to show how the scores of these three performance indices are calculated evaluating the performance of an algorithm for a benchmark microarray data matrix using a specific performance index the simulation procedure for evaluating the performance of an imputation algorithm for a given complete benchmark microarray data matrix using a performance index is divided into four steps. step 1: generate five testing matrices having missing values with different percentages from the complete matrix. step 2: generate five imputed matrices by imputing the missing values in the five testing matrices using knn. step 3: calculate five cpp scores using the complete matrix and five imputed matrices. step 4: repeat steps 1– for b times, where b is the number of simulation runs per missing percentage. then the final cpp score of knn for the given benchmark microarray data matrix is defined as the average of the 5*b cpp scores. fig.  illustrates the whole simulation procedure.fig. the simulation procedure for evaluating the performance of an algorithm. the simulation procedure for evaluating the performance of an imputation algorithm for a given complete benchmark microarray data matrix using a performance index is divided into four steps two existing comprehensive performance scores in mviaeval, we implemented two existing comprehensive performance scores to provide the overall performance comparison results for the selected benchmark microarray datasets and performance indices. the first one, termed the overall ranking score, is defined as the sum of the rankings of an algorithm for the selected performance indices and benchmark microarray datasets. the ranking of an algorithm for a specific performance index and a specific benchmark microarray dataset is d if its performance ranks #d among all the compared algorithms. for instance, the best algorithm has ranking therefore, small ors indicates that an algorithm has good overall performance. the other comprehensive performance score, termed the overall normalized score, is calculated by the sum of the normalized scores for the benchmark microarray datasets and performance indices. the ons of the algorithm k is calculated like the following: ons=∑i=1i∑j=1jnij=∑i=1i∑j=1jmax,sij,…,sij)) where s ij and n ij is the original score and the normalized score of the algorithm k for the selected performance index i and benchmark microarray dataset j, respectively; i is the number of the selected indices; j is the number of the selected benchmark microarray datasets and m is the number of the algorithms being compared. note that 0 ≤ n ij ≤  and n ij =  when the algorithm k performs best for the selected performance index i and benchmark microarray dataset j  = max, s ij, …, s ij)). therefore, large ons indicates that an algorithm has good overall performance.  usage figure  illustrates the usage of mviaeval. the easy-to-use web interface allows users to upload the r code of their newly developed algorithm. subsequently five types of settings of mviaeval need to be set. first, the test datasets have to be chosen from benchmark microarray datasets. the collected benchmark datasets consist of two types of data: non-time series data and time series data. second, the compared algorithms have to be chosen from existing algorithms. the collected existing algorithms consist of two global approach algorithms and local approach algorithms. third, the performance indices have to be chosen from three existing ones. fourth, the comprehensive performance scores have to be chosen from two existing ones. fifth, the number of simulation runs have to be specified. the larger the number of simulation runs is, the more accurate the comprehensive performance comparison result is. but be cautious that the simulation time increases linearly with the number of simulation runs. after submission, a comprehensive performance comparison between the user’s algorithm and the selected existing algorithms is executed by mviaeval using the selected benchmark datasets and performance indices. then a webpage of the comprehensive performance comparison results is generated and the webpage link is sent to the users by e-mails.fig. the flowchart of mviaeval. the flowchart shows how mviaeval conducts a comprehensive performance comparison for a new algorithm a case study in mviaeval, the r code of a sample algorithm is provided. for demonstration purpose, we regard the sample algorithm as the user’s newly developed algorithm and would like to use mviaeval to conduct a comprehensive performance comparison of this new algorithm to various existing algorithms. for example, users may upload the r code of the new algorithm and select two benchmark datasets, existing algorithms, three performance indices, the overall ranking score as the comprehensive performance score, and simulation runs. after submission, mviaeval outputs the comprehensive comparison results in both tables and figures. among the compared algorithms, the overall performance of the new algorithm ranks six. actually, mviaeval can provide the performance comparison results in many scenarios. it can be concluded that the new algorithm is mediocre because its performance is always in the middle of all the compared algorithms in different data types, different performance indices and different comprehensive performance scores. receiving the comprehensive comparison results from mviaeval, researchers immediately know that there is much room to improve the performance of their new algorithm.fig. the input and five settings of mviaeval. users need to a upload the r code of their new algorithm, b select the test datasets among benchmark microarray datasets, c select the compared algorithms among existing algorithms, d select the performance indices from three existing ones, the comprehensive performance scores from two possible choices, and the number of simulation runs fig. the output of mviaeval. for demonstration purpose, we upload the r code of a sample algorithm as the user’s new algorithm and select two benchmark datasets, existing algorithms, three performance indices, the overall ranking score as the comprehensive performance score, and simulation runs. a the webpage of the comprehensive performance comparison results shows that the overall performance of the user’s algorithm ranks six among all the compared algorithms. b by clicking “details” in the row of blci for the benchmark dataset gds users can see the performance comparison results using only blci score for the benchmark dataset gds it can be seen that the user’s algorithm ranks five among the compared algorithms using only blci score for the benchmark dataset gds the details of blci score for each algorithm can also be found performance the performance comparison results of the user’s algorithm and various existing algorithms using different types of datasets, different performance indices, and different overall performance scores or overall normalized score ) are shown. more details could be seen at http://cosbi.ee.ncku.edu.tw/mviaeval/a_case_study  missing value imputation is an inevitable pre-processing step of microarray data analyses. this is why the computational imputation of the missing values in microarray data has become a hot research topic. the newest algorithm is published in year and we believe that many new algorithms will be developed in the near future. using mviaeval, bioinformaticians can easily get a comprehensive and objective performance comparison results of their new algorithm. therefore, bioinformaticians now can focus on developing new algorithms instead of putting a lot of efforts for conducting a comprehensive and objective performance evaluation of their new algorithm. in conclusion, mviaeval will definitely be a very useful tool for developing missing value imputation algorithms. abbreviations mviaevalmissing value imputation algorithm evaluator nrmsenormalized root mean square error cppcluster pair proportion blcibiomarker list concordance index orsoverall ranking score onsoverall normalized score 
the comprehensive performance comparison results are then generated and shown as both figures and tables. to save researchers’ time and efforts, here we present an easy-to-use web tool named mviaeval which implements our performance comparison framework. missing value imputation is important for microarray data analyses because microarray data with missing values would significantly degrade the performance of the downstream analyses. mviaeval is a useful tool for researchers to easily conduct a comprehensive and objective performance evaluation of their newly developed missing value imputation algorithm for microarray data or any data which can be represented as a matrix form. web toolmissing value imputationmicroarray dataperformance indexperformance comparisonalgorithmhttp://dx.doi.org/ /501100004663ministry of science and technology, taiwanmost-103-2221-e- - -my2most-105-2221-e-006-203-my2wu wei-sheng national cheng kung univesity issue-copyright-statement© the author 2017 however, constructing our framework is not an easy task for the interested researchers. although many microarray missing value imputation algorithms have been developed, an objective and comprehensive performance comparison framework is still lacking. thus, mviaeval will greatly expedite the progress in the research of missing value imputation algorithms. mviaeval provides a user-friendly interface allowing users to upload the r code of their new algorithm and select the test datasets among benchmark microarray datasets, the compared algorithms among existing algorithms, the performance indices from three existing ones, the comprehensive performance scores from two possible choices, and the number of simulation runs. also the performance of a new algorithm can be evaluated by our performance comparison framework. to solve this problem, we previously proposed a framework which can perform a comprehensive performance comparison of different existing algorithms.
 missing value imputation is important for microarray data analyses because microarray data with missing values would significantly degrade the performance of the downstream analyses. although many microarray missing value imputation algorithms have been developed, an objective and comprehensive performance comparison framework is still lacking. to solve this problem, we previously proposed a framework which can perform a comprehensive performance comparison of different existing algorithms. also the performance of a new algorithm can be evaluated by our performance comparison framework. however, constructing our framework is not an easy task for the interested researchers. to save researchers’ time and efforts, here we present an easy-to-use web tool named mviaeval which implements our performance comparison framework.  mviaeval provides a user-friendly interface allowing users to upload the r code of their new algorithm and select the test datasets among benchmark microarray datasets, the compared algorithms among existing algorithms, the performance indices from three existing ones, the comprehensive performance scores from two possible choices, and the number of simulation runs. the comprehensive performance comparison results are then generated and shown as both figures and tables.  mviaeval is a useful tool for researchers to easily conduct a comprehensive and objective performance evaluation of their newly developed missing value imputation algorithm for microarray data or any data which can be represented as a matrix form. thus, mviaeval will greatly expedite the progress in the research of missing value imputation algorithms. keywords web toolmissing value imputationmicroarray dataperformance indexperformance comparisonalgorithmhttp://dx.doi.org/ /501100004663ministry of science and technology, taiwanmost-103-2221-e- - -my2most-105-2221-e-006-203-my2wu wei-sheng national cheng kung univesity issue-copyright-statement© the author 2017 