From: eesnyder@boulder.colorado.edu (Eric E. Snyder)
Newsgroups: bionet.biology.computational
Subject: Benchmarks for Gene Identification Programs
Date: 9 Sep 92 16:37:48 GMT
Distribution: bionet
Organization: University of Colorado, Boulder
Originator: biocomp@lerkim.umiacs.umd.edu

A couple of weeks ago, I posted a 'Wish-List' for gene identification 
programs.  This wish-list contained a number of statistics I would like
to see for coding sequence identification programs like GeneID and GRAIL. 
The purpose of having data of this type is two-fold: it would aid those
involved in the development of such programs (such as myself) to gauge
the performance of their programs as well as assist researcher seeking to
find the program best suited to their particular problem.  

I have received a number of very informative responses to the problem
I posed.  In addition to the need for a standardized battery of performance
statistics, chosing the appropriate set of test sequences is equally 
important.  All programs I have seen have been parameterized to perform
best on a specific type of sequence.  GRAIL, for example, is tailored to
identifying exons in human genes and performs less well on non-human
sequences.   In contrast, GeneID is geared to parsing genes from
vertebrates in general, at the expense of reduced performance on human
sequences.  While these differences in intent are clearly documented,
they may not be clear to the casual user.  Furthermore, one would like 
to have a quantitative measure of performance on a number of different
input data types, regardless of the original objective of the program. 

Steen Knudsen and Roderic Guigo have supplied me with the data from 
their comparisons of GeneID and GRAIL.  These test sets are independent
of those used to train the programs and differ enough to get a feel
for how these programs perform on different types of data.  The results
obtained on these datasets also fulfill most of the requirements of 
my original 'wish-list'.   As suggested by Steen Kundsen, I hope these
benchmark datasets and statistics can serve as a model for others 
working on this problem and also provide 'consumers' with useful
information about the utility of these programs in their research.  

Finally, I would encourage others working in this field to do use these
datasets to test their own programs and make suggestions for better
test sets.   If researchers will send me their data,  I will maintain a 
list of the results and standard test sets and periodically post 
this information to the net.  

Thanks again to Steen Knudsen and Roderic Guigo for sharing their data
and for providing a starting point from which we can begin to structure 
the research in this field.  




From: rgs@temin.Lanl.GOV (Roderic Guigo

DATA.
The sets of genes described in PNAS(GRAIL) and JMB(GeneId) as tests sets
were considered. 
>From JMB, the independent data set of 28 genes was used (GI-set), 
instead of the larger one of 169 genes, since this 
last set was partially used during the developement of GeneId. 
>From PNAS, the set of 19 genes was used . 
One gene (HUMTPA) was removed from the GR-set, because it excedes the maximum
allowed length in the current version of GeneID
and one gene (HAMRPS14A) was removed from GI-set  
because I was unable to retrieve the sequence from GenBank, rel. 72. 

In summary, two sets of genes were considered:
GR-set: 18 human genes
GI-set: 27 vertebrate genes

for which the **entire** sequence in the GenBank rel. 72 entry was obtained.


METHOD

1. Both sets were submitted to both GRAIL and GeneId e-mail servers.

	[
	objection 1: GRAIL was specifically  developed to analyze human genes, 
	while GeneId was intended more generally to vertebrate genes. The
	GI-set contains some non-human genes.

	objection 2: GeneId was primarily designed to analyze transription units
	-that is DNA sequences corresponding  exactly to the transcribed mRNA.  
	The entire sequence in a GenBank entry it is not usually restricted to
	the transcription unit.
	]

2. Output from GRAIL and GeneId was parsed to obtain the GRAIL and GeneId
predictions for each gene. 
The top model from the GeneId predictions was considered the GeneId prediction
(GI-prediction).
The set of all 'forward' exons (excellent+good+marginal) in the GRAIL 
predictions were considered the GRAIL prediction (GR-prediction).

In a sense, GI and GR predictions are not exactly comparable. GI-predictions
are assembled genes -linear combinations of exons satisfying a number of
rules-, while GR predictions are linear arrangements of non-overlapping
exons, which not necessarily and not usually can be assembled in
continuos genes. However, in another sense they are comparable, since both 
assign to each possition on the input sequence a binary value: 
coding/non-coding. And this is the way in which they are evaluated in their
respective papers.

	[
	objection 1: Only excellent (or excellent+good) exons in the GRAIL 
	predictions could be considered to obtain the GR-predictions.

	objection 2: Predictions in the complementary strand could be taken into 	
	account. Since the GR-set and the GI-set 
	do not contain genes in the complementary strand this means 1) 
	considering all GRAIL 'reverse' exons as false positives. 
	2) running GeneId on the complementary strand and consider the putative 
	whole GeneId prediction as false positive.
	]

3.  For each gene and prediction,  I computed DNA_length, true_CDS, 
predicted_CDS and true_predicted_CDS (true positive, tp). 
>From this data I computed the proportion of true_CDS being correctly 
predicted (s1), the proportion of predicted_CDS being actually true (s2) 
and the corresponding correlation coefficient (cc).

In addition,  I computed the **average** s1, s2 and cc for each separate data 
set, GR-set and GI-set, and for the combined data set GR+GI-set, as in the
JMB paper.
I also computed s1, s2 and cc for the **total** number of positions analyzed.
Again for each separate data set, GI and GR, and for the combined set
GI+GR, as in the PNAS paper.

RESULTS

(Note that for both GeneId and GRAIL the results obtaine for their own data
sets (GI-set and GR-set) differ from those published in their respective paper.
For GeneId, results are slightly worse,  because 1) a different version of
GeneId is currently being executed and 2)  Now the entire GenBank entry,
and not only the transcription unit, is considered.
For GRAIL, the results are slightly better, because 1) a different version of
GRAIL is currently being executed and ?.


EVALUATION OF GENEID AND GRAIL (entire GenBank entry) R.guigo, lanl, aug-92

A: GRAIL-SET (locus HUMTPA removed)
A.1. GRAIL                                       
                        DNA   CDS  pred   TP      s1    s2    cc
HUMALPHA               4556  1599  1010  1000    0.63  0.99  0.71
HUMAPRT                3016   543   566   366    0.67  0.65  0.58
HUMFOS                 6210  1143   538   412    0.36  0.77  0.46
HUMMETIA               2941   185   133    30    0.16  0.23  0.15
HUMNMYCA               8762  1630  1307  1290    0.79  0.99  0.86
HUMP45C17              8549  1526   898   869    0.57  0.97  0.71
HUMPAIA               17509  1208  1151   840    0.70  0.73  0.69
HUMPLPSPC              3409   593   323   287    0.48  0.89  0.61
HUMPNMTA               4174   848   804   751    0.89  0.93  0.89
HUMPOMC                8658   803   750   704    0.88  0.94  0.90
HUMPRCA               11725  1386  1065   918    0.66  0.86  0.73
HUMPRPH1               4946   501   363   261    0.52  0.72  0.58
HUMRASH                6453   570   605   503    0.88  0.83  0.84
HUMSAA                 3460   369    82    82    0.22  1.00  0.45
HUMTBB5                8874  1335  1415  1225    0.92  0.87  0.87
HUMTCRAC               5089   426   253   186    0.44  0.74  0.54
HUMTHB                20801  1869  1152  1067    0.57  0.93  0.71
HUMTKRA               13500   705   577   436    0.62  0.76  0.67

average                                          0.61  0.82  0.66

TOTAL                142632 17239 12992 11227    0.65  0.86  0.72


A.2. GENEID 

                        DNA   CDS  pred   TP      s1    s2    cc
HUMALPHA               4556  1599   996   825    0.52  0.83  0.53
HUMAPRT                3016   543   399   298    0.55  0.75  0.58
HUMFOS                 6210  1143  1194  1031    0.90  0.86  0.86
HUMMETIA               2941   185   168   157    0.85  0.93  0.88
HUMNMYCA               8762  1630  1530  1370    0.84  0.90  0.84
HUMP45C17              8549  1526  1551  1387    0.91  0.89  0.88
HUMPAIA               17509  1208   213   206    0.17  0.97  0.39
HUMPLPSPC              3409   593   762   282    0.48  0.37  0.28
HUMPNMTA               4174   848   555   554    0.65  1.00  0.77
HUMPOMC                8658   803   930   803    1.00  0.86  0.92
HUMPRCA               11725  1386  1089   863    0.62  0.79  0.67
HUMPRPH1               4946   501   516   337    0.67  0.65  0.62
HUMRASH                6453   570  1545   554    0.97  0.36  0.53
HUMSAA                 3460   369   243   230    0.62  0.95  0.75
HUMTBB5                8874  1335  1158  1058    0.79  0.91  0.83
HUMTCRAC               5089   426   273   273    0.64  1.00  0.79
HUMTHB                20801  1869  1230   958    0.51  0.78  0.60
HUMTKRA               13500   705  1155   279    0.40  0.24  0.26
                                                 
average                                          0.67  0.78  0.67

TOTAL                142632 17239 15507 11465    0.67  0.74  0.66



B: GENEID-SET (locus HAMRPS14A removed)
B.1. GRAIL 

                        DNA   CDS  pred   TP      s1    s2    cc
BOVPNMTB               2649   852   637   637    0.75  1.00  0.82
CHKCMYCA               3517  1251  1261  1173    0.94  0.93  0.90
HUMCYP2DG              5503  1503  1173  1029    0.68  0.88  0.71
HUMEMBPA               3608   669   242   221    0.33  0.91  0.50
HUMGAPDHG              5378  1008   665   647    0.64  0.97  0.76
HUMHBBAAZ              2149   444   313   297    0.67  0.95  0.76
HUMHNRNPA              5368   963   474   298    0.31  0.63  0.36
HUMIBP3               10884   876   687   550    0.63  0.80  0.69
HUMLD78A               3176   279    61    61    0.22  1.00  0.45
HUMLD78B               3112   282    41    41    0.15  1.00  0.37
HUMPDHBET              8872  1080   194   163    0.15  0.84  0.33
HUMPP14B               8076   543   477   351    0.65  0.74  0.67
HUMREGB                4251   501   252   216    0.43  0.86  0.58
HUMTRPY1B              2609   828   536   437    0.53  0.82  0.54
MNKHGBGGAG             1705   444   182   182    0.41  1.00  0.58
MUSCD14A               1873  1101   332   332    0.30  1.00  0.39
MUSCYTCB1              4828  1515   839   765    0.50  0.91  0.59
MUSCYTCC               5661  1518   660   620    0.41  0.94  0.55
MUSMB1AA               5629   663   395   243    0.37  0.62  0.42
MUSREN1D              11231  1209   466   403    0.33  0.86  0.51
RAB15LOX               8443  1992  1251  1051    0.53  0.84  0.59
RABBCAS               11174   687   122   121    0.18  0.99  0.41
RATCOXIVA              6980   510   343   340    0.67  0.99  0.80
RATCYP2A1             18820  1479   832   638    0.43  0.77  0.55
RATCYP2A3A            14315  1485  1194  1018    0.69  0.85  0.74
RATODCAA               7780  1386   346   339    0.24  0.98  0.45
RATPPP                 4542   297    11    11    0.04  1.00  0.19

                                                 0.45  0.89  0.56

TOTAL                172133 25365 13986 12184    0.48  0.87  0.61


B.2.  GENEID 

                        DNA   CDS  pred   TP      s1    s2    cc
BOVPNMTB               2649   852   483   483    0.57  1.00  0.69
CHKCMYCA               3517  1251   966   966    0.77  1.00  0.83
HUMCYP2DG              5503  1503  1347  1062    0.71  0.79  0.66
HUMEMBPA               3608   669  1254   636    0.95  0.51  0.60
HUMGAPDHG              5378  1008  1029   976    0.97  0.95  0.95
HUMHBBAAZ              2149   444   444   444    1.00  1.00  1.00
HUMHNRNPA              5368   963   867   697    0.72  0.80  0.71
HUMIBP3               10884   876  1023   812    0.93  0.79  0.84
HUMLD78A               3176   279   348   237    0.85  0.68  0.74
HUMLD78B               3112   282   225   213    0.76  0.95  0.83
HUMPDHBET              8872  1080   789   750    0.69  0.95  0.79
HUMPP14B               8076   543   828   354    0.65  0.43  0.49
HUMREGB                4251   501   501   501    1.00  1.00  1.00
HUMTRPY1B              2609   828   909   703    0.85  0.77  0.72
MNKHGBGGAG             1705   444   210   150    0.34  0.71  0.39
MUSCD14A               1873  1101  1101  1101    1.00  1.00  1.00
MUSCYTCB1              4828  1515  1353  1248    0.82  0.92  0.82
MUSCYTCC               5661  1518  1140  1131    0.75  0.99  0.82
MUSMB1AA               5629   663   312    69    0.10  0.22  0.08
MUSREN1D              11231  1209   561   454    0.38  0.81  0.52
RAB15LOX               8443  1992   531   123    0.06  0.23 -0.00
RABBCAS               11174   687   642   630    0.92  0.98  0.95
RATCOXIVA              6980   510   282   269    0.53  0.95  0.69
RATCYP2A1             18820  1479  1251  1042    0.70  0.83  0.75
RATCYP2A3A            14315  1485  1161   343    0.23  0.30  0.19
RATODCAA               7780  1386  1170   962    0.69  0.82  0.71
RATPPP                 4542   297   246   106    0.36  0.43  0.35

                                                 0.68  0.77  0.67

TOTAL                172133 25365 20973 16462    0.65  0.78  0.67




C. SUMMARY (GLOBAL set -GeneId+GRAIL-)


 
        |  s1    |  s2    |  cc    |   average
--------|--------|--------|--------|--
        |        |        |        |
GRAIL   |  0.51  |  0.86  |  0.60  |
        |        |        |        |
--------|--------|--------|--------|--
        |        |        |        |
GeneId  |  0.68  |  0.77  |  0.67  |
        |        |        |        |
--------|--------|--------|--------|--



        |  s1    |  s2    |  cc    |   total
--------|--------|--------|--------|--
        |        |        |        |            DNA   CDS  pred   TP
GRAIL   |  0.55  |  0.87  |  0.66  |          314765 42604 26978 23411
        |        |        |        |
--------|--------|--------|--------|--
        |        |        |        |
GeneId  |  0.66  |  0.77  |  0.67  |          314765 42604 36480 27927
        |        |        |        |
--------|--------|--------|--------|--



		
From: steen@darwin.bu.edu (Steen Knudsen


                                        GRAIL(1)        GENEID

Exactly predicted exons (GeneID test set)
    Sensitivity ( tp/(tp+fn) )          N/A             45%
    Specificity ( tp/(tp+fp) )          N/A             50%

Exons predicted by overlap
    GRAIL test set (2)
        Sensitivity                     81%             62%
        Specificity                     77%             73%
    GeneID test set (3)
        Sensitivity                     66%             70%
        Specificity                     77%             77%
    
NOTES:
(1) GRAIL server version as of August 25 1992 (not PNAS paper)
(2) Test set from PNAS paper minus one gene longer than current GeneID
    limit.
(3) Test set (28) from GeneID paper. Contains vertebrates for which
    GRAIL was not explicitly trained.




---------------------------------------------------------------------------
TTGATTGCTAAACACTGGGCGGCGAATCAGGGTTGGGATCTGAACAAAGACGGTCAGATTCAGTTCGTACTGCTG
Eric E. Snyder                            
Department of MCD Biology              ...making feet for childrens' shoes.
University of Colorado, Boulder   
Boulder, Colorado 80309-0347
LeuIleAlaLysHisTrpAlaAlaAsnGlnGlyTrpAspLeuAsnLysAspGlyGlnIleGlnPheValLeuLeu
---------------------------------------------------------------------------
Newsgroups: bionet.journals.contents,bit.listserv.bionews,sci.bio