BACKGROUND
the biological research literature is a major repository of knowledge.
unfortunately, the amount of literature has gotten so large that it is often hard to find the information of interest on a particular topic.
there has been an increasing amount of work on text mining for this literature, but currently, there is no way to compare the systems developed because they are run on different data sets to perform different tasks  <cit> .
challenge evaluations have been successful in making such comparisons.
examples include the ongoing casp evaluations  for protein structure prediction  <cit> , the series of message understanding conferences  for information extraction on newswire text  <cit> , and the ongoing text retrieval conferences  for information retrieval  <cit> .
also, in  <dig>  we ran the first challenge evaluation of text mining for biology; this was an evaluation for classifying papers and genes based on whether they contained experimental evidence for gene products  <cit> .
as mentioned in  <cit> , the idea behind these series of open evaluations has been to attract teams to work on a problem by providing them with real  training and test data, as well as objective evaluation metrics.
these data sets are often hard to obtain, and the open evaluation makes it much easier for groups to build systems and compare performance on a common problem.
if many teams are involved, the results are a measure of the state-of-the-art for that task.
in addition, when the teams share information about their approaches and the evaluations are repeated over time, then the research community can demonstrate measurable forward progress in a field.
to further the field of biological text mining, the biocreative evaluation was run in  <dig>  with a workshop in march  <dig> to discuss the results  <cit> .
the evaluation consisted of two tasks: task  <dig> focused on extraction of gene names  and normalization of genes  from pubmed abstracts.
task  <dig> was a more advanced task focused on functional annotation, using full text information to classify a protein as to its molecular function, biological process and/or location within a cell  <cit> .
this paper reports on task 1a, entity mention extraction.
this extraction is a basic text mining operation.
its output is the input text, annotated with the mentions of interest; this can be used as a building block for other tasks, such as task 1b and task  <dig>
the gene mention task presents a number of difficulties.
one difficulty is that gene  mentions are often english common nouns  and they are often descriptions.
in fact, many entities are named with ordinary words; examples from drosophila  gene names are blistery, inflated, period, punt, vein, dorsal, kayak, canoe and midget.
in addition, new entities are constantly being discovered and/or renamed with these common nouns.
many new names originate as descriptions and can be quite complex, e.g., hereditary non-polyposis colorectal cancer  tumor suppressor genes.
task and data
the data and evaluation software for task 1a were provided by w. john wilbur and lorraine tanabe at the national center for biotechnology information .
every mention of interest is marked, so this task corresponds to the "named entity" task used in the natural language processing community.
the data consists of sentences from medline  <cit>  abstracts that have been manually annotated for mentions of names of genes and related entities.
half of the sentences were chosen from abstracts likely to contain such names.
the other half were chosen from abstracts likely not to contain such names.
see  <cit>   for further detail on the construction of the task 1a data.
the approximate sizes of the various data sets are given in table  <dig>
participants were provided with  <dig> training sentences and  <dig> development test sentences.
the  test set  consisted of  <dig> sentences.
for the evaluation, its sentences were renumbered to give no indication of what medline abstracts they came from .
the data is marked for mentions of "names" related to genes, including binding sites, motifs, domains, proteins, promoters, etc.
the data comes with a particular tokenization , and this tokenization determines the boundaries of what is marked.
a token is either entirely markable or not.
a token cannot be split between a marked part and an unmarked part.
for example, if "egf-induced" is a token and one wants to mark the "egf" part of that token, then one also needs to mark the "induced" part.
for testing, the systems take as input the tokenized unannotated sentences; the output is the list of gene names for each sentence, with the start and stop token offsets.
for evaluation, the system output is then compared to the "gold standard" hand-annotated answer key.
there is no detailed, multi-page explicit set of guidelines describing what is markable.
instead, there is a description provided with the data that gives a page or two listing of the types of entities that are and are not markable.
examples of markables are mutants  and words like codon or antibody when combined with a gene name.
examples of non-markables include generic terms  and mutations .
here are  <dig> excerpts from the training corpus :
the lmw fgf- <dig> up-regulated the pkc epsilon levels by  <dig>  fold; by contrast the hmw isoform down-regulated the level...
...a protein related to snf1 protein kinase.
the underlines indicate the markable entities.
the italic boldface indicates what alternative mentions can substitute for a markable.
note that for "snf1" and "protein kinase", an allowed alternative is "snf <dig> protein kinase", which includes both of them.
the answer file for the sentences contains the following mentions: "lmw fgf-2", "pkc epsilon", "hmw isoform", "snf1" and "protein kinase".
stored in another file are the alternative mentions that can be tagged and still count as being correct.
for the answers mentioned above, here are the allowed alternative mentions: "fgf-2", "pkc", "hmw", and "snf <dig> protein kinase".
when scoring, an exact match to an answer or an allowed alternative is needed to get credit for finding an answer.
so, for example, if for the answer lmw fgf- <dig>  a system instead returns "the lmw fgf-2", that system would get both a false negative  and also a false positive .
RESULTS
<dig> teams entered submissions for this evaluation.
submissions were classified as either "open" or "closed".
closed: the system producing the submission is only trained on the task 1a "train" and " test"  data sets, with no additional lexical resources.
open: the system producing the submission can make use of external word lists, other data sets, etc.
most teams provided an "open/closed" classification for their submissions.
if they did not, we made a classification based on a short system description that the teams provided.
when we were not sure, we assumed "open".
teams were allowed to send up to  <dig> submissions each, as long as they included a "closed" submission.
teams only sending "open" submissions were allowed to send up to  <dig> submissions.
we received a total of  <dig> "closed" submissions  and  <dig> "open" submissions .
the submissions were measured by their balanced f-score, recall and precision.
• balanced f-score is the harmonic mean between recall and precision.
balanced f-score = 2*recall*precision/
• recall is the fraction or percentage of the answers in the answer key that were found by a submission.
• precision is the fraction or percentage of the answers returned by a submission that are scored as correct.
scores achieved by the submissions
many of the high performing submissions achieved scores quite close together.
for example, with balanced f-score, the first and second highest teams were only  <dig> % apart, and the second and third highest teams were even closer, at  <dig> % apart.
this is close enough to possibly be affected by the disagreements in annotation that arise with just about any task on finding entity mentions.
an example is that with this particular task, a partial review of the test set changed  <dig> %  of the answers.
these differences are also close enough so that they are often not statistically significant.
at a normal threshold of 5%  for statistical significance, the difference between the first and third highest teams was borderline statistically significant, while the difference between the first and second highest teams  was not statistically significant.
computationally-intensive randomization tests  <cit>  ) were used to test statistical significance.
like most significance tests,  <dig> variants exist
<dig>  a 1-sided version: under the null hypothesis, how likely will the higher ranked submission be better than the lower ranked submission by at least the observed difference?
<dig>  a 2-sided version: under the null hypothesis, how likely will the difference between the two submissions  be at least the observed difference?
the 2-sided version is more demanding and will, for the same score difference, produce about twice the probability of the 1-sided version.
complicating the shuffling done in the tests for these results was the existence of alternative answers, so that the correspondence between correct responses and answers in the key was not one-to-one.
another complication was that due to the format for submissions, no submission could give responses that overlapped with each other.
comparing the first and third highest teams, the 1-sided test produced a significance level of  <dig> %, while the 2-sided produced  <dig> % .
comparing the first and second highest teams, the 1-sided test was 11% and the 2-sided was 22% .
comparing the second and third highest teams, the 1-sided test was 34% and the 2-sided was 68% .
• the high, 1st, 2nd and even 3rd quartile scores are relatively close to each other compared to the low scores
• with f-score, the top  <dig> teams had f-scores within 1% of each other
• with recall, the top  <dig> teams were separated by  <dig> % in recall
• with precision, the top  <dig> teams were separated by  <dig> % in precision
the difference in recall was statistically significant:  <dig> out of  <dig> trials passed the threshold for either the 1-sided or 2-sided test.
the difference in precision was not statistically significant:  <dig> % for the 1-sided test and 14% for the 2-sided .
the f-score differences were discussed above.
generally, the open submissions did better than the closed submissions.
an exception is that for the highest recall score, the top closed score was actually better than the top open score.
the compression of the high scores also occurred when comparing the open and closed submissions.
• for the higher scores , there was little difference  between the open and closed submission scores
• for the lower scores , the open submissions scores were measurably better than the closed scores
some observations
like many name identification tasks, task 1a has its own unique features.
most teams made use of the training data in their system development.
however, in reading the task 1a participants' system descriptions  <cit> , team k did not.
also, as far as we can tell, neither did team m .
this is probably a reason why, relative to the other teams, these two teams did not get very good results: k's submission had a 61% balanced f-score, while m's submission had 55% .
one indication of these unique features comes from tamames  <cit> , whose system had not considered entities like domains, regions and mutants as "gene names" that should be marked, where as task 1a did include such entities.
a common comment from several task 1a participants  and kinoshita  <cit> ) was that one of the more difficult aspects of task 1a was determining the starting and ending boundaries of the gene-or-protein names.
the requirement for an exact match to the answer key  increased the difficulty.
as has been mentioned, many of the open and closed submissions achieved fairly close results.
one possible reason for this is that, to the extent that this task is unique, outside sources will not help performance that much.
when comparing the results between different teams, another possible reason is that for the most part, we relied on the teams themselves to classify their submissions as being "open or closed".
in viewing the task 1a system descriptions  <cit> , one can see that the different teams varied in what resources they thought were allowed in a closed submission.
as an example, when using a sub-system that generates part-of-speech  tags, some  teams use such a pos sub-system for a "closed" submission even when the sub-system itself was trained on another annotated corpus, an indirect reliance on an outside corpus.
some teams treated this indirect reliance as permissible for a closed submission  and zhou  <cit> ), some teams did not.
summary of system descriptions
for task 1a, the teams tended to use one of the three following approaches at the top level of their system :
<dig>  some type of markov modelling.
<dig>  support vector machine .
typically, the input information on the word being classified would come from a small window of the words near that word of interest.
<dig>  rules.
as far as we could tell, the rules were usually manually generated.
many of the systems had pre- and/or post-processing stages in addition to the main approach taken.
one system combined several other systems via a voting scheme  <cit> .
the teams used a variety of features in their systems.
many teams used entire sub-systems to find the values of certain features.
an example is using a part-of-speech  tagger to find a word's part-of-speech.
these sub-systems often used an approach that differed from the overall system's approach.
the four teams with 80% or higher f-scores had post-processing stages in addition to the main approach taken, and made use of many different features.
all four of these teams performed some type of markov modelling at the system's top level  <cit> .
however, the teams used different techniques on their markov models: maximum entropy, hidden markov models  and conditional random fields.
in addition, one team  <cit> , also had an svm system at the top level: decisions were made by having two hmms and an svm system vote.
also, note that when comparing different systems, the choice of features used is often at least as important as the approach/algorithm used.
yeh  <cit>  gives an example of this.
task 1a as a building block
one reason for evaluating on task 1a is that a task 1a system can serve as a building block for other tasks, like task 1b or task  <dig> of the biocreative evaluation.
the task 1b evaluation focused on finding the list of the distinct genes  mentioned in a medline abstract, where the list contained the normalized, unique identifiers for those genes.
task  <dig> focused on functional annotation , and on returning text passages as evidence to support these classifications.
to what extent was it viable to use task 1a systems as a building block for more advanced capabilities?
it turns out that three of the teams taking part in task 1a also took part in task 1b.
in addition, one of the three teams also took part in a portion of task  <dig>  so an interesting question is whether these three teams found their task 1a systems to be useful when working on task 1b or  <dig>
one team  with a high precision  task 1a system used the mentions found by their 1a system as the input for their 1b system  <cit> : their 1b system then tried to find the normalized version of the mentions found by their task 1a system.
the story was more complicated for two other teams with both high precision  and high recall  task 1a systems.
one team was from pennsylvania .
the other team was from edinburgh and stanford .
both these teams looked at some version of finding mentions with their task 1a system and then compared the found mentions against the synonym lists for the genes of interest for task 1b.
both teams found that this approach could easily produce a low precision for 1b, due to ambiguity .
the pennsylvania team also found that for genes from two  of the three organisms of interest in task 1b , the task 1a tagger was not that accurate.
a possible explanation given was that the task 1a training data did not have enough examples from these two organisms.
for task 1b, the pennsylvania team in the end did not use their task 1a tagger.
the edinburgh/stanford team found that using the original task 1a training set and lots of features tended to lower their recall of the 1b genes.
they raised the recall by retraining their 1a system using the noisy task 1b training data and a reduced set of the possible features.
the edinburgh/stanford team also took part in task  <dig> .
in this task, a system was given an article, a protein mentioned in that article, and a classification of that protein that a person made based on that article.
the system's job was to find a passage of text in that article that supported the classification made for that protein.
the description for the team's task  <dig>  system  <cit>  made no mention of using their task 1a system or trying it on some part of task  <dig> .
discussion
one unique aspect of the data: enforcing a particular tokenization
as mentioned before, every entity mention task such as task 1a will have some features that are more or less unique to it.
for task 1a, one such feature is that the data comes with a particular tokenization .
furthermore, this tokenization affects what counts as a mention, because either all of a token is tagged as part of a mention, or none of that token is tagged.
this can cause problems when one just wants to tag part of a token as part of a mention.
an example is the phrase "a protein kinase a-mediated pathway", shown in figure  <dig>  with the red vertical bars indicating the word token boundaries.
here the token "a-mediated" is not useful, since the mention that one would really like to tag is "protein kinase a".
this tokenization is important because it affects what counts as a mention; below are some rules :
<dig>  if "x" is a token which is a gene name, then "x" is usually marked.
an example is "cbf1" in the phrase "... of cbf <dig> in yeast ..." .
<dig>  if a token is of the form "x-" or "x-y", where "x" is a gene name and "y" is an adjective or verb, then the token is usually not marked.
an example is "egf-induced" in the phrase "... block egf-induced mitogenesis and ... " .
<dig>  an exception to  <dig> : when the y in "x-y" is "like", then "x-y" is usually marked .
also, if the form is "x-y z", where "x-y" is as in , and "z" is a token like "domain", then "x-y" is usually marked as part of the mention "x-y z".
an example is "sh2-binding domain".
disagreements in the data
in tasks like task 1a, small disagreements usually exist on what to annotate and what not to annotate.
an example in task 1a is phrases of the form "x pathway", where x is a phrase that is marked as part of a gene mention.
an initial review of the test set found the following annotation variations " both allowed as alternative answers):
•  <dig> cases where "x pathway was not an allowed alternative to "x".
an example was x = "mek-erk1/2" in the phrase
"... the mek-erk1/ <dig> pathway by ..." .
•  <dig> cases where "x" and "x pathway" were both allowed alternatives.
an example was x = "ras/raf/mapk" in the phrase
"... the ras/raf/mapk pathway."
.
similarly, the training set had
•  <dig> cases where "x pathway" was not an allowed alternative to "x".
•  <dig> cases where "x pathway" and "x" were allowed alternative answers.
such variation in annotation makes it more difficult to learn or to formulate a rule for how to handle these kinds of constructions.
lessons learned for future evaluations
if and when a future task 1a evaluation is run, we list the following issues to consider:
<dig>  tokenization is non-trivial for biological terms.
perhaps one should not enforce a fixed tokenization of the data.
this non-enforcement will be expensive because it requires changing both how the data is annotated and how the system results are compared against the gold standard.
<dig>  on a related matter, because of the difficulties in exactly determining a mention's boundaries, there is interest in also counting inexact matches to answers as being correct.
this must be defined carefully.
for example, if missing either the first or last token still counts as correct, then just returning "epsilon" would count as finding "pkc epsilon".
<dig>  for open versus closed submissions, we should either remove the distinction, or be more explicit as to what is allowed for a closed submission.
<dig>  a suggestion was made to pad the test set with extra material that would not be scored, which would make it harder to "cheat" by manually examining the test set.
if this were done, one would need to announce this ahead of time.
one reason is that some automated approaches need more processing time than others.
another reason is that some automated approaches, such as transductive support vector machines  <cit> , make use of statistics derived from the entire un-annotated test set.
<dig>  at least one team  <cit>  automatically searched for the pubmed/medline abstract associated with each test set sentence.
they used the abstract as a surrounding context, and it seemed to be helpful.
in many "real uses" of a task 1a system, a system will probably have such surrounding text.
so it may make sense to just give these abstracts to every participant in the future.
<dig>  there is also a question of what is a permissible resource to use:
• one example is that with pubmed/medline, a system could also look-up mesh terms, etc.
associated with the medline abstract for each sentence.
if a tagging system is applied before an abstract is assigned mesh labels , then such information will not be available in real usage, and such information should not be permitted.
• given a possible entity "x", at least one team  <cit>  did web searches for contexts like "x gene", which support "x" being a possible entity.
this seemed to be of limited help.
should this be permitted in the future?
this probably depends on the anticipated "real" uses for such a feature.
when tagging older material , the web will have relevant material.
when tagging new text that describes new gene, the web will probably not have much, if any material.
CONCLUSIONS
for the biocreative task 1a of gene mention finding, a number of teams achieved an 80–83% balanced f-score.
unless a system was not performing well, using external resources  did not seem to help that much.
these 80%+ results are similar to results for some other similar biological mention finding tasks, and are somewhat behind the 90%+ balanced f-scores achieved on english newswire named entity tasks  <cit> .
based on an observation offered by kevin cohen , one hypothesis for the discrepancy is that gene names tend to be longer  than comparable newswire names.
to investigate this, we compared the length distribution of gene names in the test set for task 1a; this distribution is shown in figure  <dig>  and is compared to the distribution for name length of organization names in a newswire task.
the newswire results are computed from the muc- <dig> data, which is available from the linguistic data consortium  <cit> .
the average length of the task 1a gene names was  <dig> , compared to  <dig>  for organization names in the muc- <dig> data.
given this distribution, we fitted a simple logistic regression model to both data sets.
we modeled the performance  for a name of n words as n, where e is the performance on a single-word name.
this allowed us to extrapolate back to a single-word error rate for both tasks, allowing us to factor out differences in name length.
for gene names, a 91% success rate on a single word gene name gave an overall task performance of 83%, the observed high score.
for the muc- <dig> organization names, a  <dig> % single word success rate yielded a 93% success rate overall, which was the highest recorded result for muc- <dig>  in using this simple model, we recognize that it is not mathematically valid to use f-measure in place of accuracy.
however, it does provide a crude approximation for how much of the task difficulty can be attributed to difference in name lengths among different tasks.
this comparison leaves a residual 4–5% discrepancy between performance on the tasks for the single-word case.
we hypothesize that this may be due to interannotator variability, leading to "noise" in the training and test data.
for the muc- <dig> task  <cit> , interannotator agreement was measured at 97%, which is almost certainly significantly higher than for the gene mention task, which has not yet been formally measured.
this variability affected at least one high performing task 1a system  <cit> .
the discussion section of this paper gives some sample disagreements in the task 1a data.
in terms of successful approaches, the teams that achieved an 80% or more balanced f-score tended to use some type of markov modelling at the top system level.
however, these teams also had post-processing stages in addition to the main approach taken, and the different teams made use of different features.
these stages and features can have just as much an effect on performance as the main approach taken.
one of the reasons to have task 1a is that it should be a useful building block to work on other tasks, like biocreative task 1b.
three teams tried using their task 1a system for task 1b.
their experiences were mixed, with two of the three teams finding that a task 1a system trained on the task 1a training data often did not work so well on task 1b.
one of these two teams improved things by retraining their 1a system using the noisy task 1b data.
a 2nd test set is available for task 1a, so it would be straightforward to run a task 1a evaluation in the future using this 2nd test set.
four questions to think about in any future evaluation are the following:
• what will it take to improve task 1a performance?
• how much will improving task 1a performance help with other tasks ?
• how can one make a task 1a system be a more useful building block for other tasks?
• why are outside resources not more useful in task 1a?
is it because task 1a is unique?

 the biological research literature is a major repository of knowledge. unfortunately, the amount of literature has gotten so large that it is often hard to find the information of interest on a particular topic. there has been an increasing amount of work on text mining for this literature, but currently, there is no way to compare the systems developed because they are run on different data sets to perform different tasks. challenge evaluations have been successful in making such comparisons. examples include the ongoing casp evaluations for protein structure prediction, the series of message understanding conferences for information extraction on newswire text, and the ongoing text retrieval conferences for information retrieval. also, in we ran the first challenge evaluation of text mining for biology; this was an evaluation for classifying papers and genes based on whether they contained experimental evidence for gene products. as mentioned in, the idea behind these series of open evaluations has been to attract teams to work on a problem by providing them with real training and test data, as well as objective evaluation metrics. these data sets are often hard to obtain, and the open evaluation makes it much easier for groups to build systems and compare performance on a common problem. if many teams are involved, the results are a measure of the state-of-the-art for that task. in addition, when the teams share information about their approaches and the evaluations are repeated over time, then the research community can demonstrate measurable forward progress in a field. to further the field of biological text mining, the biocreative evaluation was run in with a workshop in march to discuss the results. the evaluation consisted of two tasks: task focused on extraction of gene names and normalization of genes from pubmed abstracts. task was a more advanced task focused on functional annotation, using full text information to classify a protein as to its molecular function, biological process and/or location within a cell. this paper reports on task 1a, entity mention extraction. this extraction is a basic text mining operation. its output is the input text, annotated with the mentions of interest; this can be used as a building block for other tasks, such as task 1b and task the gene mention task presents a number of difficulties. one difficulty is that gene mentions are often english common nouns and they are often descriptions. in fact, many entities are named with ordinary words; examples from drosophila gene names are blistery, inflated, period, punt, vein, dorsal, kayak, canoe and midget. in addition, new entities are constantly being discovered and/or renamed with these common nouns. many new names originate as descriptions and can be quite complex, e.g., hereditary non-polyposis colorectal cancer tumor suppressor genes. task and data the data and evaluation software for task 1a were provided by w. john wilbur and lorraine tanabe at the national center for biotechnology information. every mention of interest is marked, so this task corresponds to the "named entity" task used in the natural language processing community. the data consists of sentences from medline abstracts that have been manually annotated for mentions of names of genes and related entities. half of the sentences were chosen from abstracts likely to contain such names. the other half were chosen from abstracts likely not to contain such names. see for further detail on the construction of the task 1a data. the approximate sizes of the various data sets are given in table participants were provided with training sentences and development test sentences. the test set consisted of sentences. for the evaluation, its sentences were renumbered to give no indication of what medline abstracts they came from. the data is marked for mentions of "names" related to genes, including binding sites, motifs, domains, proteins, promoters, etc. the data comes with a particular tokenization, and this tokenization determines the boundaries of what is marked. a token is either entirely markable or not. a token cannot be split between a marked part and an unmarked part. for example, if "egf-induced" is a token and one wants to mark the "egf" part of that token, then one also needs to mark the "induced" part. for testing, the systems take as input the tokenized unannotated sentences; the output is the list of gene names for each sentence, with the start and stop token offsets. for evaluation, the system output is then compared to the "gold standard" hand-annotated answer key. there is no detailed, multi-page explicit set of guidelines describing what is markable. instead, there is a description provided with the data that gives a page or two listing of the types of entities that are and are not markable. examples of markables are mutants and words like codon or antibody when combined with a gene name. examples of non-markables include generic terms and mutations. here are excerpts from the training corpus: the lmw fgf- up-regulated the pkc epsilon levels by fold; by contrast the hmw isoform down-regulated the level......a protein related to snf1 protein kinase. the underlines indicate the markable entities. the italic boldface indicates what alternative mentions can substitute for a markable. note that for "snf1" and "protein kinase", an allowed alternative is "snf protein kinase", which includes both of them. the answer file for the sentences contains the following mentions: "lmw fgf-2", "pkc epsilon", "hmw isoform", "snf1" and "protein kinase". stored in another file are the alternative mentions that can be tagged and still count as being correct. for the answers mentioned above, here are the allowed alternative mentions: "fgf-2", "pkc", "hmw", and "snf protein kinase". when scoring, an exact match to an answer or an allowed alternative is needed to get credit for finding an answer. so, for example, if for the answer lmw fgf- a system instead returns "the lmw fgf-2", that system would get both a false negative and also a false positive.  teams entered submissions for this evaluation. submissions were classified as either "open" or "closed". closed: the system producing the submission is only trained on the task 1a "train" and " test" data sets, with no additional lexical resources. open: the system producing the submission can make use of external word lists, other data sets, etc. most teams provided an "open/closed" classification for their submissions. if they did not, we made a classification based on a short system description that the teams provided. when we were not sure, we assumed "open". teams were allowed to send up to submissions each, as long as they included a "closed" submission. teams only sending "open" submissions were allowed to send up to submissions. we received a total of "closed" submissions and "open" submissions. the submissions were measured by their balanced f-score, recall and precision. • balanced f-score is the harmonic mean between recall and precision. balanced f-score = 2*recall*precision/ • recall is the fraction or percentage of the answers in the answer key that were found by a submission. • precision is the fraction or percentage of the answers returned by a submission that are scored as correct. scores achieved by the submissions many of the high performing submissions achieved scores quite close together. for example, with balanced f-score, the first and second highest teams were only % apart, and the second and third highest teams were even closer, at % apart. this is close enough to possibly be affected by the disagreements in annotation that arise with just about any task on finding entity mentions. an example is that with this particular task, a partial review of the test set changed % of the answers. these differences are also close enough so that they are often not statistically significant. at a normal threshold of 5% for statistical significance, the difference between the first and third highest teams was borderline statistically significant, while the difference between the first and second highest teams was not statistically significant. computationally-intensive randomization tests ) were used to test statistical significance. like most significance tests, variants exist a 1-sided version: under the null hypothesis, how likely will the higher ranked submission be better than the lower ranked submission by at least the observed difference? a 2-sided version: under the null hypothesis, how likely will the difference between the two submissions be at least the observed difference? the 2-sided version is more demanding and will, for the same score difference, produce about twice the probability of the 1-sided version. complicating the shuffling done in the tests for these results was the existence of alternative answers, so that the correspondence between correct responses and answers in the key was not one-to-one. another complication was that due to the format for submissions, no submission could give responses that overlapped with each other. comparing the first and third highest teams, the 1-sided test produced a significance level of %, while the 2-sided produced %. comparing the first and second highest teams, the 1-sided test was 11% and the 2-sided was 22%. comparing the second and third highest teams, the 1-sided test was 34% and the 2-sided was 68%. • the high, 1st, 2nd and even 3rd quartile scores are relatively close to each other compared to the low scores • with f-score, the top teams had f-scores within 1% of each other • with recall, the top teams were separated by % in recall • with precision, the top teams were separated by % in precision the difference in recall was statistically significant: out of trials passed the threshold for either the 1-sided or 2-sided test. the difference in precision was not statistically significant: % for the 1-sided test and 14% for the 2-sided. the f-score differences were discussed above. generally, the open submissions did better than the closed submissions. an exception is that for the highest recall score, the top closed score was actually better than the top open score. the compression of the high scores also occurred when comparing the open and closed submissions. • for the higher scores, there was little difference between the open and closed submission scores • for the lower scores, the open submissions scores were measurably better than the closed scores some observations like many name identification tasks, task 1a has its own unique features. most teams made use of the training data in their system development. however, in reading the task 1a participants' system descriptions, team k did not. also, as far as we can tell, neither did team m. this is probably a reason why, relative to the other teams, these two teams did not get very good results: k's submission had a 61% balanced f-score, while m's submission had 55%. one indication of these unique features comes from tamames, whose system had not considered entities like domains, regions and mutants as "gene names" that should be marked, where as task 1a did include such entities. a common comment from several task 1a participants and kinoshita ) was that one of the more difficult aspects of task 1a was determining the starting and ending boundaries of the gene-or-protein names. the requirement for an exact match to the answer key increased the difficulty. as has been mentioned, many of the open and closed submissions achieved fairly close results. one possible reason for this is that, to the extent that this task is unique, outside sources will not help performance that much. when comparing the results between different teams, another possible reason is that for the most part, we relied on the teams themselves to classify their submissions as being "open or closed". in viewing the task 1a system descriptions, one can see that the different teams varied in what resources they thought were allowed in a closed submission. as an example, when using a sub-system that generates part-of-speech tags, some teams use such a pos sub-system for a "closed" submission even when the sub-system itself was trained on another annotated corpus, an indirect reliance on an outside corpus. some teams treated this indirect reliance as permissible for a closed submission and zhou ), some teams did not. summary of system descriptions for task 1a, the teams tended to use one of the three following approaches at the top level of their system: some type of markov modelling. support vector machine. typically, the input information on the word being classified would come from a small window of the words near that word of interest. rules. as far as we could tell, the rules were usually manually generated. many of the systems had pre- and/or post-processing stages in addition to the main approach taken. one system combined several other systems via a voting scheme. the teams used a variety of features in their systems. many teams used entire sub-systems to find the values of certain features. an example is using a part-of-speech tagger to find a word's part-of-speech. these sub-systems often used an approach that differed from the overall system's approach. the four teams with 80% or higher f-scores had post-processing stages in addition to the main approach taken, and made use of many different features. all four of these teams performed some type of markov modelling at the system's top level. however, the teams used different techniques on their markov models: maximum entropy, hidden markov models and conditional random fields. in addition, one team, also had an svm system at the top level: decisions were made by having two hmms and an svm system vote. also, note that when comparing different systems, the choice of features used is often at least as important as the approach/algorithm used. yeh gives an example of this. task 1a as a building block one reason for evaluating on task 1a is that a task 1a system can serve as a building block for other tasks, like task 1b or task of the biocreative evaluation. the task 1b evaluation focused on finding the list of the distinct genes mentioned in a medline abstract, where the list contained the normalized, unique identifiers for those genes. task focused on functional annotation, and on returning text passages as evidence to support these classifications. to what extent was it viable to use task 1a systems as a building block for more advanced capabilities? it turns out that three of the teams taking part in task 1a also took part in task 1b. in addition, one of the three teams also took part in a portion of task so an interesting question is whether these three teams found their task 1a systems to be useful when working on task 1b or one team with a high precision task 1a system used the mentions found by their 1a system as the input for their 1b system: their 1b system then tried to find the normalized version of the mentions found by their task 1a system. the story was more complicated for two other teams with both high precision and high recall task 1a systems. one team was from pennsylvania. the other team was from edinburgh and stanford. both these teams looked at some version of finding mentions with their task 1a system and then compared the found mentions against the synonym lists for the genes of interest for task 1b. both teams found that this approach could easily produce a low precision for 1b, due to ambiguity. the pennsylvania team also found that for genes from two of the three organisms of interest in task 1b, the task 1a tagger was not that accurate. a possible explanation given was that the task 1a training data did not have enough examples from these two organisms. for task 1b, the pennsylvania team in the end did not use their task 1a tagger. the edinburgh/stanford team found that using the original task 1a training set and lots of features tended to lower their recall of the 1b genes. they raised the recall by retraining their 1a system using the noisy task 1b training data and a reduced set of the possible features. the edinburgh/stanford team also took part in task. in this task, a system was given an article, a protein mentioned in that article, and a classification of that protein that a person made based on that article. the system's job was to find a passage of text in that article that supported the classification made for that protein. the description for the team's task system made no mention of using their task 1a system or trying it on some part of task. discussion one unique aspect of the data: enforcing a particular tokenization as mentioned before, every entity mention task such as task 1a will have some features that are more or less unique to it. for task 1a, one such feature is that the data comes with a particular tokenization. furthermore, this tokenization affects what counts as a mention, because either all of a token is tagged as part of a mention, or none of that token is tagged. this can cause problems when one just wants to tag part of a token as part of a mention. an example is the phrase "a protein kinase a-mediated pathway", shown in figure with the red vertical bars indicating the word token boundaries. here the token "a-mediated" is not useful, since the mention that one would really like to tag is "protein kinase a". this tokenization is important because it affects what counts as a mention; below are some rules: if "x" is a token which is a gene name, then "x" is usually marked. an example is "cbf1" in the phrase "... of cbf in yeast...". if a token is of the form "x-" or "x-y", where "x" is a gene name and "y" is an adjective or verb, then the token is usually not marked. an example is "egf-induced" in the phrase "... block egf-induced mitogenesis and... ". an exception to: when the y in "x-y" is "like", then "x-y" is usually marked. also, if the form is "x-y z", where "x-y" is as in, and "z" is a token like "domain", then "x-y" is usually marked as part of the mention "x-y z". an example is "sh2-binding domain". disagreements in the data in tasks like task 1a, small disagreements usually exist on what to annotate and what not to annotate. an example in task 1a is phrases of the form "x pathway", where x is a phrase that is marked as part of a gene mention. an initial review of the test set found the following annotation variations " both allowed as alternative answers): • cases where "x pathway was not an allowed alternative to "x". an example was x = "mek-erk1/2" in the phrase "... the mek-erk1/ pathway by...". • cases where "x" and "x pathway" were both allowed alternatives. an example was x = "ras/raf/mapk" in the phrase "... the ras/raf/mapk pathway.". similarly, the training set had • cases where "x pathway" was not an allowed alternative to "x". • cases where "x pathway" and "x" were allowed alternative answers. such variation in annotation makes it more difficult to learn or to formulate a rule for how to handle these kinds of constructions. lessons learned for future evaluations if and when a future task 1a evaluation is run, we list the following issues to consider: tokenization is non-trivial for biological terms. perhaps one should not enforce a fixed tokenization of the data. this non-enforcement will be expensive because it requires changing both how the data is annotated and how the system results are compared against the gold standard. on a related matter, because of the difficulties in exactly determining a mention's boundaries, there is interest in also counting inexact matches to answers as being correct. this must be defined carefully. for example, if missing either the first or last token still counts as correct, then just returning "epsilon" would count as finding "pkc epsilon". for open versus closed submissions, we should either remove the distinction, or be more explicit as to what is allowed for a closed submission. a suggestion was made to pad the test set with extra material that would not be scored, which would make it harder to "cheat" by manually examining the test set. if this were done, one would need to announce this ahead of time. one reason is that some automated approaches need more processing time than others. another reason is that some automated approaches, such as transductive support vector machines, make use of statistics derived from the entire un-annotated test set. at least one team automatically searched for the pubmed/medline abstract associated with each test set sentence. they used the abstract as a surrounding context, and it seemed to be helpful. in many "real uses" of a task 1a system, a system will probably have such surrounding text. so it may make sense to just give these abstracts to every participant in the future. there is also a question of what is a permissible resource to use: • one example is that with pubmed/medline, a system could also look-up mesh terms, etc. associated with the medline abstract for each sentence. if a tagging system is applied before an abstract is assigned mesh labels, then such information will not be available in real usage, and such information should not be permitted. • given a possible entity "x", at least one team did web searches for contexts like "x gene", which support "x" being a possible entity. this seemed to be of limited help. should this be permitted in the future? this probably depends on the anticipated "real" uses for such a feature. when tagging older material, the web will have relevant material. when tagging new text that describes new gene, the web will probably not have much, if any material.  for the biocreative task 1a of gene mention finding, a number of teams achieved an 80–83% balanced f-score. unless a system was not performing well, using external resources did not seem to help that much. these 80%+ results are similar to results for some other similar biological mention finding tasks, and are somewhat behind the 90%+ balanced f-scores achieved on english newswire named entity tasks. based on an observation offered by kevin cohen, one hypothesis for the discrepancy is that gene names tend to be longer than comparable newswire names. to investigate this, we compared the length distribution of gene names in the test set for task 1a; this distribution is shown in figure and is compared to the distribution for name length of organization names in a newswire task. the newswire results are computed from the muc- data, which is available from the linguistic data consortium. the average length of the task 1a gene names was, compared to for organization names in the muc- data. given this distribution, we fitted a simple logistic regression model to both data sets. we modeled the performance for a name of n words as n, where e is the performance on a single-word name. this allowed us to extrapolate back to a single-word error rate for both tasks, allowing us to factor out differences in name length. for gene names, a 91% success rate on a single word gene name gave an overall task performance of 83%, the observed high score. for the muc- organization names, a % single word success rate yielded a 93% success rate overall, which was the highest recorded result for muc- in using this simple model, we recognize that it is not mathematically valid to use f-measure in place of accuracy. however, it does provide a crude approximation for how much of the task difficulty can be attributed to difference in name lengths among different tasks. this comparison leaves a residual 4–5% discrepancy between performance on the tasks for the single-word case. we hypothesize that this may be due to interannotator variability, leading to "noise" in the training and test data. for the muc- task, interannotator agreement was measured at 97%, which is almost certainly significantly higher than for the gene mention task, which has not yet been formally measured. this variability affected at least one high performing task 1a system. the discussion section of this paper gives some sample disagreements in the task 1a data. in terms of successful approaches, the teams that achieved an 80% or more balanced f-score tended to use some type of markov modelling at the top system level. however, these teams also had post-processing stages in addition to the main approach taken, and the different teams made use of different features. these stages and features can have just as much an effect on performance as the main approach taken. one of the reasons to have task 1a is that it should be a useful building block to work on other tasks, like biocreative task 1b. three teams tried using their task 1a system for task 1b. their experiences were mixed, with two of the three teams finding that a task 1a system trained on the task 1a training data often did not work so well on task 1b. one of these two teams improved things by retraining their 1a system using the noisy task 1b data. a 2nd test set is available for task 1a, so it would be straightforward to run a task 1a evaluation in the future using this 2nd test set. four questions to think about in any future evaluation are the following: • what will it take to improve task 1a performance? • how much will improving task 1a performance help with other tasks? • how can one make a task 1a system be a more useful building block for other tasks? • why are outside resources not more useful in task 1a? is it because task 1a is unique? 
the biological research literature is a major repository of knowledge. the task makes use of data and evaluation software provided by the national center for biotechnology information. to address this, we worked with colleagues at the protein design group, cnb-csic, madrid to develop biocreative, an open common evaluation of systems on a number of biological text mining tasks. we report here on task 1a, which deals with finding mentions of genes and related entities in text. teams took part in task 1a. the teams that tried to use their task 1a systems to help on other biocreative tasks reported mixed results. the 80% plus f-measure results are good, but still somewhat lag the best scores achieved in some other domains such as newswire, due in part to the complexity and length of gene names, compared to person or organization names in newswire. as the amount of literature increases, it will get harder to find the information of interest on a particular topic.
 the biological research literature is a major repository of knowledge. as the amount of literature increases, it will get harder to find the information of interest on a particular topic. there has been an increasing amount of work on text mining this literature, but comparing this work is hard because of a lack of standards for making comparisons. to address this, we worked with colleagues at the protein design group, cnb-csic, madrid to develop biocreative, an open common evaluation of systems on a number of biological text mining tasks. we report here on task 1a, which deals with finding mentions of genes and related entities in text. "finding mentions" is a basic task, which can be used as a building block for other text mining tasks. the task makes use of data and evaluation software provided by the national center for biotechnology information.  teams took part in task 1a. a number of teams achieved scores over 80% f-measure. the teams that tried to use their task 1a systems to help on other biocreative tasks reported mixed results.  the 80% plus f-measure results are good, but still somewhat lag the best scores achieved in some other domains such as newswire, due in part to the complexity and length of gene names, compared to person or organization names in newswire. march 28– a critical assessment of text mining methods in molecular biology granada, spain 