BACKGROUND
in the biological sciences, describing species has long been a basic and essential component to studies of biodiversity, systematics, and evolution.
scientists have published detailed descriptions of organisms for centuries, amassing a rich legacy of taxonomic literature that includes descriptions of phenotypic characters  for millions of species, written in natural language.
microbial descriptions of prokaryotes, in particular, also contain rich phenotypic information relating to metabolism and environmental niche space.
with the genomic revolution, molecular  data is often relatively easy to obtain, and integrating molecular and phenotypic information has proven to be an important approach to study the evolution of microbial traits on a broad scale .
however, the majority of phenotypic information is currently locked in natural language descriptions and not readily usable for bioinformatic comparative analyses.
in order to study the evolution of phenotypic traits on a large scale, i.e., covering the phenome  <cit> —the full set of phenotypic characters—of an organism, character information needs to be in a standard structured format  that can be used by existing phylogenetic comparative analysis software packages, such as mesquite  <cit>  and macclade  <cit> .
traditionally, taxon-by-character matrices  are created manually by experts through reading and selecting phenotypic data in published taxonomic descriptions.
this process is time-consuming and tedious.
hence, most studies have focused on a small number of taxa, or a small number of characters .
clearly, more efficient means of obtaining large amounts of phenotypic information for large numbers of taxa are required in order to develop a comprehensive synthesis of the distribution and evolution of diverse prokaryotic traits in the fields of microbial systematics and evolutionary biology.
information extraction techniques from text mining can provide solutions to this problem by  automatically extracting character states  from natural language text.
text mining has been used to find patterns from plain text within the biological and biomedical domains  <cit> .
examples include the extraction of text to annotate protein-protein interaction networks  <cit> , metabolic pathways  <cit> , gene regulatory networks  <cit> , and signaling networks  <cit> .
within a text mining system, information extraction is an important task to extract knowledge elements from texts.
the extraction targets are typically entities , the relationships among entities , and biological events   <cit> .
the techniques used to recognize the extraction targets may be categorized as employing 1) an existing knowledge base , 2) machine learning or computational statistics methods, and/or 3) natural language processing  methods.
these techniques are typically used in combination to achieve the best performance.
the discovery of domain entities is often achieved using an existing knowledge base and enhanced with machine learning mechanisms.
machine learning algorithms use statistical methods to model the problem of interest and make predictions based on the model  <cit> .
for example, support vector machine  is a general-purpose, robust, supervised classifier based on mathematical theory of hyperplane arrangements  <cit>  that can be trained with tagged examples, then used to classify new examples.
because of its robustness and wide applicability, svm is used in text mining for document classification  <cit> , sentence classification  <cit> , named entity recognition , and beyond .
extensively used in information extraction, nlp techniques combine language theories and statistical methods to analyze syntactic structure of plain text  <cit> , entity roles, and their relationships.
to capture a variety of phenotypic characters from microbial descriptions, we applied multiple techniques, including svms and syntactic analyses.
in biodiversity domains, significant strides have been made recently in the development of automated text mining and natural language processing approaches to extract morphological data from the taxonomic literature of plants and animals.
one such example is charaparser  <cit>  developed for extracting morphological characters from semi-structured taxonomic descriptions written in a telegraphic style  <cit> .
initial tests , however, showed that charaparser performed poorly at extracting data from prokaryotic taxonomic descriptions because these descriptions contain little morphological data, but rather focus on physiological and ecological traits .table  <dig> high-level categories  and characters implemented in micropie
strictly anaerobic.
obligately anaerobic.
colony morphology*
metabolic end products and secondary metabolites*
microbe-host interaction*
metabolic substrate*
example source sentences for each character within each category are provided.
bolded text in the source sentences indicates the values that micropie should extract
domain-independent information extraction systems have been reported, which include open information extraction   <cit>  and deepdive  <cit> .
oie extracts entity pairs connected through verbs so that some kinds of character values could be extracted; for example, “bacteria” will be returned as an appropriate answer for the question “what produces methane?” by oie from the sentence “these bacteria eat through practically anything and produce methane gas in the process.” however, although oie relies on textual context to extract information, it was not adequate for our task, which required substantial domain knowledge.
deepdive provides statistical machine learning and inference support for building domain-specific, information extraction systems for knowledge base construction from texts  <cit> .
deepdive has been used successfully to build information extraction systems for specific domains, for example, pharmacogenomics  <cit> .
however, it is essentially an interactive system and requires considerable user-developer interactions and input from domain experts.
algorithms/programs specifically designed to extract microbial-related characters also exist.
humphreys et al.
<cit>  designed the empathie system based on domain concepts and various nlp techniques to extract enzyme and metabolic pathways.
their system was evaluated on a small corpus with seven journal articles, and the performance was less optimal   <cit> .
czarnecki et al.
<cit>  proposed a text-mining system for extracting three types of metabolic reactions.
the task was accomplished through two steps: entity recognition by existing text mining tools  and reaction extraction by scoring the permutations of the recognized entities in sentences selected via heuristic/syntactic rules.
the performance for these methods was poor to high, having recall scores ranging from  <dig> to 82% and precision scores ranging from  <dig> to 88%  <cit> .
these existing text mining systems could provide part of the solutions to phenotypic character extraction but were insufficient for extracting extensive, comprehensive microbial phenotypic traits needed to generate character matrices.
ontologies are used not only as a source of knowledge in information extraction systems, they are also used as a way to standardize the results of information extraction.
matching natural language phrases to ontology terms is often treated as a separate step/task after the task of extracting character information from text.
for example, bacteria biotope task of the bionlp shared task  <cit>  includes a task of extracting bacteria and their habitat information, and a separate task of matching extracted entities to an ontology.
this paper reports the development and evaluation of a software application, micropie  which extracts  <dig> microbial phenotypic characters from prokaryote descriptions and generates a character matrix.
examples of characters  include cell diameter , cell length , motility , and g + c content .
the input to micropie is a set of prokaryotic taxonomic descriptions .
an example output matrix is shown in fig.
<dig>  in this paper we focus on character extraction from natural language descriptions.
the standardization of the extracted characters using ontologies to facilitate data sharing and integration  is discussed in future work.fig.
<dig> an example input description to micropie, simplified from  <cit> , used with permission
fig.
<dig> part of a hypothetical output matrix.
the row in bold corresponds to the description in fig.
1
here, we describe the process of defining the extraction targets for micropie, its system architecture and character extraction methods, as well as its performance evaluation metrics.
we then report the performance results of micropie with and without its character prediction component, and compare its performance to the performance of a group of undergraduate microbiology students.
after discussions on system performance and algorithm refinements, we conclude the paper with a future development plan for micropie.
methods
extraction target identification and selection
exploratory studies were first conducted to identify the characters that needed to be extracted.
to broadly represent the diversity of text and traits in prokaryotic taxonomic descriptions, a corpus of  <dig> descriptions was initially sampled from three evolutionarily distant groups .
published taxonomic descriptions were obtained as pdf files from a variety of journals, including international journal of systematic and evolutionary microbiology  <cit> , proceedings of the national academy of sciences of the united states of america  <cit> , etc.
each taxonomic description was semi-automatically transferred into a text file.
pdf-to-text conversion and/or formatting errors were manually corrected so that the extracted text matched the original.
the collected microbial taxonomic descriptions were then segmented to  <dig> sentences using the stanford corenlp toolkit  <cit> .
two r packages implementing lsa   <cit>  and topic models  <cit>  were used to analyze the content of microbial taxonomic descriptions.
this analysis identified  <dig> topics as raw categories, such as g + c content, growth temperature, and cell size.
these topics were then combined, resulting in a set of  <dig> high-level categories that cover general traits of prokaryotes.
consulting the taxonomic description corpus, the characters were specified under each category and a set of  <dig> characters were defined as the extraction targets for micropie .
micropie system architecture
figure  <dig> shows the system architecture of micropie.
text input is first converted into a simple xml format where publication metadata , taxon names and description paragraphs are wrapped in separate elements.
next, the xml files are analyzed by micropie preprocessor, character predictor, character extractor, and matrix generator in sequence to produce a taxon-by-character matrix.
micropie does not automatically detect/remove descriptions that are potentially repetitive or highly similar, because they could represent different taxon concepts  <cit> , the subject of research for some potential users of micropie.fig.
<dig> system architecture of micropie
the micropie preprocessor component has two main steps: sentence splitting and sentence cleaning.
in the sentence splitting step, the description paragraphs are split into sentences using stanford corenlp  <cit> .
in the sentence cleaning step, sentences are normalized by replacing predefined xml entities  and characters that are not compatible with the utf- <dig> charset, and by removing non-alphanumeric characters .
micropie preprocessor supplies the subsequent components with clean, normalized sentences to reduce potential syntactic parsing errors.
the character predictor consists of a set of trained svm classifiers , one for each character.
the function of the character predictor is to predict the characters a sentence describes by classifying the sentence into one or more of the  <dig> character classes .
the svm classifier had been trained with the  <dig> sentences from the  <dig> descriptions mentioned before, then semi-manually labelled with appropriate characters .
for characters with fewer than ten training sentences, more sentences were added by searching google scholar.
in total,  <dig>  training instances  were generated.
the number of training sentences per character ranged from tens to more than  <dig>  when training the svms for a character, training sentences labeled with the character were used as its positive training examples, and the same number of negative training examples were randomly selected from the remaining sentences.
this is done to keep the positive and negative training examples balanced as svms are sensitive to unbalanced training examples  <cit> .
the trained svms achieved an average  <dig> % precision and  <dig> % recall in character prediction as evaluated via a 10-fold cross validation.
the character predictor uses each of the  <dig> trained libsvm classifiers to classify a sentence.
for example, the sentence, “cells are  <dig> – <dig>  μm wide and  <dig> – <dig>  μm long”, may be classified to both cell width and cell length characters.
after character prediction, sentences are passed to the appropriate character extractors to extract the target values.
a variety of character extractor algorithms have been implemented in the character extractor component; and they can be categorized into two main groups: linguistic rules and term matching .
since one sentence may be classified into multiple characters by the character predictor component, the sentences may be processed by multiple extractors.
on the other hand, one extractor may be used for several character classes of similar nature.
for example, cell diameter, cell width, and cell length are processed by the same extractor that is based on linguistic rules shown in fig.
<dig>  the target values for the three characters share a similar linguistic pattern: the keywords that indicate the character follow the candidate numeric values within a 3-word window.fig.
<dig> a shared extractor based on linguistic rules for the characters cell diameter, cell width, and cell length
the results from these extractors are then post-processed using predefined heuristic rules to generate the final results.
for example, the cell length value “ <dig> – <dig>  microns” in fig.
<dig> was standardized as “ <dig> – <dig>  μm” in the post-processing step.
the last component of micropie, matrix generator, creates a taxon-by-character matrix with the values extracted for the characters.
multiple values for a character are separated by number signs .
micropie character extraction methods
linguistic rules
the linguistic rules for a character were identified from its svm training examples when developing micropie.
text segments are extracted as candidate character values if the text segments satisfy relevant linguistic rules.
linguistic rules used in micropie include regular expressions, part-of-speech  tag patterns and syntactic patterns  <cit> .
in some cases, multiple linguistic rules are often integrated in one character extractor to deal with varied textual expressions of a character.
regular expressions are used to extract character values with clear lexical clues; for example, a minimum and maximum for the range of ph values  at which a bacteria could grow can be extracted from “grows at ph 6–8”.
regular expressions only exploit lexical patterns of the sentences.
pos tag patterns, in contrast, also take the pos tags of the words into consideration.
pos tag patterns often consist of pos tags and specific words.
for example, one simple pattern “<cd > mol%” is used to extract the number with the pos tag “cd”  followed by “mol%” as the value of g + c content.
syntactic patterns used in micropie are the character-indicating syntactic dependencies  <cit>  identified from microbial descriptions.
for example, if a sentence classified as fermentation substrate used contains the phrase “is produced from”, then this verb phrase often indicates fermentation related characters : noun phrase  <dig> , or the subject  of the verb phrase, is often a fermentation product, while noun phrase  <dig> , or the object , is a fermentation substrate.fig.
<dig> an example of a syntactic pattern used in micropie
term matching
the term matching approach references the knowledge base which consists of lists of terms  that were created manually for  <dig> of the  <dig> characters, as well as terms from other ontologies such as the plant ontology  <cit>   and the human disease ontology  <cit>  .
the knowledge base provides the domain knowledge micropie cannot gather from the text descriptions alone.
for example, if it is known that “chains” is associated with the character cell relationships & aggregations, a value  may be extracted from the sentence “some cells are arranged in short chains”.
noun phrases of sentences are first obtained by applying heuristic rules with pos tag information.
each term in the term list is used to formulate a regular expression.
then, the regular expressions for a character are applied to a noun phrase to determine whether this phrase is a candidate value of the character.
for some characters whose character states appear as in the term list without extra variations, the character values are extracted by checking whether the terms exist in the sentences.
in addition, syntactic rules can be combined with term matching to disambiguate the characters of the phrases through restricting the dependency roles of the matched phrases in the sentence.
for example, if the np <dig> in fig.
<dig> matches with a term in the term list of fermentation substrate used, the confidence in this phrase being the character value is further improved.
it is important to note that use of term matching alone is not sufficient for the character extraction task, because  these terms could occur in descriptions outside of the designated character classes, and  extraction targets are often more complex than a single phrase.
to overcome the insufficient coverage of the term lists for some characters, coordinated noun phrases e.g., “chloramphenicol, streptomycin and kanamycin”, are identified.
if one phrase in the coordinated noun phrases is recognized as a character value, all phrases in the coordination are assumed to be the values for the character.
micropie performance evaluation
the performance of micropie was evaluated against a gold standard character matrix  prepared by microbiologists on our team, and also compared to the performance of undergraduate microbiology students who manually extracted a subset of microbial phenotypic characters.
development of the gold standard matrix
taxonomic descriptions  from international journal of systematic and evolutionary microbiology  <cit>  were used to create a gold standard matrix .
the gsm contains the ground truth of the characters for each taxon.
gsm creation was time-consuming due to the ambiguities in some taxonomic descriptions or the complexity of the characters and syntax.
to accelerate the manual process, the trained svm classifiers were used to predict the characters in the sentences.
any missed or incorrect predictions of the svm classifiers were manually identified and corrected.
despite the use of the svm classifiers, the generation of the gsm took well over  <dig> person-hours.
in order to compare the character values in the gsm to machine-generated values from micropie, the values were formulated in a structured way.
string-based values were formatted as “negation │ modifier │ main value” with the main value as the only required field.
modifiers are phrases that refine the main value in terms of degree, extent, or frequently.
for example, “not strictly anaerobic” is formatted as “not │ strictly │ anaerobic”.
the format for characters with numerical values had additional fields, “negation │ modifier │ main value │ unit │ sub-character”, with the main value being the required field which could hold a single or range numeric value.
the field sub-character is only used to indicate the kind of salt , for the characters of nacl minimum, nacl optimum or nacl maximum.
for example, “2–3% nacl” is formatted as “2– <dig> │ % │ nacl” .
of the  <dig> taxonomic descriptions in the gsm,  <dig> were used to optimize/test the character extractors, and the remaining  <dig> were used for the evaluation of the system.
micropie performance evaluation metrics
micropie performance metrics were calculated by comparing the character matrix produced by micropie against the gsm.
precision , recall , and f1-measure  scores were adopted to evaluate micropie performance for each character.
for each value within a character, a hit score was assigned denoting how well the extracted value matches the gsm value on a scale of 0– <dig>  with  <dig> being a perfect match,  <dig> being no match and  <dig>  being a partial match.
from these hit scores, p, r and f <dig> were calculated as follows: p=sumofhitscores#ofallextractedvalues
r=sumofhitscores#ofvaluesingsm
f1=2prp+r
there were various scenarios to consider in scoring the match between an extracted and a gsm value.
thus, we designed both a rigid method and a relaxed method to calculate hit scores which in turn produces rigid and relaxed p, r, and f <dig> scores.
if all formatted fields of the two values matched exactly, both the rigid and the relaxed hit scores were  <dig> .
a score of  <dig> was given if the main value was not extracted , a wrong value was extracted, or the negations did not match .
if the main values and negations matched, but the other fields did not, the rigid hit score was  <dig>  and the relaxed hit score was  <dig> .
if the negations matched, and main values partially matched, both the rigid and relaxed hit scores were set to  <dig>  .
for characters with multiple values, each of the values was compared individually with the values in gsm and the sum of the individual scores was the final hit score .table  <dig> examples of how performance evaluation metrics were calculated
rigid and relaxed hit scores measuring the match between extracted values and gold standard matrix  values, illustrated with examples
hit scores were computed using string matching.
for imperfect matches, we carried out additional manual review using a designed user interface through amazon mechanical turk  <cit> .
this method allowed for the manual adjustment of relaxed scores, if warranted, since the automatic scoring algorithm biased the extracted values because it often assigned  <dig> to partial matches.
it also allowed, in some cases, for the correction of gsm values that were initially coded incorrectly, and for the rescoring of these values.
additionally, we compared the sum of the relaxed hit scores to the sum of the total value occurrences in the gsm to calculate the % of values that micropie extracted successfully.
comparison to a student-generated character matrix
undergraduate microbiology students at university of southern maine and kenyon college created a matrix in their guided class projects for  <dig> of the  <dig> prokaryotic characters from a subset  of the  <dig> taxonomic descriptions in  <dig>  the relaxed evaluation scores for characters in the student-generated matrix compared to the gsm were calculated manually because the values in the student-generated matrix were formatted differently from the gsm.
the statistical significance of the performance differences between the student-generated matrix and the micropie output was evaluated by using student's t-test comparing description-based relaxed_p, relaxed_r, and relaxed_f <dig> scores.
impact of character predictor on micropie performance
in micropie, the character predicator triages sentences into different character extractors .
to assess the effects of the character predictor component on character extraction results, the same matrix generation task was run with and without character predictor.
in the latter case, sentences were passed into character extractor from preprocessor directly.
the performance differences in the evaluation metrics between micropie with/without character predictor were compared.
RESULTS
the performance of micropie with character predictor
it took 47 min and 45 s for micropie to process the  <dig> taxonomic descriptions used as evaluation dataset collected from international journal of systematic and evolutionary microbiology  <cit>   and output a matrix, running on a standard pc laptop  running java .
in contrast, it took over  <dig> person-hours to create the gold standard matrix.
table  <dig> shows the performance scores of micropie with the character predictor  enabled.
the number of values in the gsm and micropie matrix is shown for each character as well.
micropie extracted  <dig> total values, resulting in a total relaxed hit score of  <dig> , out of the maximum total values of  <dig> from the gsm .
in other words, micropie extracted  <dig> % of character values successfully.
half of the  <dig> characters had relaxed_f <dig> scores greater than  <dig> .
among those, the relaxed_f <dig> score for  <dig> characters was greater than  <dig> , the relaxed_f <dig> score for  <dig> characters was between  <dig>  and  <dig> , while the relaxed_f <dig> score for only  <dig> characters was equal to or below  <dig> .
of all the characters,  <dig> had the same rigid and relaxed scores, while the other  <dig> characters had greater relaxed scores.
thus, micropie extracted at least partially correct values for most characters.table  <dig> performance of micropie with character predictor
%g + c
<dig>
cell diameter
<dig>
cell length
<dig>
cell width
<dig>
cell relationships & aggregations
<dig>
gram stain type
<dig>
internal features
<dig>
motility
<dig>
pigment compounds
<dig>
nacl minimum
<dig>
nacl optimum
<dig>
nacl maximum
<dig>
ph minimum
<dig>
ph optimum
<dig>
ph maximum
<dig>
antibiotic sensitivity
<dig>
antibiotic resistant
<dig>
colony shape
<dig>
colony margin
<dig>
colony texture
<dig>
fermentation substrates not used
<dig>
in total
4098
4049
total relaxed hit scores
<dig>
abbreviations: superscript n numerical character, s string-based/categorical character.
the characters with > =  <dig>  in relaxed_f <dig> score are shown in bold
the performance of micropie without character predictor
the performance scores for micropie with the character predictor  component disabled were also calculated.
without character predictor component choosing the appropriate character extractor, micropie applied all the character extractors to each sentence and extracted a total of  <dig> values,  <dig> % more than what was in gsm .
compared to micropie with svm, the total relaxed hit score was increased by  <dig>  .
at the same time, about  <dig> incorrect values were also extracted, resulting in a decreased accuracy of  <dig> % ,  <dig> % lower than the accuracy of micropie with svm .
the performance differences in relaxed_p, relaxed_r and relaxed_f <dig> scores with and without character predictor are presented in fig.
<dig>  micropie with svm achieved higher relaxed_p scores for  <dig> characters and resulted in a slightly lower score for only  <dig> character .
micropie without svm achieved higher relaxed_r scores for  <dig> characters .
note that the differences in recall  were smaller than they were in precision .
the combined effect on micropie performance can be seen in fig.
6c in terms of relaxed_f <dig> scores: micropie with svm performed better for  <dig> of the characters but performed worse for  <dig> characters , albeit by a small margin .fig.
<dig> the performance comparison between micropie with and without character predictor.
a relaxed_p, b relaxed_r, and c relaxed_f <dig> scores
comparison between micropie and student performance
the performance scores of the student-generated matrix were compared to those of the micropie-generated matrix.
micropie performed significantly better than undergraduate microbiology students .
the total relaxed hit score of the micropie matrix  was 123% higher than that of student-generated matrix .
in addition, micropie consistently performed better than students on all the compared characters and in all metrics .
student's t-tests on the description-based relaxed_p, relaxed_r and relaxed_f <dig> scores indicate statistically significant differences between micropie and student performances .table  <dig> the performance comparison between the students and micropie on  <dig> characters in  <dig> taxonomic descriptions
discussion
micropie performance analysis
characters with more than  <dig> character values in the gsm produced relaxed_f <dig> scores greater than  <dig>  , indicating that micropie performed better when character values appeared more frequently in the gsm.
these same characters also appeared more frequently in the corpus of  <dig> taxonomic descriptions, providing abundant domain terms and robust linguistics rules during the development of micropie and hence resulting in a final algorithm that can more reliably extract the appropriate target values.fig.
<dig> scatter plot showing the relationship between relaxed_f <dig> scores and frequency of character value occurrence in gsm for each of the  <dig> characters.
the axis of the number of gsm values is log-transformed
the higher performance  for the  <dig> numerical characters  reflects the effectiveness of various linguistic rules.
linguistic rules, specifically syntactic patterns, also resulted in higher performance for some string-valued characters  when useful patterns about the character values were well captured by the extractors; for example, antibiotic sensitivity and antibiotic resistant achieved high relax_f <dig> scores of  <dig>  and  <dig>  respectively.
the term matching method achieved high extraction performance for some characters as well, such as gram stain type and colony shape .
however, multiple factors could contribute to the failure of term matching method .
the coverage of the term lists may be limited , terms may be included in the lists of multiple characters making it difficult to determine the correct character in a specific sentence, or terms may be only part of a value to be extracted .
in addition, micropie attempts to extract a complete phrase as opposed to a single word, but this does not solve the cases where one phrase contains multiple values  or where one value is expressed across phrase boundaries .
learning the relationships within and between phrases will help address these problems.
during the development of the gsm and evaluation of micropie, we experienced a fair amount of tension between the need to extract as much information as possible and the need to fit the values in the matrix meaningfully so the values of each character were comparable between taxa.
this largely stemmed from the variations in description content, and sometimes the vagueness of the language, in taxonomic descriptions.
for example, the following sentence,“growth occurs at −2 °c in marine broth.
good growth occurs on ma at 1–25 °c.
no growth occurs at 30 °c or higher”,
is ambiguous with respect to the optimal growth temperature.
although experts could judge “1–25 °c” as the optimal growth temperature , micropie identified “1–25 °c” as a temperature related growth condition, but did not output the value as the optimal.
such ambiguity presents a challenge for micropie.
thus, we designed micropie to adhere to the principle of “stay true to the original descriptions” and to not attempt to make a judgement that needs much expert knowledge.
the effects of character predictor
micropie with the svm character predictor showed significantly higher precision than micropie without character predictor .
with highly accurate character prediction, sentences containing specific characters were triaged to their specific character extractors, rather than to all character extractors non-discriminatorily.
the triage alleviated issues in term matching caused by terms that are shared by multiple characters.
for example, the term “filamentous” appears in term lists of both cell shape and colony shape.
thus, micropie without character predictor extracted this term as values of both characters, while character predictor utilized the information in the complete sentence to predict the correct character so that the correct term list was used.
our results show that using svm resulted in increased relaxed_p scores for five characters with shared terms in the term lists: cell shape improved by  <dig> , external features  <dig> , internal features  <dig> , colony shape  <dig> , and colony texture  <dig> .
precision and recall scores are known to have an inverse relationship  <cit> —higher precision often means lower recall—which is also manifested in this case with the slightly lower recalls for micropie with character predictor .
characters showing the larger decreases in recall scores were the characters with values having low frequencies of occurrence  and hence a few missed values resulted in much lower recall scores.
for these characters, the insufficient training examples  affected the svms ability to make correct predictions for new data .
although more training examples were available for nacl minimum, aerophilicity, and inorganic substances not used, the character predictor predicted more than 10% of relevant sentences as irrelevant, resulting in decreased recall performance for these characters.
this suggests that the training examples of these characters were not very good representatives of the test descriptions.
despite the lower recall scores, the extraction accuracy and f <dig> scores suggest that svms contributed positively to micropie performance.
comparison between micropie and student performance
a detailed analysis of student’s matrix showed that their extraction errors may be attributed to students’ inability to understand the descriptions completely.
their lower performance suggests that, lacking sufficient domain knowledge, novices with only some background knowledge are not capable of extracting microbial characters with high accuracy.
the performance comparison between micropie and students highlights the challenges in this task and the need for high quality domain knowledge for computer algorithms to perform well and further improve.
differentiating micropie with entity annotators
many annotation tools  match individual phrases in textual descriptions to ontology terms.
such systems are “entity recognition” systems, where individual entities  are recognized, but relationships among these entities are not identified.
in contrast, a key function of micropie is to relate character values to appropriate characters.
once the character values are extracted and related to their characters, they can then be matched to terms in ontologies using algorithms such as bioportal annotator algorithm .
future work
although the performance of current micropie is promising, we have identified several areas it can be further improved.
new methods for detecting extraction boundaries and for constructing linguistic rules automatically need to be explored.
one such candidate is unsupervised semantic parsing , which combines markov network and first-order logic to cluster terms with similar semantic roles based on their syntactic dependencies with surrounding text  <cit> .
with this technique, instead of handcrafting a rule to extract anything following the verb “hydrolyze”, the cluster of terms, “hydrolyse”, “hydrolyze” and “degrade”, will enable micropie to generate a more robust rule of extracting anything following any of the terms in the cluster.
another way in which micropie could be improved would be to replace simple term lists with an ontology.
the term lists in this study formed the foundation for a new microbial ontology, micro  <cit> .
micro contains classes of phenotypic and metabolic characters, assays, and culture media of prokaryotes, connecting terms to  <dig> other ontologies using logical axioms.
it also includes a large number of synonyms and alternate spellings for terms found in the corpus of taxonomic descriptions used in this study.
micro would provide a richer knowledge base for micropie, and would assist in the disambiguation of competing characters.
it would also provide a controlled vocabulary for characters and character states, and identify synonymous phrases that frequently occur in prokaryotic taxonomic descriptions.
in addition, the logical axioms in the ontology would make it possible for higher-order knowledge to be automatically inferred from extracted characters .
future versions of micropie will utilize micro and a few other ontologies such as the cell ontology  and the ontology of units of measure  to improve extraction performance and also standardize character output with ontology terms.
in addition to a lot of new microbial phenotypic terms, micro has imported many terms relevant to our task from other ontologies , using it in micropie is the most logical way forward.
a larger number of training sentences and expanded knowledge resources  will also improve the performance of character extraction methods, especially for those characters with fewer occurrences in the current training sentences and the gsm .
micropie could also be improved by expanding the variety of target characters for extraction.
the current version of micropie does not extract characters such as microbial filterability, habitats, physiological tests, lipids, and hosts.
micropie also could be enhanced by giving it the capability of discovering new characters from taxonomic descriptions, and by processing descriptions from different microbial taxa, e.g., cyanobacteria and eukaryotic phototrophs which follow the botanical code rather than the bacteriological code  <cit> .
although micropie was designed to extract data from semi-structured taxonomic descriptions, in future experiments, we intent to evaluate its performance on other description texts, such as less structured descriptions of prokaryotic taxa in pubmed or in online databases , and to research and expand its capability and robustness.
the combination of data extracted from semi-structured descriptions and incorporation of the rich micro ontology will certainly be useful for micropie to tackle different description styles/formats.
CONCLUSIONS
we report the development and evaluation of micropie, a phenotypic information extraction system for prokaryotic taxonomic descriptions.
we show that micropie greatly increases the speed by which phenotypic character information can be extracted from published taxonomic descriptions.
evaluation against a hand-generated gold standard matrix showed that micropie performed well  on over half of designated characters and achieved an overall accuracy of  <dig> % and overall performance that was significantly better than the performance of undergraduate microbiology students.
it was also shown that inclusion of the character predictor component using svm  improved the overall performance of micropie.
we therefore conclude that micropie has the potential to significantly accelerate the construction of phenotypic character matrices for prokaryotic taxa, removing a fundamental roadblock for the study of large-scale character evolution in microorganisms.
we expect future improvements will make it more robust and effective.
abbreviations
g + cdna molar guanine and cytosine content
gsmgold standard matrix
lsalatent semantic analysis
micropiemicrobial phenomics information extractor
nlpnatural language processing
oieopen information extraction
pospart-of-speech
svmsupport vector machine
utf-8universal transformation format-8

 in the biological sciences, describing species has long been a basic and essential component to studies of biodiversity, systematics, and evolution. scientists have published detailed descriptions of organisms for centuries, amassing a rich legacy of taxonomic literature that includes descriptions of phenotypic characters for millions of species, written in natural language. microbial descriptions of prokaryotes, in particular, also contain rich phenotypic information relating to metabolism and environmental niche space. with the genomic revolution, molecular data is often relatively easy to obtain, and integrating molecular and phenotypic information has proven to be an important approach to study the evolution of microbial traits on a broad scale. however, the majority of phenotypic information is currently locked in natural language descriptions and not readily usable for bioinformatic comparative analyses. in order to study the evolution of phenotypic traits on a large scale, i.e., covering the phenome —the full set of phenotypic characters—of an organism, character information needs to be in a standard structured format that can be used by existing phylogenetic comparative analysis software packages, such as mesquite and macclade. traditionally, taxon-by-character matrices are created manually by experts through reading and selecting phenotypic data in published taxonomic descriptions. this process is time-consuming and tedious. hence, most studies have focused on a small number of taxa, or a small number of characters. clearly, more efficient means of obtaining large amounts of phenotypic information for large numbers of taxa are required in order to develop a comprehensive synthesis of the distribution and evolution of diverse prokaryotic traits in the fields of microbial systematics and evolutionary biology. information extraction techniques from text mining can provide solutions to this problem by automatically extracting character states from natural language text. text mining has been used to find patterns from plain text within the biological and biomedical domains. examples include the extraction of text to annotate protein-protein interaction networks, metabolic pathways, gene regulatory networks, and signaling networks. within a text mining system, information extraction is an important task to extract knowledge elements from texts. the extraction targets are typically entities, the relationships among entities, and biological events. the techniques used to recognize the extraction targets may be categorized as employing 1) an existing knowledge base, 2) machine learning or computational statistics methods, and/or 3) natural language processing methods. these techniques are typically used in combination to achieve the best performance. the discovery of domain entities is often achieved using an existing knowledge base and enhanced with machine learning mechanisms. machine learning algorithms use statistical methods to model the problem of interest and make predictions based on the model. for example, support vector machine is a general-purpose, robust, supervised classifier based on mathematical theory of hyperplane arrangements that can be trained with tagged examples, then used to classify new examples. because of its robustness and wide applicability, svm is used in text mining for document classification, sentence classification, named entity recognition, and beyond. extensively used in information extraction, nlp techniques combine language theories and statistical methods to analyze syntactic structure of plain text, entity roles, and their relationships. to capture a variety of phenotypic characters from microbial descriptions, we applied multiple techniques, including svms and syntactic analyses. in biodiversity domains, significant strides have been made recently in the development of automated text mining and natural language processing approaches to extract morphological data from the taxonomic literature of plants and animals. one such example is charaparser developed for extracting morphological characters from semi-structured taxonomic descriptions written in a telegraphic style. initial tests, however, showed that charaparser performed poorly at extracting data from prokaryotic taxonomic descriptions because these descriptions contain little morphological data, but rather focus on physiological and ecological traits.table high-level categories and characters implemented in micropie strictly anaerobic. obligately anaerobic. colony morphology* metabolic end products and secondary metabolites* microbe-host interaction* metabolic substrate* example source sentences for each character within each category are provided. bolded text in the source sentences indicates the values that micropie should extract domain-independent information extraction systems have been reported, which include open information extraction and deepdive. oie extracts entity pairs connected through verbs so that some kinds of character values could be extracted; for example, “bacteria” will be returned as an appropriate answer for the question “what produces methane?” by oie from the sentence “these bacteria eat through practically anything and produce methane gas in the process.” however, although oie relies on textual context to extract information, it was not adequate for our task, which required substantial domain knowledge. deepdive provides statistical machine learning and inference support for building domain-specific, information extraction systems for knowledge base construction from texts. deepdive has been used successfully to build information extraction systems for specific domains, for example, pharmacogenomics. however, it is essentially an interactive system and requires considerable user-developer interactions and input from domain experts. algorithms/programs specifically designed to extract microbial-related characters also exist. humphreys et al. designed the empathie system based on domain concepts and various nlp techniques to extract enzyme and metabolic pathways. their system was evaluated on a small corpus with seven journal articles, and the performance was less optimal. czarnecki et al. proposed a text-mining system for extracting three types of metabolic reactions. the task was accomplished through two steps: entity recognition by existing text mining tools and reaction extraction by scoring the permutations of the recognized entities in sentences selected via heuristic/syntactic rules. the performance for these methods was poor to high, having recall scores ranging from to 82% and precision scores ranging from to 88%. these existing text mining systems could provide part of the solutions to phenotypic character extraction but were insufficient for extracting extensive, comprehensive microbial phenotypic traits needed to generate character matrices. ontologies are used not only as a source of knowledge in information extraction systems, they are also used as a way to standardize the results of information extraction. matching natural language phrases to ontology terms is often treated as a separate step/task after the task of extracting character information from text. for example, bacteria biotope task of the bionlp shared task includes a task of extracting bacteria and their habitat information, and a separate task of matching extracted entities to an ontology. this paper reports the development and evaluation of a software application, micropie which extracts microbial phenotypic characters from prokaryote descriptions and generates a character matrix. examples of characters include cell diameter, cell length, motility, and g + c content. the input to micropie is a set of prokaryotic taxonomic descriptions. an example output matrix is shown in fig.  in this paper we focus on character extraction from natural language descriptions. the standardization of the extracted characters using ontologies to facilitate data sharing and integration is discussed in future work.fig. an example input description to micropie, simplified from, used with permission fig. part of a hypothetical output matrix. the row in bold corresponds to the description in fig. 1 here, we describe the process of defining the extraction targets for micropie, its system architecture and character extraction methods, as well as its performance evaluation metrics. we then report the performance results of micropie with and without its character prediction component, and compare its performance to the performance of a group of undergraduate microbiology students. after discussions on system performance and algorithm refinements, we conclude the paper with a future development plan for micropie. methods extraction target identification and selection exploratory studies were first conducted to identify the characters that needed to be extracted. to broadly represent the diversity of text and traits in prokaryotic taxonomic descriptions, a corpus of descriptions was initially sampled from three evolutionarily distant groups. published taxonomic descriptions were obtained as pdf files from a variety of journals, including international journal of systematic and evolutionary microbiology, proceedings of the national academy of sciences of the united states of america, etc. each taxonomic description was semi-automatically transferred into a text file. pdf-to-text conversion and/or formatting errors were manually corrected so that the extracted text matched the original. the collected microbial taxonomic descriptions were then segmented to sentences using the stanford corenlp toolkit. two r packages implementing lsa and topic models were used to analyze the content of microbial taxonomic descriptions. this analysis identified topics as raw categories, such as g + c content, growth temperature, and cell size. these topics were then combined, resulting in a set of high-level categories that cover general traits of prokaryotes. consulting the taxonomic description corpus, the characters were specified under each category and a set of characters were defined as the extraction targets for micropie. micropie system architecture figure  shows the system architecture of micropie. text input is first converted into a simple xml format where publication metadata, taxon names and description paragraphs are wrapped in separate elements. next, the xml files are analyzed by micropie preprocessor, character predictor, character extractor, and matrix generator in sequence to produce a taxon-by-character matrix. micropie does not automatically detect/remove descriptions that are potentially repetitive or highly similar, because they could represent different taxon concepts, the subject of research for some potential users of micropie.fig. system architecture of micropie the micropie preprocessor component has two main steps: sentence splitting and sentence cleaning. in the sentence splitting step, the description paragraphs are split into sentences using stanford corenlp. in the sentence cleaning step, sentences are normalized by replacing predefined xml entities and characters that are not compatible with the utf- charset, and by removing non-alphanumeric characters. micropie preprocessor supplies the subsequent components with clean, normalized sentences to reduce potential syntactic parsing errors. the character predictor consists of a set of trained svm classifiers, one for each character. the function of the character predictor is to predict the characters a sentence describes by classifying the sentence into one or more of the character classes. the svm classifier had been trained with the sentences from the descriptions mentioned before, then semi-manually labelled with appropriate characters. for characters with fewer than ten training sentences, more sentences were added by searching google scholar. in total, training instances were generated. the number of training sentences per character ranged from tens to more than when training the svms for a character, training sentences labeled with the character were used as its positive training examples, and the same number of negative training examples were randomly selected from the remaining sentences. this is done to keep the positive and negative training examples balanced as svms are sensitive to unbalanced training examples. the trained svms achieved an average % precision and % recall in character prediction as evaluated via a 10-fold cross validation. the character predictor uses each of the trained libsvm classifiers to classify a sentence. for example, the sentence, “cells are –  μm wide and –  μm long”, may be classified to both cell width and cell length characters. after character prediction, sentences are passed to the appropriate character extractors to extract the target values. a variety of character extractor algorithms have been implemented in the character extractor component; and they can be categorized into two main groups: linguistic rules and term matching. since one sentence may be classified into multiple characters by the character predictor component, the sentences may be processed by multiple extractors. on the other hand, one extractor may be used for several character classes of similar nature. for example, cell diameter, cell width, and cell length are processed by the same extractor that is based on linguistic rules shown in fig.  the target values for the three characters share a similar linguistic pattern: the keywords that indicate the character follow the candidate numeric values within a 3-word window.fig. a shared extractor based on linguistic rules for the characters cell diameter, cell width, and cell length the results from these extractors are then post-processed using predefined heuristic rules to generate the final results. for example, the cell length value “ – microns” in fig.  was standardized as “ –  μm” in the post-processing step. the last component of micropie, matrix generator, creates a taxon-by-character matrix with the values extracted for the characters. multiple values for a character are separated by number signs. micropie character extraction methods linguistic rules the linguistic rules for a character were identified from its svm training examples when developing micropie. text segments are extracted as candidate character values if the text segments satisfy relevant linguistic rules. linguistic rules used in micropie include regular expressions, part-of-speech tag patterns and syntactic patterns. in some cases, multiple linguistic rules are often integrated in one character extractor to deal with varied textual expressions of a character. regular expressions are used to extract character values with clear lexical clues; for example, a minimum and maximum for the range of ph values at which a bacteria could grow can be extracted from “grows at ph 6–8”. regular expressions only exploit lexical patterns of the sentences. pos tag patterns, in contrast, also take the pos tags of the words into consideration. pos tag patterns often consist of pos tags and specific words. for example, one simple pattern “<cd > mol%” is used to extract the number with the pos tag “cd” followed by “mol%” as the value of g + c content. syntactic patterns used in micropie are the character-indicating syntactic dependencies identified from microbial descriptions. for example, if a sentence classified as fermentation substrate used contains the phrase “is produced from”, then this verb phrase often indicates fermentation related characters: noun phrase, or the subject of the verb phrase, is often a fermentation product, while noun phrase, or the object, is a fermentation substrate.fig. an example of a syntactic pattern used in micropie term matching the term matching approach references the knowledge base which consists of lists of terms that were created manually for of the characters, as well as terms from other ontologies such as the plant ontology and the human disease ontology. the knowledge base provides the domain knowledge micropie cannot gather from the text descriptions alone. for example, if it is known that “chains” is associated with the character cell relationships & aggregations, a value may be extracted from the sentence “some cells are arranged in short chains”. noun phrases of sentences are first obtained by applying heuristic rules with pos tag information. each term in the term list is used to formulate a regular expression. then, the regular expressions for a character are applied to a noun phrase to determine whether this phrase is a candidate value of the character. for some characters whose character states appear as in the term list without extra variations, the character values are extracted by checking whether the terms exist in the sentences. in addition, syntactic rules can be combined with term matching to disambiguate the characters of the phrases through restricting the dependency roles of the matched phrases in the sentence. for example, if the np in fig.  matches with a term in the term list of fermentation substrate used, the confidence in this phrase being the character value is further improved. it is important to note that use of term matching alone is not sufficient for the character extraction task, because these terms could occur in descriptions outside of the designated character classes, and extraction targets are often more complex than a single phrase. to overcome the insufficient coverage of the term lists for some characters, coordinated noun phrases e.g., “chloramphenicol, streptomycin and kanamycin”, are identified. if one phrase in the coordinated noun phrases is recognized as a character value, all phrases in the coordination are assumed to be the values for the character. micropie performance evaluation the performance of micropie was evaluated against a gold standard character matrix prepared by microbiologists on our team, and also compared to the performance of undergraduate microbiology students who manually extracted a subset of microbial phenotypic characters. development of the gold standard matrix taxonomic descriptions from international journal of systematic and evolutionary microbiology were used to create a gold standard matrix. the gsm contains the ground truth of the characters for each taxon. gsm creation was time-consuming due to the ambiguities in some taxonomic descriptions or the complexity of the characters and syntax. to accelerate the manual process, the trained svm classifiers were used to predict the characters in the sentences. any missed or incorrect predictions of the svm classifiers were manually identified and corrected. despite the use of the svm classifiers, the generation of the gsm took well over person-hours. in order to compare the character values in the gsm to machine-generated values from micropie, the values were formulated in a structured way. string-based values were formatted as “negation │ modifier │ main value” with the main value as the only required field. modifiers are phrases that refine the main value in terms of degree, extent, or frequently. for example, “not strictly anaerobic” is formatted as “not │ strictly │ anaerobic”. the format for characters with numerical values had additional fields, “negation │ modifier │ main value │ unit │ sub-character”, with the main value being the required field which could hold a single or range numeric value. the field sub-character is only used to indicate the kind of salt, for the characters of nacl minimum, nacl optimum or nacl maximum. for example, “2–3% nacl” is formatted as “2– │ % │ nacl”. of the taxonomic descriptions in the gsm, were used to optimize/test the character extractors, and the remaining were used for the evaluation of the system. micropie performance evaluation metrics micropie performance metrics were calculated by comparing the character matrix produced by micropie against the gsm. precision, recall, and f1-measure scores were adopted to evaluate micropie performance for each character. for each value within a character, a hit score was assigned denoting how well the extracted value matches the gsm value on a scale of 0– with being a perfect match, being no match and being a partial match. from these hit scores, p, r and f were calculated as follows: p=sumofhitscores#ofallextractedvalues r=sumofhitscores#ofvaluesingsm f1=2prp+r there were various scenarios to consider in scoring the match between an extracted and a gsm value. thus, we designed both a rigid method and a relaxed method to calculate hit scores which in turn produces rigid and relaxed p, r, and f scores. if all formatted fields of the two values matched exactly, both the rigid and the relaxed hit scores were. a score of was given if the main value was not extracted, a wrong value was extracted, or the negations did not match. if the main values and negations matched, but the other fields did not, the rigid hit score was and the relaxed hit score was. if the negations matched, and main values partially matched, both the rigid and relaxed hit scores were set to. for characters with multiple values, each of the values was compared individually with the values in gsm and the sum of the individual scores was the final hit score.table examples of how performance evaluation metrics were calculated rigid and relaxed hit scores measuring the match between extracted values and gold standard matrix values, illustrated with examples hit scores were computed using string matching. for imperfect matches, we carried out additional manual review using a designed user interface through amazon mechanical turk. this method allowed for the manual adjustment of relaxed scores, if warranted, since the automatic scoring algorithm biased the extracted values because it often assigned to partial matches. it also allowed, in some cases, for the correction of gsm values that were initially coded incorrectly, and for the rescoring of these values. additionally, we compared the sum of the relaxed hit scores to the sum of the total value occurrences in the gsm to calculate the % of values that micropie extracted successfully. comparison to a student-generated character matrix undergraduate microbiology students at university of southern maine and kenyon college created a matrix in their guided class projects for of the prokaryotic characters from a subset of the taxonomic descriptions in the relaxed evaluation scores for characters in the student-generated matrix compared to the gsm were calculated manually because the values in the student-generated matrix were formatted differently from the gsm. the statistical significance of the performance differences between the student-generated matrix and the micropie output was evaluated by using student's t-test comparing description-based relaxed_p, relaxed_r, and relaxed_f scores. impact of character predictor on micropie performance in micropie, the character predicator triages sentences into different character extractors. to assess the effects of the character predictor component on character extraction results, the same matrix generation task was run with and without character predictor. in the latter case, sentences were passed into character extractor from preprocessor directly. the performance differences in the evaluation metrics between micropie with/without character predictor were compared.  the performance of micropie with character predictor it took 47 min and 45 s for micropie to process the taxonomic descriptions used as evaluation dataset collected from international journal of systematic and evolutionary microbiology and output a matrix, running on a standard pc laptop running java. in contrast, it took over person-hours to create the gold standard matrix. table  shows the performance scores of micropie with the character predictor enabled. the number of values in the gsm and micropie matrix is shown for each character as well. micropie extracted total values, resulting in a total relaxed hit score of, out of the maximum total values of from the gsm. in other words, micropie extracted % of character values successfully. half of the characters had relaxed_f scores greater than. among those, the relaxed_f score for characters was greater than, the relaxed_f score for characters was between and, while the relaxed_f score for only characters was equal to or below. of all the characters, had the same rigid and relaxed scores, while the other characters had greater relaxed scores. thus, micropie extracted at least partially correct values for most characters.table performance of micropie with character predictor %g + c cell diameter cell length cell width cell relationships & aggregations gram stain type internal features motility pigment compounds nacl minimum nacl optimum nacl maximum ph minimum ph optimum ph maximum antibiotic sensitivity antibiotic resistant colony shape colony margin colony texture fermentation substrates not used in total 4098 4049 total relaxed hit scores abbreviations: superscript n numerical character, s string-based/categorical character. the characters with > = in relaxed_f score are shown in bold the performance of micropie without character predictor the performance scores for micropie with the character predictor component disabled were also calculated. without character predictor component choosing the appropriate character extractor, micropie applied all the character extractors to each sentence and extracted a total of values, % more than what was in gsm. compared to micropie with svm, the total relaxed hit score was increased by. at the same time, about incorrect values were also extracted, resulting in a decreased accuracy of %, % lower than the accuracy of micropie with svm. the performance differences in relaxed_p, relaxed_r and relaxed_f scores with and without character predictor are presented in fig.  micropie with svm achieved higher relaxed_p scores for characters and resulted in a slightly lower score for only character. micropie without svm achieved higher relaxed_r scores for characters. note that the differences in recall were smaller than they were in precision. the combined effect on micropie performance can be seen in fig. 6c in terms of relaxed_f scores: micropie with svm performed better for of the characters but performed worse for characters, albeit by a small margin.fig. the performance comparison between micropie with and without character predictor. a relaxed_p, b relaxed_r, and c relaxed_f scores comparison between micropie and student performance the performance scores of the student-generated matrix were compared to those of the micropie-generated matrix. micropie performed significantly better than undergraduate microbiology students. the total relaxed hit score of the micropie matrix was 123% higher than that of student-generated matrix. in addition, micropie consistently performed better than students on all the compared characters and in all metrics. student's t-tests on the description-based relaxed_p, relaxed_r and relaxed_f scores indicate statistically significant differences between micropie and student performances.table the performance comparison between the students and micropie on characters in taxonomic descriptions discussion micropie performance analysis characters with more than character values in the gsm produced relaxed_f scores greater than, indicating that micropie performed better when character values appeared more frequently in the gsm. these same characters also appeared more frequently in the corpus of taxonomic descriptions, providing abundant domain terms and robust linguistics rules during the development of micropie and hence resulting in a final algorithm that can more reliably extract the appropriate target values.fig. scatter plot showing the relationship between relaxed_f scores and frequency of character value occurrence in gsm for each of the characters. the axis of the number of gsm values is log-transformed the higher performance for the numerical characters reflects the effectiveness of various linguistic rules. linguistic rules, specifically syntactic patterns, also resulted in higher performance for some string-valued characters when useful patterns about the character values were well captured by the extractors; for example, antibiotic sensitivity and antibiotic resistant achieved high relax_f scores of and respectively. the term matching method achieved high extraction performance for some characters as well, such as gram stain type and colony shape. however, multiple factors could contribute to the failure of term matching method. the coverage of the term lists may be limited, terms may be included in the lists of multiple characters making it difficult to determine the correct character in a specific sentence, or terms may be only part of a value to be extracted. in addition, micropie attempts to extract a complete phrase as opposed to a single word, but this does not solve the cases where one phrase contains multiple values or where one value is expressed across phrase boundaries. learning the relationships within and between phrases will help address these problems. during the development of the gsm and evaluation of micropie, we experienced a fair amount of tension between the need to extract as much information as possible and the need to fit the values in the matrix meaningfully so the values of each character were comparable between taxa. this largely stemmed from the variations in description content, and sometimes the vagueness of the language, in taxonomic descriptions. for example, the following sentence,“growth occurs at −2 °c in marine broth. good growth occurs on ma at 1–25 °c. no growth occurs at 30 °c or higher”, is ambiguous with respect to the optimal growth temperature. although experts could judge “1–25 °c” as the optimal growth temperature, micropie identified “1–25 °c” as a temperature related growth condition, but did not output the value as the optimal. such ambiguity presents a challenge for micropie. thus, we designed micropie to adhere to the principle of “stay true to the original descriptions” and to not attempt to make a judgement that needs much expert knowledge. the effects of character predictor micropie with the svm character predictor showed significantly higher precision than micropie without character predictor. with highly accurate character prediction, sentences containing specific characters were triaged to their specific character extractors, rather than to all character extractors non-discriminatorily. the triage alleviated issues in term matching caused by terms that are shared by multiple characters. for example, the term “filamentous” appears in term lists of both cell shape and colony shape. thus, micropie without character predictor extracted this term as values of both characters, while character predictor utilized the information in the complete sentence to predict the correct character so that the correct term list was used. our results show that using svm resulted in increased relaxed_p scores for five characters with shared terms in the term lists: cell shape improved by, external features, internal features, colony shape, and colony texture. precision and recall scores are known to have an inverse relationship —higher precision often means lower recall—which is also manifested in this case with the slightly lower recalls for micropie with character predictor. characters showing the larger decreases in recall scores were the characters with values having low frequencies of occurrence and hence a few missed values resulted in much lower recall scores. for these characters, the insufficient training examples affected the svms ability to make correct predictions for new data. although more training examples were available for nacl minimum, aerophilicity, and inorganic substances not used, the character predictor predicted more than 10% of relevant sentences as irrelevant, resulting in decreased recall performance for these characters. this suggests that the training examples of these characters were not very good representatives of the test descriptions. despite the lower recall scores, the extraction accuracy and f scores suggest that svms contributed positively to micropie performance. comparison between micropie and student performance a detailed analysis of student’s matrix showed that their extraction errors may be attributed to students’ inability to understand the descriptions completely. their lower performance suggests that, lacking sufficient domain knowledge, novices with only some background knowledge are not capable of extracting microbial characters with high accuracy. the performance comparison between micropie and students highlights the challenges in this task and the need for high quality domain knowledge for computer algorithms to perform well and further improve. differentiating micropie with entity annotators many annotation tools match individual phrases in textual descriptions to ontology terms. such systems are “entity recognition” systems, where individual entities are recognized, but relationships among these entities are not identified. in contrast, a key function of micropie is to relate character values to appropriate characters. once the character values are extracted and related to their characters, they can then be matched to terms in ontologies using algorithms such as bioportal annotator algorithm. future work although the performance of current micropie is promising, we have identified several areas it can be further improved. new methods for detecting extraction boundaries and for constructing linguistic rules automatically need to be explored. one such candidate is unsupervised semantic parsing, which combines markov network and first-order logic to cluster terms with similar semantic roles based on their syntactic dependencies with surrounding text. with this technique, instead of handcrafting a rule to extract anything following the verb “hydrolyze”, the cluster of terms, “hydrolyse”, “hydrolyze” and “degrade”, will enable micropie to generate a more robust rule of extracting anything following any of the terms in the cluster. another way in which micropie could be improved would be to replace simple term lists with an ontology. the term lists in this study formed the foundation for a new microbial ontology, micro. micro contains classes of phenotypic and metabolic characters, assays, and culture media of prokaryotes, connecting terms to other ontologies using logical axioms. it also includes a large number of synonyms and alternate spellings for terms found in the corpus of taxonomic descriptions used in this study. micro would provide a richer knowledge base for micropie, and would assist in the disambiguation of competing characters. it would also provide a controlled vocabulary for characters and character states, and identify synonymous phrases that frequently occur in prokaryotic taxonomic descriptions. in addition, the logical axioms in the ontology would make it possible for higher-order knowledge to be automatically inferred from extracted characters. future versions of micropie will utilize micro and a few other ontologies such as the cell ontology and the ontology of units of measure to improve extraction performance and also standardize character output with ontology terms. in addition to a lot of new microbial phenotypic terms, micro has imported many terms relevant to our task from other ontologies, using it in micropie is the most logical way forward. a larger number of training sentences and expanded knowledge resources will also improve the performance of character extraction methods, especially for those characters with fewer occurrences in the current training sentences and the gsm. micropie could also be improved by expanding the variety of target characters for extraction. the current version of micropie does not extract characters such as microbial filterability, habitats, physiological tests, lipids, and hosts. micropie also could be enhanced by giving it the capability of discovering new characters from taxonomic descriptions, and by processing descriptions from different microbial taxa, e.g., cyanobacteria and eukaryotic phototrophs which follow the botanical code rather than the bacteriological code. although micropie was designed to extract data from semi-structured taxonomic descriptions, in future experiments, we intent to evaluate its performance on other description texts, such as less structured descriptions of prokaryotic taxa in pubmed or in online databases, and to research and expand its capability and robustness. the combination of data extracted from semi-structured descriptions and incorporation of the rich micro ontology will certainly be useful for micropie to tackle different description styles/formats.  we report the development and evaluation of micropie, a phenotypic information extraction system for prokaryotic taxonomic descriptions. we show that micropie greatly increases the speed by which phenotypic character information can be extracted from published taxonomic descriptions. evaluation against a hand-generated gold standard matrix showed that micropie performed well on over half of designated characters and achieved an overall accuracy of % and overall performance that was significantly better than the performance of undergraduate microbiology students. it was also shown that inclusion of the character predictor component using svm improved the overall performance of micropie. we therefore conclude that micropie has the potential to significantly accelerate the construction of phenotypic character matrices for prokaryotic taxa, removing a fundamental roadblock for the study of large-scale character evolution in microorganisms. we expect future improvements will make it more robust and effective. abbreviations g + cdna molar guanine and cytosine content gsmgold standard matrix lsalatent semantic analysis micropiemicrobial phenomics information extractor nlpnatural language processing oieopen information extraction pospart-of-speech svmsupport vector machine utf-8universal transformation format-8 
to quickly obtain knowledge on the distribution and evolution of microbial traits, an information extraction system needed to be developed to extract phenotypic characters from large numbers of taxonomic descriptions so they can be used as input to existing phylogenetic analysis software packages. inclusion of a character prediction component improved the overall performance of micropie, notably the precision. the performance of micropie was evaluated against a gold standard matrix and another student-made matrix. results show that, compared to the gold standard, micropie extracted characters with a relaxed f score >  and characters with relaxed f scores ranging between and. micropie is a natural language processing application that uses a robust supervised classification algorithm to identify characters from sentences in prokaryotic taxonomic descriptions, followed by a combination of algorithms applying linguistic rules with groups of known terms to extract characters as well as character states. micropie is a promising new tool for the rapid and efficient extraction of phenotypic character information from prokaryotic taxonomic descriptions. information extractionphenotypic data extractionprokaryotic taxonomic descriptionsmicrobial phenotypescharacter matricessupport vector machinemachine learningtext miningalgorithm evaluationnatural language processinghttp://dx.doi.org/ /100000001national science foundationdeb-1208685dbi-1147266cui hong issue-copyright-statement© the author 2016 the input to micropie is a set of taxonomic descriptions.
 the large-scale analysis of phenomic data in microbial bioinformatics has been hampered by the lack of tools to rapidly and accurately extract phenotypic data from existing legacy text in the field of microbiology. to quickly obtain knowledge on the distribution and evolution of microbial traits, an information extraction system needed to be developed to extract phenotypic characters from large numbers of taxonomic descriptions so they can be used as input to existing phylogenetic analysis software packages.  we report the development and evaluation of microbial phenomics information extractor. micropie is a natural language processing application that uses a robust supervised classification algorithm to identify characters from sentences in prokaryotic taxonomic descriptions, followed by a combination of algorithms applying linguistic rules with groups of known terms to extract characters as well as character states. the input to micropie is a set of taxonomic descriptions. the output is a taxon-by-character matrix—with taxa in the rows and a set of pre-defined characters in the columns. the performance of micropie was evaluated against a gold standard matrix and another student-made matrix. results show that, compared to the gold standard, micropie extracted characters with a relaxed f score >  and characters with relaxed f scores ranging between and. inclusion of a character prediction component improved the overall performance of micropie, notably the precision. evaluated against the same gold standard, micropie performed significantly better than the undergraduate students.  micropie is a promising new tool for the rapid and efficient extraction of phenotypic character information from prokaryotic taxonomic descriptions. however, further development, including incorporation of ontologies, will be necessary to improve the performance of the extraction for some character types. keywords information extractionphenotypic data extractionprokaryotic taxonomic descriptionsmicrobial phenotypescharacter matricessupport vector machinemachine learningtext miningalgorithm evaluationnatural language processinghttp://dx.doi.org/ /100000001national science foundationdeb-1208685dbi-1147266cui hong issue-copyright-statement© the author 2016 