BACKGROUND
the study of protein-protein interactions  is one of the most critical issues in life-science research for understanding the function of individual proteins and the organization of biological processes.
a plethora of biomedical literature that describes protein-protein interaction experiments by specifying individual interacting proteins and the corresponding interaction types exists.
since the vast majority of protein interaction information still exists in research articles, many efforts have been made to create protein interaction databases such as bind  <cit> , mint  <cit> , intact  <cit> , and dip  <cit> .
however, several constraints such as the problems of manual curation of a database, the rapid growth of the biomedical literature, and of newly discovered proteins, make it difficult for database curators to keep up with the published information  <cit> .
the biocreative  challenge is a community-wide effort to build an evaluation framework for assessing text mining systems in biological domains  <cit> .
ppi tasks were specially designed to study the detection of protein-protein interactions from literature, which have two subtasks in biocreative iii, act  and imt .
act is the task to choose relevant abstracts to ppis.
imt is the task to find experimental evidence of interacting protein pairs.
particularly, act is important since filtering ppi-relevant articles is a fundamental step for building annotation databases.
thus, high performance act systems can help reduce the curation burden at the initial curation stage.
various approaches have been proposed to extract ppi information from biomedical literature.
one popular method is to use predefined phrase patterns or to exploit co-occurrence of two protein names from text.
these methods, however, have inherent limitations because they only find predefined ppi patterns, and are not able to discover new patterns.
machine learning  techniques can discover new patterns not captured in a known trigger word list.
hence, ml approaches have gained popularity in recent years.
support vector machines  have been widely used, and demonstrated outstanding performance  <cit> .
naive bayes, k-nearest neighbor, decision trees, and neural networks have been alternatively used to extract ppi information  <cit> .
natural language processing  is a strategy utilizing linguistic features obtained from text, and also has been used for ppi extraction  <cit> , where ppi sentences are assumed to have unique grammatical structures.
however, the effectiveness of using parsing information has been little investigated at the article classification level.
here, we present the method and the results from our participation in the biocreative iii act competition  <cit> .
our main focus on this task was to explore the effectiveness of applying word and grammatical features for our supervised learning approach to ppi article classification.
it includes minimizing external knowledge other than training set such as templates or rule-based approaches developed on other tasks, and external databases, e.g., gene/protein dictionaries or full text information.
the proposed method combines nlp strategies with ml techniques to utilize both word and syntactic features from text.
to obtain gene names, articles are first tagged using a priority model  <cit> .
this step is essential because protein names are the most important words triggering ppi descriptions.
the gene-tagged articles are further analyzed to obtain word and syntactic features.
for word features, multi-words, sub-strings, and mesh  terms are applied for classifier input.
multi-word features are unigrams, bigrams, and trigrams of words.
sub-string features are sub-strings with n characters, which may help reduce the difference between distributions on training and test sets  <cit> .
mesh terms are also considered word features since mesh is a controlled vocabulary for indexing and searching biomedical literature  <cit> .
for syntactic features, the dependency relationships between words are mainly investigated.
by using a dependency parser  <cit> , a head word and a dependent word are determined as a two-word combination.
this combination increases the problem space by increasing the total number of features.
therefore, we anonymize the gene names in dependent word positions by replacing with a special tag, e.g., ‘ptnword’.
this process reduces the total number of features while leaving dependency information intact.
another aspect of features considered is to extract higher-order patterns by evaluating a set of feature combinations.
when the proposed system predicts a part of the training corpus incorrectly, each feature combination is evaluated by a sum of partial derivatives of the loss function terms on data points  <cit> .
this adds candidate features detected as potentially useful for the classification task.
the last step is to learn article classification based on the extracted word and syntactic features.
the constraint here was to minimize computational cost and processing time, but with reasonable classification performance.
to achieve this purpose, a large margin classifier with huber loss function  <cit>  was adopted.
figure  <dig> depicts the overview of the proposed approach.
although the current approach has much room for improvement, it produced the top-ranked performance among all submitted runs in the biocreative iii act task.
as a result, we found that, in our system pipeline, syntactic patterns along with word features can effectively help distinguish between ppi and non-ppi articles.
note that the only external resource we used for the task was gene name data for the priority model, so the learning was solely limited to the given training corpus, which was a series of biocreative datasets.
the paper is organized as follows.
in the next section, we describe the results of our submission on the biocreative act task.
this is followed by discussion and conclusions drawn from our experience in biocreative iii.
lastly, our methods employed are explained.
RESULTS
our goal for the act task is to develop a data-driven system with minimal external resources.
to achieve this goal, choosing the right corpus is critical, whereas available benchmark sets are very limited.
for this task, we collected gold standard sets from previous biocreative competitions in addition to the biocreative iii corpus.
the ppi article classification task has been a major topic since biocreative ii.
although the number of examples is still small, we assumed it was large enough to learn common positive and negative ppi patterns.
table  <dig> shows the corpus name and the number of positive and negative examples used for learning and testing.
biocreative ii , biocreative ii.
<dig> , and biocreative iii training data  were used as the training corpus for all submitted runs.
the biocreative iii development set was alternatively used to add more ppi information for training.
the development set is the articles selected from the same pool as the biocreative test set, hence it was also used to tune our system for official submission.
the final candidate set for training consists of  <dig>  positive and  <dig>  negative examples.
the test set includes  <dig> positive and  <dig>  negative examples, which is more imbalanced compared to training data.
this imbalance problem is discussed later in the discussion section.
biocreative ii, biocreative ii.
<dig>  biocreative iii training, and development sets were used as the training corpus for the act competition.
while the training corpus is balanced, the biocreative iii test set is an imbalanced set with the number of negative examples about six times higher than the number of positive examples.
hence, for the official submission, system parameters were tuned for the biocreative iii development set.
to assess the performance of submitted results, the biocreative iii competition relies on various performance measures, accuracy, specificity , sensitivity , f <dig> score, mcc  score, and auc ip/r .
however, we discuss official runs based on f <dig> score, mcc score, and auc ip/r.
f <dig> score and mcc score evaluate the performance of binary classification, and do not account for ranked results.
auc ip/r, on the other hand, measures the quality of ranked results.
accuracy is commonly used to evaluate classification performance, which counts true positives and true negatives against the total number of predictions.
but, in an unbalanced-class setting, accuracy does not successfully measure classification performance because if the number of true cases is strongly biased toward the negative class, e.g., accuracy is high simply by producing all negative predictions.
the f <dig> score provides a more balanced evaluation by averaging precision and recall.
the mcc score also fairly evaluates binary classification since it uses all four cases, tp , tn , fp , and fn .
in particular, it is known to be more stable in the unbalanced class cases  <cit> .
the f <dig> score and the mcc score are calculated as follows:
where  and .
unlike f <dig> and mcc scores, auc ip/r rather evaluates the performance of ranked results by considering precision rates for all recall points.
for ranking systems or search engines, the performance at high ranks is more important than overall ranking, hence auc ip/r is a good indicator of ranking-based performance.
in discussion, we instead use average precision for the ranking performance because it measures ranking performance in a more conservative way.
average precision is the average of the precisions at the ranks where relevant documents appear.
it corresponds to the non-interpolated auc p/r score.
it is generally a lower value than auc ip/r, but also emphasizes the higher ranks.
we submitted five runs for the act task, each using the same pipeline, but with different data and detailed feature sets .
for run  <dig> and run  <dig>  unigrams and bigrams were used as multi-word features.
dependency relations were used in original form after anonymizing dependent genes/proteins to ‘ptnword’.
the difference between run  <dig> and run  <dig> was use of the biocreative iii development set, which is also the difference between run  <dig> and run  <dig>  for run  <dig> and run  <dig>  word trigrams were added as features.
to reduce complexity and also to make various forms into a single one, all words in dependency relations were stemmed using the porter stemming algorithm  <cit> .
the stemming increases the probability of matching the same relation in different word forms.
in addition, feature selection was performed by removing features less frequent than four.
this feature selection prevents escalating the number of features by ignoring the least frequent patterns, which might be insignificant for ppi classification.
however, less frequent patterns may be very specific forms for describing ppis.
therefore, removing such patterns may result in a performance decrease.
run  <dig> used the same strategy as run  <dig>  but utilized higher-order feature combinations as introduced in background.
for higher-order features, only binary combinations between features were evaluated to better fit the training corpus.
the partial derivative threshold for this approach was empirically optimized for the biocreative iii development set.
our system was originally designed to give ranked results, rather than labels.
however, the system output was binarized by using the sign of the huber classifier output.
the training data used in official submissions includes all examples of previous biocreative ppi article tasks.
however, the biocreative iii development set was selectively added for training in different runs.
unigrams , bigrams , and trigrams  were used as multi-word features.
mesh feature is unigrams and bigrams from mesh terms.
for grammar relations , stemming was performed on run  <dig> through run  <dig>  feature cut was performed based on the frequency threshold four.
tp, fp, fn, and tn are true positive, false positive, false negative, and true negative, respectively.
mcc means matthews’ correlation coefficient measure.
auc ip/r means the area under the interpolated precision and recall curve.
f <dig> score and mcc evaluate the performance of binary classification.
auc ip/r evaluates system performance in terms of ranked results.
for the submitted runs, our intention for dealing with gene names was to handle each gene name as a single entity.
so, gene names having multiple words are not separable during parsing and the result is more precise gene anonymization.
however, we found afterwards that this was not applied for the official runs, i.e., gene names having multiple words were not treated as a unit.
table  <dig> shows the corrected performance for run  <dig> and run  <dig> by fixing this gene handling issue.
run 2’ and run 4’ are newly obtained results for run  <dig> and run  <dig> respectively.
for both cases, the number of true positives are increased, which results in higher f <dig>  mcc, and auc scores.
here, run 4’ has the best performance among all runs by increasing those scores up to 1%.
run 2’ and run 4’ are the corrected performance results for run  <dig> and run  <dig> respectively.
for the official runs, gene names consisting of more than a single word were not treated as a single entity.
only this issue was fixed for run 2’ and run 4’.
run  <dig> utilized binary feature combinations to capture higher-order relationships between features.
the performance in run  <dig> changed very little compared to run  <dig> and run  <dig>  which proves to be an unsuccessful attempt, and it is not as we expected.
for run  <dig>  we did not have time to analyze and optimize for the submission.
according to our post-workshop experiments, classification performance is very sensitive to higher-order feature combinations, and difficult to optimize.
for run  <dig>  we simply found a weight threshold which retained as many features as possible and yet increased performance for the biocreative development set.
that resulted in a total of  <dig>  features.
in the discussion, we further investigate the effect of higher-order features.
given the time available for the task, the submitted runs are obviously not fully optimized results.
we believe further improvement is possible based on the act development set and also the recently released gold standard test set.
but, we did not have sufficient time to investigate all the options for optimizing the current system with both datasets.
overtraining classification performance on the development set leads to an overfitting problem and decreased classification performance on the test set.
so, our tuning for submitted runs was centered rather on different data and feature combinations, not fine tuning for parameters and heuristic knowledge.
the performance produced by our system shows that the strategy of using both word and syntactic features in our classification framework is a good combination for the ppi article classification task.
discussion
article filtering with imbalanced classes
one main issue in the biocreative iii act competition is the imbalance problem between the number of positive and negative articles.
negative examples in the act development set are  <dig> % of the whole development set.
in the biocreative test set, the ratio goes up to  <dig> %.
however, the training corpus gathered from previous biocreative competitions is rather a balanced dataset.
to overcome this problem, we tried several approaches.
the popular method to solve the imbalance problem on training data is balancing the number of training examples by over- or under-sampling  <cit> .
this sampling technique can be utilized for the imbalance problem on test data.
for example, the training corpus can be reorganized by over-sampling non-ppi articles or under-sampling ppi articles.
another approach for addressing the imbalance issue is the careful selection of negative examples from unlabeled data as an additional training source.
this method is similar to active learning  <cit> .
also, cost-sensitive learning  <cit>  can be used along with an ensemble machine with multiple classifiers.
nevertheless, those attempts were not successful for the biocreative act task.
the performance drop with an imbalanced test set compared to a balanced one can be easily explained.
assuming there is a prediction system performing at 90% precision for balanced data, 10% of positive predictions are false positive cases.
if negative examples of the same kind are increased by a factor of six, false positive predictions are six times higher than in the former case.
that results in a precision drop to 60% from 90%.
this imbalance problem affects most of the performance scores except for accuracy.
accuracy can remain high because of dominant negative examples as explained in the results section.
in our system, the classification performance on training data exceeds 96% f <dig> score and 99% average precision.
but this cannot ensure high performance on unbalanced test data.
utilizing word and syntactic feature types
the best score is obtained by using both single words and grammar relations for all classifiers.
the used training data was biocreative ii, biocreative ii.
<dig>  and biocreative iii training corpora.
the performance was measured for the biocreative development set.
as shown in the table, adding word-word relationships to single-word features boosts up the performance by  <dig> % in naïve bayes classifiers.
for svm and huber classifiers, the improvement is less, however it shows that word dependency provides a positive effect for ppi article classification.
the huber classifier is the chosen approach for both data scalability and classification performance.
based on the performance comparison in table  <dig>  our huber approach produces the best average precision overall.
for the biocreative act task, possible feature candidates were tested and analyzed including both word and syntactic features.
as a result, five feature types were further selected for better classification.
table  <dig> shows the performance changes on the biocreative iii development set by varying those five feature types, gene anonymization, multi-words, sub-strings, mesh terms, and higher-order features.
the baseline performance is the result when run  <dig> settings are applied.
a row shows the evaluation results when all of the features without the feature type at that row are used.
since the higher-order features were not used in the run  <dig>  the features were rather added to the baseline in the last row of the table.
we tried several feature combinations of the five feature types, but it was difficult to understand what feature type contributes more than others.
hence, the performance table was drawn from those simple variations.
according to the table, removing each feature affects average precision and f <dig> score differently.
for average precision, mesh terms, gene anonymization, and sub-strings contribute positively, but for f <dig> score, gene anonymization contributes more.
however, the feature contribution differs greatly depending on methods used and parameters.
figure  <dig> shows the non-interpolated precision-recall curve performance on the biocreative iii test set.
the precision-recall curves present run  <dig> and the result with single word features alone in the same classification pipeline.
it is clearly seen that the word and syntactic feature types used in this paper improve the classification performance at most recall points.
the baseline performance is the result obtained from our system pipeline with the same setting used for run  <dig>  a row shows the evaluation results when a specific feature type is not used for the experiment.
however, the last row is the performance results when higher-order features are applied.
the system reaches top performance on the biocreative iii development set when baseline and higher-order features are both used, which is the setting in run  <dig>  however, higher-order features are not easy to tune.
more importantly, higher-order features do not provide the best result for the biocreative iii test set.
in the proposed approach, gene name detection is a critical component of the system since gene names are handled individually and gene anonymization is based on this gene detection.
during the biocreative iii period, we found some flaws of the priority model in detecting correct gene names.
therefore, current performance is also limited by this detection capability.
ranking system for ppi article classification
in a binary classification system, f <dig> and mcc scores are useful to evaluate system performance.
but, in a ranking system, top-ranking performance is more important than overall ranking.
auc ip/r and average precision are sensitive indicators for ranked results, and our system was basically tuned to achieve better average precision  for submitted results.
the best auc ip/r score we obtained from official results is  <dig> , whereas the average auc score of all participants is  <dig>  and the median auc score is  <dig> .
the precision-recall curves between our system and others also show significant differences in top-ranking results .
figure  <dig> depicts the precision-recall curve for run  <dig>  the precision is over 90% until reaching 22% recall.
another perspective of ranking performance is the precision at rank n .
for run  <dig>  p@ <dig>  p@ <dig>  and p@ <dig> are 94%, 92%, and 85%, respectively.
this shows that the proposed approach is effective for a ranking-based search system even though the overall performance is far from fully automating ppi article selection for annotation  <cit> .
CONCLUSIONS
in the paper, we present our system and its performance for the biocreative iii act competition.
our focus for the task was to develop a machine learning framework to effectively capture ppi articles from biomedical literature with minimal external resource use.
the main idea here is detecting gene names and utilizing word-to-word relationships for automatically learning unique ppi patterns.
the proposed approach identifies gene names by a priority model, and dependency relations are extracted by analyzing grammatical structures in sentences.
a large margin classifier using the huber loss function is used to learn from extracted word and syntactic features.
data scalability was also considered in selecting huber classifiers for expanding target data to the whole pubmed corpus in the future.
different feature types, including multi-words and grammar relations with stemming, and feature selection were exploited for submitted runs.
different training corpora were also used.
higher-order features were studied to see the possibility of automatic feature expansion.
through these studies, we found that syntactic features are useful at the article classification level as well as at the sentence classification level.
even though there is a limit to detection of correct gene names and the system is not optimized enough for the imbalanced nature of the dataset, the proposed system performs well in both binary classification performance and ppi ranking performance in all different data and feature combinations.
current classification performance was achieved by only using a data-driven model containing different types of machine learning techniques.
however, in the current setup, identifying gene names and analyzing dependency relationship are critical components, which need careful setup through utilizing ppi-related heuristic knowledge.
solving how many higher-order features may help for the ppi classification task is also a remaining issue.
as a fully automatic annotation tool, the state-of-the-art systems are still far from real-world use.
but, they can be utilized as support systems for manual curation.
in particular, based on the biocreative iii act performance, our system is already useful for ppi article search in a web environment.

 the study of protein-protein interactions is one of the most critical issues in life-science research for understanding the function of individual proteins and the organization of biological processes. a plethora of biomedical literature that describes protein-protein interaction experiments by specifying individual interacting proteins and the corresponding interaction types exists. since the vast majority of protein interaction information still exists in research articles, many efforts have been made to create protein interaction databases such as bind, mint, intact, and dip. however, several constraints such as the problems of manual curation of a database, the rapid growth of the biomedical literature, and of newly discovered proteins, make it difficult for database curators to keep up with the published information. the biocreative challenge is a community-wide effort to build an evaluation framework for assessing text mining systems in biological domains. ppi tasks were specially designed to study the detection of protein-protein interactions from literature, which have two subtasks in biocreative iii, act and imt. act is the task to choose relevant abstracts to ppis. imt is the task to find experimental evidence of interacting protein pairs. particularly, act is important since filtering ppi-relevant articles is a fundamental step for building annotation databases. thus, high performance act systems can help reduce the curation burden at the initial curation stage. various approaches have been proposed to extract ppi information from biomedical literature. one popular method is to use predefined phrase patterns or to exploit co-occurrence of two protein names from text. these methods, however, have inherent limitations because they only find predefined ppi patterns, and are not able to discover new patterns. machine learning techniques can discover new patterns not captured in a known trigger word list. hence, ml approaches have gained popularity in recent years. support vector machines have been widely used, and demonstrated outstanding performance. naive bayes, k-nearest neighbor, decision trees, and neural networks have been alternatively used to extract ppi information. natural language processing is a strategy utilizing linguistic features obtained from text, and also has been used for ppi extraction, where ppi sentences are assumed to have unique grammatical structures. however, the effectiveness of using parsing information has been little investigated at the article classification level. here, we present the method and the results from our participation in the biocreative iii act competition. our main focus on this task was to explore the effectiveness of applying word and grammatical features for our supervised learning approach to ppi article classification. it includes minimizing external knowledge other than training set such as templates or rule-based approaches developed on other tasks, and external databases, e.g., gene/protein dictionaries or full text information. the proposed method combines nlp strategies with ml techniques to utilize both word and syntactic features from text. to obtain gene names, articles are first tagged using a priority model. this step is essential because protein names are the most important words triggering ppi descriptions. the gene-tagged articles are further analyzed to obtain word and syntactic features. for word features, multi-words, sub-strings, and mesh terms are applied for classifier input. multi-word features are unigrams, bigrams, and trigrams of words. sub-string features are sub-strings with n characters, which may help reduce the difference between distributions on training and test sets. mesh terms are also considered word features since mesh is a controlled vocabulary for indexing and searching biomedical literature. for syntactic features, the dependency relationships between words are mainly investigated. by using a dependency parser, a head word and a dependent word are determined as a two-word combination. this combination increases the problem space by increasing the total number of features. therefore, we anonymize the gene names in dependent word positions by replacing with a special tag, e.g., ‘ptnword’. this process reduces the total number of features while leaving dependency information intact. another aspect of features considered is to extract higher-order patterns by evaluating a set of feature combinations. when the proposed system predicts a part of the training corpus incorrectly, each feature combination is evaluated by a sum of partial derivatives of the loss function terms on data points. this adds candidate features detected as potentially useful for the classification task. the last step is to learn article classification based on the extracted word and syntactic features. the constraint here was to minimize computational cost and processing time, but with reasonable classification performance. to achieve this purpose, a large margin classifier with huber loss function was adopted. figure depicts the overview of the proposed approach. although the current approach has much room for improvement, it produced the top-ranked performance among all submitted runs in the biocreative iii act task. as a result, we found that, in our system pipeline, syntactic patterns along with word features can effectively help distinguish between ppi and non-ppi articles. note that the only external resource we used for the task was gene name data for the priority model, so the learning was solely limited to the given training corpus, which was a series of biocreative datasets. the paper is organized as follows. in the next section, we describe the results of our submission on the biocreative act task. this is followed by discussion and conclusions drawn from our experience in biocreative iii. lastly, our methods employed are explained.  our goal for the act task is to develop a data-driven system with minimal external resources. to achieve this goal, choosing the right corpus is critical, whereas available benchmark sets are very limited. for this task, we collected gold standard sets from previous biocreative competitions in addition to the biocreative iii corpus. the ppi article classification task has been a major topic since biocreative ii. although the number of examples is still small, we assumed it was large enough to learn common positive and negative ppi patterns. table shows the corpus name and the number of positive and negative examples used for learning and testing. biocreative ii, biocreative ii., and biocreative iii training data were used as the training corpus for all submitted runs. the biocreative iii development set was alternatively used to add more ppi information for training. the development set is the articles selected from the same pool as the biocreative test set, hence it was also used to tune our system for official submission. the final candidate set for training consists of positive and negative examples. the test set includes positive and negative examples, which is more imbalanced compared to training data. this imbalance problem is discussed later in the discussion section. biocreative ii, biocreative ii. biocreative iii training, and development sets were used as the training corpus for the act competition. while the training corpus is balanced, the biocreative iii test set is an imbalanced set with the number of negative examples about six times higher than the number of positive examples. hence, for the official submission, system parameters were tuned for the biocreative iii development set. to assess the performance of submitted results, the biocreative iii competition relies on various performance measures, accuracy, specificity, sensitivity, f score, mcc score, and auc ip/r. however, we discuss official runs based on f score, mcc score, and auc ip/r. f score and mcc score evaluate the performance of binary classification, and do not account for ranked results. auc ip/r, on the other hand, measures the quality of ranked results. accuracy is commonly used to evaluate classification performance, which counts true positives and true negatives against the total number of predictions. but, in an unbalanced-class setting, accuracy does not successfully measure classification performance because if the number of true cases is strongly biased toward the negative class, e.g., accuracy is high simply by producing all negative predictions. the f score provides a more balanced evaluation by averaging precision and recall. the mcc score also fairly evaluates binary classification since it uses all four cases, tp, tn, fp, and fn. in particular, it is known to be more stable in the unbalanced class cases. the f score and the mcc score are calculated as follows: where and. unlike f and mcc scores, auc ip/r rather evaluates the performance of ranked results by considering precision rates for all recall points. for ranking systems or search engines, the performance at high ranks is more important than overall ranking, hence auc ip/r is a good indicator of ranking-based performance. in discussion, we instead use average precision for the ranking performance because it measures ranking performance in a more conservative way. average precision is the average of the precisions at the ranks where relevant documents appear. it corresponds to the non-interpolated auc p/r score. it is generally a lower value than auc ip/r, but also emphasizes the higher ranks. we submitted five runs for the act task, each using the same pipeline, but with different data and detailed feature sets. for run and run unigrams and bigrams were used as multi-word features. dependency relations were used in original form after anonymizing dependent genes/proteins to ‘ptnword’. the difference between run and run was use of the biocreative iii development set, which is also the difference between run and run for run and run word trigrams were added as features. to reduce complexity and also to make various forms into a single one, all words in dependency relations were stemmed using the porter stemming algorithm. the stemming increases the probability of matching the same relation in different word forms. in addition, feature selection was performed by removing features less frequent than four. this feature selection prevents escalating the number of features by ignoring the least frequent patterns, which might be insignificant for ppi classification. however, less frequent patterns may be very specific forms for describing ppis. therefore, removing such patterns may result in a performance decrease. run used the same strategy as run but utilized higher-order feature combinations as introduced in background. for higher-order features, only binary combinations between features were evaluated to better fit the training corpus. the partial derivative threshold for this approach was empirically optimized for the biocreative iii development set. our system was originally designed to give ranked results, rather than labels. however, the system output was binarized by using the sign of the huber classifier output. the training data used in official submissions includes all examples of previous biocreative ppi article tasks. however, the biocreative iii development set was selectively added for training in different runs. unigrams, bigrams, and trigrams were used as multi-word features. mesh feature is unigrams and bigrams from mesh terms. for grammar relations, stemming was performed on run through run feature cut was performed based on the frequency threshold four. tp, fp, fn, and tn are true positive, false positive, false negative, and true negative, respectively. mcc means matthews’ correlation coefficient measure. auc ip/r means the area under the interpolated precision and recall curve. f score and mcc evaluate the performance of binary classification. auc ip/r evaluates system performance in terms of ranked results. for the submitted runs, our intention for dealing with gene names was to handle each gene name as a single entity. so, gene names having multiple words are not separable during parsing and the result is more precise gene anonymization. however, we found afterwards that this was not applied for the official runs, i.e., gene names having multiple words were not treated as a unit. table shows the corrected performance for run and run by fixing this gene handling issue. run 2’ and run 4’ are newly obtained results for run and run respectively. for both cases, the number of true positives are increased, which results in higher f mcc, and auc scores. here, run 4’ has the best performance among all runs by increasing those scores up to 1%. run 2’ and run 4’ are the corrected performance results for run and run respectively. for the official runs, gene names consisting of more than a single word were not treated as a single entity. only this issue was fixed for run 2’ and run 4’. run utilized binary feature combinations to capture higher-order relationships between features. the performance in run changed very little compared to run and run which proves to be an unsuccessful attempt, and it is not as we expected. for run we did not have time to analyze and optimize for the submission. according to our post-workshop experiments, classification performance is very sensitive to higher-order feature combinations, and difficult to optimize. for run we simply found a weight threshold which retained as many features as possible and yet increased performance for the biocreative development set. that resulted in a total of features. in the discussion, we further investigate the effect of higher-order features. given the time available for the task, the submitted runs are obviously not fully optimized results. we believe further improvement is possible based on the act development set and also the recently released gold standard test set. but, we did not have sufficient time to investigate all the options for optimizing the current system with both datasets. overtraining classification performance on the development set leads to an overfitting problem and decreased classification performance on the test set. so, our tuning for submitted runs was centered rather on different data and feature combinations, not fine tuning for parameters and heuristic knowledge. the performance produced by our system shows that the strategy of using both word and syntactic features in our classification framework is a good combination for the ppi article classification task. discussion article filtering with imbalanced classes one main issue in the biocreative iii act competition is the imbalance problem between the number of positive and negative articles. negative examples in the act development set are % of the whole development set. in the biocreative test set, the ratio goes up to %. however, the training corpus gathered from previous biocreative competitions is rather a balanced dataset. to overcome this problem, we tried several approaches. the popular method to solve the imbalance problem on training data is balancing the number of training examples by over- or under-sampling. this sampling technique can be utilized for the imbalance problem on test data. for example, the training corpus can be reorganized by over-sampling non-ppi articles or under-sampling ppi articles. another approach for addressing the imbalance issue is the careful selection of negative examples from unlabeled data as an additional training source. this method is similar to active learning. also, cost-sensitive learning can be used along with an ensemble machine with multiple classifiers. nevertheless, those attempts were not successful for the biocreative act task. the performance drop with an imbalanced test set compared to a balanced one can be easily explained. assuming there is a prediction system performing at 90% precision for balanced data, 10% of positive predictions are false positive cases. if negative examples of the same kind are increased by a factor of six, false positive predictions are six times higher than in the former case. that results in a precision drop to 60% from 90%. this imbalance problem affects most of the performance scores except for accuracy. accuracy can remain high because of dominant negative examples as explained in the results section. in our system, the classification performance on training data exceeds 96% f score and 99% average precision. but this cannot ensure high performance on unbalanced test data. utilizing word and syntactic feature types the best score is obtained by using both single words and grammar relations for all classifiers. the used training data was biocreative ii, biocreative ii. and biocreative iii training corpora. the performance was measured for the biocreative development set. as shown in the table, adding word-word relationships to single-word features boosts up the performance by % in naïve bayes classifiers. for svm and huber classifiers, the improvement is less, however it shows that word dependency provides a positive effect for ppi article classification. the huber classifier is the chosen approach for both data scalability and classification performance. based on the performance comparison in table our huber approach produces the best average precision overall. for the biocreative act task, possible feature candidates were tested and analyzed including both word and syntactic features. as a result, five feature types were further selected for better classification. table shows the performance changes on the biocreative iii development set by varying those five feature types, gene anonymization, multi-words, sub-strings, mesh terms, and higher-order features. the baseline performance is the result when run settings are applied. a row shows the evaluation results when all of the features without the feature type at that row are used. since the higher-order features were not used in the run the features were rather added to the baseline in the last row of the table. we tried several feature combinations of the five feature types, but it was difficult to understand what feature type contributes more than others. hence, the performance table was drawn from those simple variations. according to the table, removing each feature affects average precision and f score differently. for average precision, mesh terms, gene anonymization, and sub-strings contribute positively, but for f score, gene anonymization contributes more. however, the feature contribution differs greatly depending on methods used and parameters. figure shows the non-interpolated precision-recall curve performance on the biocreative iii test set. the precision-recall curves present run and the result with single word features alone in the same classification pipeline. it is clearly seen that the word and syntactic feature types used in this paper improve the classification performance at most recall points. the baseline performance is the result obtained from our system pipeline with the same setting used for run a row shows the evaluation results when a specific feature type is not used for the experiment. however, the last row is the performance results when higher-order features are applied. the system reaches top performance on the biocreative iii development set when baseline and higher-order features are both used, which is the setting in run however, higher-order features are not easy to tune. more importantly, higher-order features do not provide the best result for the biocreative iii test set. in the proposed approach, gene name detection is a critical component of the system since gene names are handled individually and gene anonymization is based on this gene detection. during the biocreative iii period, we found some flaws of the priority model in detecting correct gene names. therefore, current performance is also limited by this detection capability. ranking system for ppi article classification in a binary classification system, f and mcc scores are useful to evaluate system performance. but, in a ranking system, top-ranking performance is more important than overall ranking. auc ip/r and average precision are sensitive indicators for ranked results, and our system was basically tuned to achieve better average precision for submitted results. the best auc ip/r score we obtained from official results is, whereas the average auc score of all participants is and the median auc score is. the precision-recall curves between our system and others also show significant differences in top-ranking results. figure depicts the precision-recall curve for run the precision is over 90% until reaching 22% recall. another perspective of ranking performance is the precision at rank n. for run p@ p@ and p@ are 94%, 92%, and 85%, respectively. this shows that the proposed approach is effective for a ranking-based search system even though the overall performance is far from fully automating ppi article selection for annotation.  in the paper, we present our system and its performance for the biocreative iii act competition. our focus for the task was to develop a machine learning framework to effectively capture ppi articles from biomedical literature with minimal external resource use. the main idea here is detecting gene names and utilizing word-to-word relationships for automatically learning unique ppi patterns. the proposed approach identifies gene names by a priority model, and dependency relations are extracted by analyzing grammatical structures in sentences. a large margin classifier using the huber loss function is used to learn from extracted word and syntactic features. data scalability was also considered in selecting huber classifiers for expanding target data to the whole pubmed corpus in the future. different feature types, including multi-words and grammar relations with stemming, and feature selection were exploited for submitted runs. different training corpora were also used. higher-order features were studied to see the possibility of automatic feature expansion. through these studies, we found that syntactic features are useful at the article classification level as well as at the sentence classification level. even though there is a limit to detection of correct gene names and the system is not optimized enough for the imbalanced nature of the dataset, the proposed system performs well in both binary classification performance and ppi ranking performance in all different data and feature combinations. current classification performance was achieved by only using a data-driven model containing different types of machine learning techniques. however, in the current setup, identifying gene names and analyzing dependency relationship are critical components, which need careful setup through utilizing ppi-related heuristic knowledge. solving how many higher-order features may help for the ppi classification task is also a remaining issue. as a fully automatic annotation tool, the state-of-the-art systems are still far from real-world use. but, they can be utilized as support systems for manual curation. in particular, based on the biocreative iii act performance, our system is already useful for ppi article search in a web environment. 
our approach utilizes both word and syntactic features to effectively capture ppi patterns from biomedical literature. identifying protein-protein interactions from literature is an important step in mining the function of individual proteins as well as their biological network. the proposed method automatically identifies gene names by a priority model, then extracts grammar relations using a dependency parser. while current performance is far from satisfactory as an annotation tool, it is already useful for a ppi article search engine since users are mainly focused on highly-ranked results. a large margin classifier with huber loss function learns from the extracted features, and unknown articles are predicted using this data-driven model. for the biocreative iii act evaluation, our official runs were ranked in top positions by obtaining maximum % accuracy, % f score, mcc score, and % auc ip/r score. our approach is valuable in showing how to use grammatical relations for ppi article filtering, in particular, with a limited training corpus.
 identifying protein-protein interactions from literature is an important step in mining the function of individual proteins as well as their biological network. since it is known that ppis have distinctive patterns in text, machine learning approaches have been successfully applied to mine these patterns. however, the complex nature of ppi description makes the extraction process difficult.  our approach utilizes both word and syntactic features to effectively capture ppi patterns from biomedical literature. the proposed method automatically identifies gene names by a priority model, then extracts grammar relations using a dependency parser. a large margin classifier with huber loss function learns from the extracted features, and unknown articles are predicted using this data-driven model. for the biocreative iii act evaluation, our official runs were ranked in top positions by obtaining maximum % accuracy, % f score, mcc score, and % auc ip/r score.  even though problems still remain, utilizing syntactic information for article-level filtering helps improve ppi ranking performance. the proposed system is a revision of previously developed algorithms in our group for the act evaluation. our approach is valuable in showing how to use grammatical relations for ppi article filtering, in particular, with a limited training corpus. while current performance is far from satisfactory as an annotation tool, it is already useful for a ppi article search engine since users are mainly focused on highly-ranked results. 13- september the third biocreative, critical assessment of information extraction in biology challenge bethesda, md, usa 