BACKGROUND
in the domains of biomedicine and genomics, scientific discoveries and empirical knowledge are often buried in the vast amount of research publications, doctors’ notes, and other forms of text.
this poses a challenge to scientists and medical practioners in accurately and efficiently locating specific pieces of information.
addressing this need, text mining technologies have shown promise in helping to accelerate the organisation and curation of biomedical literature  <cit> .
devoted to the evaluation of biomedical ie systems, biocreative challenges   <cit>  is a series of events that brings together computational linguists, computer scientists, and biomedical researchers to develop, test and exchange text mining ideas in the context of biomedical literature curation.
the extraction of protein protein interactions  has been one of the main topics in the biocreative series of workshops.
although techniques for mining ppis have improved in recent years  <cit> , the need for associating evidence and attributes to ppis so they could easily be processed and linked together, either by human users or automated systems, has yet to be sufficiently addressed.
for example, experimental techniques applied for discovering ppis, such as yeast two-hybrid screening and anti tag coimmunoprecipitation, are important for understanding the findings and for validating and reproducing the results.
biocreative ii’s interaction method subtask   <cit>  was designed to tackle this challenge.
two teams  <cit>  participated in the task and the best f <dig> score was reported at a level of 45%, obtained by matching the document text against a set of variants of the interaction method terms in the ontology provided.
along similar goals established in biocreative ii, the biocreative iii interaction method task  is particularly directed towards the development of automated systems that detect the techniques used in experiments to confirm a given ppi from full-text research articles.
the permitted set of interaction methods comes from the psi-mi ontology  <cit>  and consists of a subset of  <dig> definitions of experimental techniques.
in this task each article may be associated with zero or more methods.
therefore, the task can be cast as a multi-class, multi-label document classification problem.
the interaction methods from pmi-mi are represented by their unique identifiers, henceforth referred to as mi ids.
imt bears some resemblance to an entity normalisation task , in that for each given document, if the terms describing interaction methods were recognised and grounded to mi ids, the task would be solved.
therefore, we also approached the task as term normalisation, using string similarity measures, and then training a model to filter out false positives.
the aim of the article classification task  is to categorise documents as being relevant or irrelevant to ppi curation.
act addresses a time-consuming but essential task in a typical manual curation work-flow: a curator spends a significant amount of time scanning through a paper to establish whether the document in question contains curatable ppis.
according to the task specification, only documents reporting ppis are considered relevant, while those describing interactions between genes or other non-protein biological entities are not.
tasks similar to act have attracted much research attention, most notably through interaction articles subtask  in biocreative ii  <cit>  and act in biocreative ii.
<dig>  <cit> .
both abstracts and full-text papers were provided in biocreative iii’s act, but only information from the abstracts were considered during evaluation.
we cast act as a binary document classification task, and our strategy was to exploit a rich set of features including linguistic information and named entities that were automatically annotated using text mining systems.
RESULTS
data
the organisers of the challenge provided three datasets for each task, referred to as training, development, and test.
the former two datasets were intended to be used in the process of developing the systems by the tasks’ participants.
once the systems were ready, the participants were asked to submit the results on the test dataset, for which the annotation was  unknown.
table  <dig> shows the distribution of articles in each dataset for each task.
we used the psi-mi ontology  <cit>  provided by the european bioinformatics institute  for grounding experimental techniques to mi ids.
distribution of articles in the training, development, and test datasets provided by the biocreative iii organisers for each of the ppi tasks.
detecting interaction methods
we devised three distinct systems for imt, two of which followed the more common multi-class, multi-label document classification framework, and the other one used a binary model to classify pairs of method names and text phrases.
we experimented with two machine-learning paradigms, namely logistic regression  and support vector machines .
svm was chosen because it has been shown to be superior to other commonly used machine learning methods for classification  <cit> .
specifically, svm demonstrated good performance on a range of tasks in biomedical text mining, including but not limited to, document classification  <cit> , named entity recognition  <cit> , named entity disambiguation  <cit> , relation extraction  <cit>  and bio-event extraction  <cit> .
lr is one of the most widely-used probabilistic modeling techniques in the field of natural language processing and has also been shown to be effective in handling large-scale classification problems  <cit> .
the applications range from simple classification tasks such as text categorisation  <cit>  to more complex structured prediction tasks such as part-of-speech  tagging  <cit>  and named entity recognition  <cit> .
lr models produce probabilistic output, which allows the information on the confidence of the decision to be used by subsequent components in the text processing pipeline, and this is an advantage over other discriminative machine learning models such as svm.
in our experiments, both svm and lr were used in the multi-class, multi-label classification scenario, whereas the binary model utilised svm only.
we refer to the three models as m-lr, m-svm, and b-svm, respectively, and we elaborate on these systems in section methods.
we performed cross-validation experiments on the training dataset, based on which the best models were selected and further tested on the development dataset.
for each document, we also took the union and the intersection of sets of mi ids obtained from the aforementioned approaches.
the results are shown in table  <dig>  as expected, the union of results improved recall and the intersection improved precision.
based on the results, we chose to submit the systems marked with asterisks , for which we trained models on the combined training and development datasets, and then applied the models on the test dataset.
macro-averaged results on the imt development dataset with  <dig> best models selected by cross-validation on the training data .
m-lr – multi-label logistic regression; m-svm – multi-label support vector machines; b-svm – binary support vector machines.
asterisks  denote systems that were submitted to the challenge.
results on the imt test dataset .
the models were trained on the combined training and development datasets.
m-lr – multi-label logistic regression; m-svm – multi-label support vector machines; b-svm – binary support vector machines.
asterisks  denote systems that were submitted to the challenge.
to investigate whether there was an overfitting problem, we plotted learning curves for the better performing systems as tested on the development data: m-svm, b-svm, and the union and the intersection of the two systems.
figure  <dig> consists of  <dig> graphs, each plotting learning curves of the  <dig> systems using the following evaluation metrics: precision, recall, f <dig> score and auc ip/r.
each system was trained on increasing amounts of data, randomly taking 20%, 40%, 60%, 80%, and finally the entire set of the training data; whereas the performance was recorded on the development dataset.
as shown in the figure, the system that takes the union of the results obtained from b-svm and m-svm excels in most of the cases.
also, according to all measures except recall, performances went up as the size of training data increased, indicating overfitting did not occur on the training and development datasets.
however, the performance decreased when the models were trained on the combined training and development datasets, and tested on the test dataset.
alongside the above analysis, it suggests that the distributions of mi ids are likely to be similar between the training and development datasets, but different between the training and test datasets.
figure  <dig> shows the histograms of the  <dig> most frequent mi ids as found in the training dataset, together with their corresponding frequencies in the development and test datasets.
the figure reveals that, in the training and development datasets, the  <dig> most frequently occurring mi ids and their order are identical, which in both datasets account for over 50% of all occurrences of mi ids.
in the test dataset, on the other hand, the frequencies of these ids are distributed differently.
as a consequence, it was likely that b-svm was overfitted to the training and development datasets.
indeed, the different class distribution affects b-svm more than m-svm and m-lr, because some features used by b-svm are tuned to the distributions of the mi ids in the training data, e.g., the features that map the frequent mi ids and their mesh equivalents.
in summary, under the current experiment settings, the multi-classification approaches showed better adaptability to a diverse range of datasets, while b-svm needs to minimise its dependence on training data by feature engineering or algorithm tuning, in order to achieve better portability.
to gain insight on how the systems could possibly be improved in the future, we performed manual error analysis following the steps listed below:
<dig>  based on the performance results on the development set, a set of most frequently occurring mi ids for which all systems produced incorrect predictions was extracted.
this step showed that all systems performed poorly for the following mi ids: mi: <dig>  mi: <dig>  mi: <dig>  mi: <dig> and mi: <dig>
<dig>  focusing on, but not limiting to, the list of mi ids from the first step, we randomly selected  <dig> document - mi id pairs from the list of incorrect predictions .
we analysed the corresponding full-text articles and gold standard annotations, manually looking for information which could have been missed  or wrongly interpreted  by our systems.
our findings showed that the errors can be attributed to: 1) more challenging aspects of the problem which our systems have not addressed yet; and 2) some inconsistent annotations in the training and development sets.
most of the false positives resulted from recognising mentions of method names which occurred in the text but not in the description of how a protein interaction was established.
for example, some authors, in describing their failed attempts to detect protein interactions, mention the methods they have used.
this resulted in a number of false positives, as the task requires that only methods which support an established interaction are to be considered.
this implies that it is necessary for a system to consider only mentions of methods in the context of an established protein interactions which, in turn, suggests that perhaps protein interactions need to be extracted beforehand.
on the other hand, several inconsistent annotations were found both in the training and development datasets.
for example, the documents with pmids  <dig>   <dig>   <dig> and  <dig> which describe “confocal microscopy” were not annotated with mi: <dig> .
having been trained on such data, the systems may have failed to learn the association between the concept and the documents which describe it.
this possibly resulted in the poor recall on the development set with respect to a few mi ids such as mi: <dig>
in addition, while the annotation guidelines seem to specify that the most specific possible mi concept should be assigned to a document, we observed that it was not always the case.
for instance, the document pmid  <dig> had been annotated with only mi: <dig> ; a review of the article, however, reveals that the methodology used in detecting the protein interactions is “anti tag coimmunoprecipitation” which is an indirect subtype of affinity chromatography in the psi-mi ontology.
we also found that some documents contain incorrect annotations.
for example, the document pmid  <dig> was annotated with mi: <dig>  even though this method was not mentioned in the main text, but only in the supplementary material which was not included in the provided data set.
in conclusion, although we were unable to systematically measure how much the annotation errors affected the systems’ results, we can speculate that an improvement to annotation consistency would result in better and more reliable models.
selecting articles relevant to ppi curation
for act, we experimented with both svm and lr, as well as different types of mesh information used as features.
the first type of mesh feature was based purely on the unique identifier of a mesh term , whereas the second one was based on a more elaborate, tree-structure representation.
in the course of experiments we eventually focused on three distinct systems: two based on svm and one based on lr.
the detailed description of the systems is provided in section methods.
the performance of the three systems tested on the test dataset is given in table 4; while table  <dig> shows 10-fold cross-validation results on the combined training and development datasets.
results on the act test dataset .
svmmesh id – svm with mesh identifiers; svmmesh tree – svm with mesh tree structure; lr – lr with mesh identifiers.
the 10-fold cross-validation results for act on the training and development datasets .
svmmesh id – svm with mesh identifiers; svmmesh tree – svm with mesh tree structure; lr – lr with mesh identifiers; svmmesh id&tree – svm with both mesh identifiers and tree structure.
the cross validation experiment results formed the basis for choosing models for testing, and the experiment was carried out on the combined set because the number of training instances for act was much smaller than the corresponding development set , and joining the two datasets for cross validation resulted in a larger amount of training data, leading to better classification models.
table  <dig> shows that the lr system obtained better results as measured by auc ip/r than the svm ones, which indicates that lr produced better ranking than svm.
in addition, comparing tables  <dig> and  <dig>  the results are lower on the test dataset than on the combined, cross-validated training and development datasets.
one reason may be the difference in the ratios of the amounts of training and test datasets used: in the 10-fold cross-validation experiments , the size of the training data was  <dig> times more than that of the test data; whereas in the other set of experiments , the two datasets were of similar size .
as using a larger amount of test data tends to produce more stably estimated performance, table  <dig> reveals that the models prepared using the combined training and development datasets are still not satisfactory in providing good coverage.
results on the act development dataset with models trained on the training dataset .
svmmesh id – svm with mesh identifiers; svmmesh tree – svm with mesh tree structure; lr – lr with mesh identifiers.
similarly to imt, we also conducted an experiment to see how the size of training data affects the models, the results of which are shown in figure  <dig>  the size of the data was gradually increased from 10% to 100%, where data at each percentage mark was randomly selected from the combined training and development datasets.
we then performed 10-fold cross validations on the selected data.
the figure shows that the performance of the svm system was rather indifferent to the size of data used to train the model.
the lr system requires more training data as the curve is steep up to the 30% mark in f <dig> score, and then the slope gradually decreases as the size of data increases.
CONCLUSIONS
we compared several approaches to the biocreative iii imt and act tasks.
for imt, we proposed a new method that first searches for candidate interaction method text strings in documents, and then classifies pair-wise relations between the candidates and their matching interaction method names, as defined in psi-mi.
this method utilises a rich set of features extracted from the candidates’ surrounding context, together with the definitions and synonyms in psi-mi.
evaluation results on the development dataset show that, overall, this method is promising and outperforms the more conventional multi-label document classification using the “one-vs-all” strategy.
however, its superiority was not confirmed on the test dataset, and the variance indicates that the model may be overfitted to the training and development data.
we also tested simple ensemble systems using heuristic rules of union and intersection to combine methods, and achieved very good overall performance on both test datasets, which provided evidence that the systems complement each other.
for act, we tested lr and svm classifiers exploiting a rich set of features obtained by linguistic processing and automatic recognition of a wide range of biomedical named entities.
a series of feature-knockout experiments  showed that the discriminative power is drawn mainly from contextual words surrounding automatically recognized named entities, as well as mesh.

 in the domains of biomedicine and genomics, scientific discoveries and empirical knowledge are often buried in the vast amount of research publications, doctors’ notes, and other forms of text. this poses a challenge to scientists and medical practioners in accurately and efficiently locating specific pieces of information. addressing this need, text mining technologies have shown promise in helping to accelerate the organisation and curation of biomedical literature. devoted to the evaluation of biomedical ie systems, biocreative challenges is a series of events that brings together computational linguists, computer scientists, and biomedical researchers to develop, test and exchange text mining ideas in the context of biomedical literature curation. the extraction of protein protein interactions has been one of the main topics in the biocreative series of workshops. although techniques for mining ppis have improved in recent years, the need for associating evidence and attributes to ppis so they could easily be processed and linked together, either by human users or automated systems, has yet to be sufficiently addressed. for example, experimental techniques applied for discovering ppis, such as yeast two-hybrid screening and anti tag coimmunoprecipitation, are important for understanding the findings and for validating and reproducing the results. biocreative ii’s interaction method subtask was designed to tackle this challenge. two teams participated in the task and the best f score was reported at a level of 45%, obtained by matching the document text against a set of variants of the interaction method terms in the ontology provided. along similar goals established in biocreative ii, the biocreative iii interaction method task is particularly directed towards the development of automated systems that detect the techniques used in experiments to confirm a given ppi from full-text research articles. the permitted set of interaction methods comes from the psi-mi ontology and consists of a subset of definitions of experimental techniques. in this task each article may be associated with zero or more methods. therefore, the task can be cast as a multi-class, multi-label document classification problem. the interaction methods from pmi-mi are represented by their unique identifiers, henceforth referred to as mi ids. imt bears some resemblance to an entity normalisation task, in that for each given document, if the terms describing interaction methods were recognised and grounded to mi ids, the task would be solved. therefore, we also approached the task as term normalisation, using string similarity measures, and then training a model to filter out false positives. the aim of the article classification task is to categorise documents as being relevant or irrelevant to ppi curation. act addresses a time-consuming but essential task in a typical manual curation work-flow: a curator spends a significant amount of time scanning through a paper to establish whether the document in question contains curatable ppis. according to the task specification, only documents reporting ppis are considered relevant, while those describing interactions between genes or other non-protein biological entities are not. tasks similar to act have attracted much research attention, most notably through interaction articles subtask in biocreative ii and act in biocreative ii.. both abstracts and full-text papers were provided in biocreative iii’s act, but only information from the abstracts were considered during evaluation. we cast act as a binary document classification task, and our strategy was to exploit a rich set of features including linguistic information and named entities that were automatically annotated using text mining systems.  data the organisers of the challenge provided three datasets for each task, referred to as training, development, and test. the former two datasets were intended to be used in the process of developing the systems by the tasks’ participants. once the systems were ready, the participants were asked to submit the results on the test dataset, for which the annotation was unknown. table shows the distribution of articles in each dataset for each task. we used the psi-mi ontology provided by the european bioinformatics institute for grounding experimental techniques to mi ids. distribution of articles in the training, development, and test datasets provided by the biocreative iii organisers for each of the ppi tasks. detecting interaction methods we devised three distinct systems for imt, two of which followed the more common multi-class, multi-label document classification framework, and the other one used a binary model to classify pairs of method names and text phrases. we experimented with two machine-learning paradigms, namely logistic regression and support vector machines. svm was chosen because it has been shown to be superior to other commonly used machine learning methods for classification. specifically, svm demonstrated good performance on a range of tasks in biomedical text mining, including but not limited to, document classification, named entity recognition, named entity disambiguation, relation extraction and bio-event extraction. lr is one of the most widely-used probabilistic modeling techniques in the field of natural language processing and has also been shown to be effective in handling large-scale classification problems. the applications range from simple classification tasks such as text categorisation to more complex structured prediction tasks such as part-of-speech tagging and named entity recognition. lr models produce probabilistic output, which allows the information on the confidence of the decision to be used by subsequent components in the text processing pipeline, and this is an advantage over other discriminative machine learning models such as svm. in our experiments, both svm and lr were used in the multi-class, multi-label classification scenario, whereas the binary model utilised svm only. we refer to the three models as m-lr, m-svm, and b-svm, respectively, and we elaborate on these systems in section methods. we performed cross-validation experiments on the training dataset, based on which the best models were selected and further tested on the development dataset. for each document, we also took the union and the intersection of sets of mi ids obtained from the aforementioned approaches. the results are shown in table as expected, the union of results improved recall and the intersection improved precision. based on the results, we chose to submit the systems marked with asterisks, for which we trained models on the combined training and development datasets, and then applied the models on the test dataset. macro-averaged results on the imt development dataset with best models selected by cross-validation on the training data. m-lr – multi-label logistic regression; m-svm – multi-label support vector machines; b-svm – binary support vector machines. asterisks denote systems that were submitted to the challenge. results on the imt test dataset. the models were trained on the combined training and development datasets. m-lr – multi-label logistic regression; m-svm – multi-label support vector machines; b-svm – binary support vector machines. asterisks denote systems that were submitted to the challenge. to investigate whether there was an overfitting problem, we plotted learning curves for the better performing systems as tested on the development data: m-svm, b-svm, and the union and the intersection of the two systems. figure consists of graphs, each plotting learning curves of the systems using the following evaluation metrics: precision, recall, f score and auc ip/r. each system was trained on increasing amounts of data, randomly taking 20%, 40%, 60%, 80%, and finally the entire set of the training data; whereas the performance was recorded on the development dataset. as shown in the figure, the system that takes the union of the results obtained from b-svm and m-svm excels in most of the cases. also, according to all measures except recall, performances went up as the size of training data increased, indicating overfitting did not occur on the training and development datasets. however, the performance decreased when the models were trained on the combined training and development datasets, and tested on the test dataset. alongside the above analysis, it suggests that the distributions of mi ids are likely to be similar between the training and development datasets, but different between the training and test datasets. figure shows the histograms of the most frequent mi ids as found in the training dataset, together with their corresponding frequencies in the development and test datasets. the figure reveals that, in the training and development datasets, the most frequently occurring mi ids and their order are identical, which in both datasets account for over 50% of all occurrences of mi ids. in the test dataset, on the other hand, the frequencies of these ids are distributed differently. as a consequence, it was likely that b-svm was overfitted to the training and development datasets. indeed, the different class distribution affects b-svm more than m-svm and m-lr, because some features used by b-svm are tuned to the distributions of the mi ids in the training data, e.g., the features that map the frequent mi ids and their mesh equivalents. in summary, under the current experiment settings, the multi-classification approaches showed better adaptability to a diverse range of datasets, while b-svm needs to minimise its dependence on training data by feature engineering or algorithm tuning, in order to achieve better portability. to gain insight on how the systems could possibly be improved in the future, we performed manual error analysis following the steps listed below: based on the performance results on the development set, a set of most frequently occurring mi ids for which all systems produced incorrect predictions was extracted. this step showed that all systems performed poorly for the following mi ids: mi: mi: mi: mi: and mi: focusing on, but not limiting to, the list of mi ids from the first step, we randomly selected document - mi id pairs from the list of incorrect predictions. we analysed the corresponding full-text articles and gold standard annotations, manually looking for information which could have been missed or wrongly interpreted by our systems. our findings showed that the errors can be attributed to: 1) more challenging aspects of the problem which our systems have not addressed yet; and 2) some inconsistent annotations in the training and development sets. most of the false positives resulted from recognising mentions of method names which occurred in the text but not in the description of how a protein interaction was established. for example, some authors, in describing their failed attempts to detect protein interactions, mention the methods they have used. this resulted in a number of false positives, as the task requires that only methods which support an established interaction are to be considered. this implies that it is necessary for a system to consider only mentions of methods in the context of an established protein interactions which, in turn, suggests that perhaps protein interactions need to be extracted beforehand. on the other hand, several inconsistent annotations were found both in the training and development datasets. for example, the documents with pmids and which describe “confocal microscopy” were not annotated with mi:. having been trained on such data, the systems may have failed to learn the association between the concept and the documents which describe it. this possibly resulted in the poor recall on the development set with respect to a few mi ids such as mi: in addition, while the annotation guidelines seem to specify that the most specific possible mi concept should be assigned to a document, we observed that it was not always the case. for instance, the document pmid had been annotated with only mi:; a review of the article, however, reveals that the methodology used in detecting the protein interactions is “anti tag coimmunoprecipitation” which is an indirect subtype of affinity chromatography in the psi-mi ontology. we also found that some documents contain incorrect annotations. for example, the document pmid was annotated with mi: even though this method was not mentioned in the main text, but only in the supplementary material which was not included in the provided data set. in conclusion, although we were unable to systematically measure how much the annotation errors affected the systems’ results, we can speculate that an improvement to annotation consistency would result in better and more reliable models. selecting articles relevant to ppi curation for act, we experimented with both svm and lr, as well as different types of mesh information used as features. the first type of mesh feature was based purely on the unique identifier of a mesh term, whereas the second one was based on a more elaborate, tree-structure representation. in the course of experiments we eventually focused on three distinct systems: two based on svm and one based on lr. the detailed description of the systems is provided in section methods. the performance of the three systems tested on the test dataset is given in table 4; while table shows 10-fold cross-validation results on the combined training and development datasets. results on the act test dataset. svmmesh id – svm with mesh identifiers; svmmesh tree – svm with mesh tree structure; lr – lr with mesh identifiers. the 10-fold cross-validation results for act on the training and development datasets. svmmesh id – svm with mesh identifiers; svmmesh tree – svm with mesh tree structure; lr – lr with mesh identifiers; svmmesh id&tree – svm with both mesh identifiers and tree structure. the cross validation experiment results formed the basis for choosing models for testing, and the experiment was carried out on the combined set because the number of training instances for act was much smaller than the corresponding development set, and joining the two datasets for cross validation resulted in a larger amount of training data, leading to better classification models. table shows that the lr system obtained better results as measured by auc ip/r than the svm ones, which indicates that lr produced better ranking than svm. in addition, comparing tables and the results are lower on the test dataset than on the combined, cross-validated training and development datasets. one reason may be the difference in the ratios of the amounts of training and test datasets used: in the 10-fold cross-validation experiments, the size of the training data was times more than that of the test data; whereas in the other set of experiments, the two datasets were of similar size. as using a larger amount of test data tends to produce more stably estimated performance, table reveals that the models prepared using the combined training and development datasets are still not satisfactory in providing good coverage. results on the act development dataset with models trained on the training dataset. svmmesh id – svm with mesh identifiers; svmmesh tree – svm with mesh tree structure; lr – lr with mesh identifiers. similarly to imt, we also conducted an experiment to see how the size of training data affects the models, the results of which are shown in figure the size of the data was gradually increased from 10% to 100%, where data at each percentage mark was randomly selected from the combined training and development datasets. we then performed 10-fold cross validations on the selected data. the figure shows that the performance of the svm system was rather indifferent to the size of data used to train the model. the lr system requires more training data as the curve is steep up to the 30% mark in f score, and then the slope gradually decreases as the size of data increases.  we compared several approaches to the biocreative iii imt and act tasks. for imt, we proposed a new method that first searches for candidate interaction method text strings in documents, and then classifies pair-wise relations between the candidates and their matching interaction method names, as defined in psi-mi. this method utilises a rich set of features extracted from the candidates’ surrounding context, together with the definitions and synonyms in psi-mi. evaluation results on the development dataset show that, overall, this method is promising and outperforms the more conventional multi-label document classification using the “one-vs-all” strategy. however, its superiority was not confirmed on the test dataset, and the variance indicates that the model may be overfitted to the training and development data. we also tested simple ensemble systems using heuristic rules of union and intersection to combine methods, and achieved very good overall performance on both test datasets, which provided evidence that the systems complement each other. for act, we tested lr and svm classifiers exploiting a rich set of features obtained by linguistic processing and automatic recognition of a wide range of biomedical named entities. a series of feature-knockout experiments showed that the discriminative power is drawn mainly from contextual words surrounding automatically recognized named entities, as well as mesh. 
nevertheless, on the test dataset the best performance was achieved by taking the union of the output of this method and that of a multi-class, multi-label document classifier, which indicates that the two types of systems complement each other in terms of recall. act aimed to automatically select relevant documents for ppi curation, whereas the goal of imt was to recognise the methods used in experiments for identifying the interactions in full-text articles. for imt, a new method that classifies pair-wise relations between every text phrase and candidate interaction method obtained promising results with an f score of %, as tested on the task’s development dataset. the evaluation results on the biocreative iii ppi test datasets showed that our systems were very competitive: one of our imt methods yielded the best performance among all participants, as measured by f score, matthew’s correlation coefficient and auc ip/r; whereas for act, our best classifier was ranked second as measured by auc ip/r, and also competitive according to other metrics. for act, our system exploited a rich set of features and also obtained encouraging results. we examined the features with respect to their contributions to the classification results, and concluded that contextual words surrounding named entities, as well as the mesh headings associated with the documents were among the main contributors to the performance. the selection of relevant articles for curation, and linking those articles to experimental techniques confirming the findings became one of the primary subjects of the recent biocreative iii contest. the contest’s protein-protein interaction task consisted of two sub-tasks: article classification task and interaction method task. for act, our classifiers exploited automatically detected named entities and other linguistic information.
 the selection of relevant articles for curation, and linking those articles to experimental techniques confirming the findings became one of the primary subjects of the recent biocreative iii contest. the contest’s protein-protein interaction task consisted of two sub-tasks: article classification task and interaction method task. act aimed to automatically select relevant documents for ppi curation, whereas the goal of imt was to recognise the methods used in experiments for identifying the interactions in full-text articles.  we proposed and compared several classification-based methods for both tasks, employing rich contextual features as well as features extracted from external knowledge sources. for imt, a new method that classifies pair-wise relations between every text phrase and candidate interaction method obtained promising results with an f score of %, as tested on the task’s development dataset. we also explored ways to combine this new approach and more conventional, multi-label document classification methods. for act, our classifiers exploited automatically detected named entities and other linguistic information. the evaluation results on the biocreative iii ppi test datasets showed that our systems were very competitive: one of our imt methods yielded the best performance among all participants, as measured by f score, matthew’s correlation coefficient and auc ip/r; whereas for act, our best classifier was ranked second as measured by auc ip/r, and also competitive according to other metrics.  our novel approach that converts the multi-class, multi-label classification problem to a binary classification problem showed much promise in imt. nevertheless, on the test dataset the best performance was achieved by taking the union of the output of this method and that of a multi-class, multi-label document classifier, which indicates that the two types of systems complement each other in terms of recall. for act, our system exploited a rich set of features and also obtained encouraging results. we examined the features with respect to their contributions to the classification results, and concluded that contextual words surrounding named entities, as well as the mesh headings associated with the documents were among the main contributors to the performance. 13- september the third biocreative, critical assessment of information extraction in biology challenge bethesda, md, usa 