# Presentation: 095299

## A Best-Fit Model for Concept Vectors in Biomedical Research Grants

- Calvin A. Johnson1, William W. Lau1, Archna Bhandari2, Tin P. Doan1, Shaohua A. Wang1, and Timothy C. Hays2
- 1Center for Information Technology;2 Division of Program Coordination, Planning, and Strategic Initiatives; National Institutes of Health, Bethesda, MD

**Abstract**
*The Research, Condition, and Disease Categorization (RCDC) project was established  to standardize budget reporting by research topic**1**. Text mining techniques have been implemented to classify NIH grants, contracts, and intramural research projects into appropriate research and disease categories.  A best-fit model is shown to achieve classification performance rivaling that of concept vectors produced by human experts.*
**Introduction**
- On a yearly basis, the National Institutes of Health (NIH) reports to the Congress and the public on how much money is allocated to approximately 360 research and disease *categories*. Congress and the NIH Office of the Director use this data to better understand NIH research spending and priority areas. The RCDC project office has selected the Collexis® software as the platform with which to define disease categories and to classify research grants*2*.  The classification is based on a technique known as *fingerprinting*, which extracts from the text a set of concepts using a domain-specific taxonomy*3*.
**Best-Fit Model**
- The similarity between a grant and a disease is measured via a normalized dot product of the respective concept vectors.   The process of creating disease fingerprints, i.e., concept vectors, is necessarily rather complex and involves a series of facilitated meetings with subject-matter experts, thesaurus experts, and others.  The complexity of this process motives the potential need for a machine-learning system that can take advantage of previously coded grant data to generate a best-fit model for the disease fingerprint.
- One of the goals of this study has been to devise a system that can operate within the existing infrastructure of the RCDC project.  Thus, methods to be considered would need to take advantage of the existing concept vectors and Collexis® software.  In generating the document concept vectors, Collexis® tokenizes certain sections of the grant applications and extracts concepts found in the RCDC thesaurus.  We have thus limited our consideration to linear-discriminant classifiers , which are trained to find a hyperplane that separates the category-labeled documents from the documents not labeled with that category.

**A Best-Fit Model for Concept Vectors in Biomedical Research Grants**

- ￼

**Methods**
- This study considers two well-known linear-discriminant classifiers, namely the perceptron and the support vector machine (SVM).   The perceptron is trained via a steepest-descent algorithm that minimizes the sum of the projections (i.e., dot products) between the category’s concept vector and the concept vectors of the misclassified training documents. If there are *N**t* documents in the training set, the resulting hyperplane will completely separate the two training categories if there are *N**c* ≥ *N**t*/2 independent concepts in concept space.  Fig. 1 illustrates the growth of concept space with the number of labeled positives in perceptron training.  The number of positives available for training, and hence the size of concept space, affects linear classifier performance. The SVM training algorithm finds the maximally separating hyperplane via a dual-space implementation.  Since *N**c* » *N**t*/2 for the linear-kernel SVM, dimensionality-increasing SVM kernels are not needed for this application.
** **

**Discussion**
- This investigation demonstrates the feasibility of machine-learning methods to train linear discriminant functions for grant classification.  Performance as measured by F-scores is similar across all three methods tested. The results presented here were drawn from 2006 data (for which there is no gold standard) using draft category fingerprints and, for certain categories, incomplete testing sets.  The RCDC category fingerprints were initially generated based on concept frequencies and later refined in a machine-assisted process by human experts. More recent refinements including validity feedback have made possible the future incorporation of a best-fit model.
- The first numerical column in Table 2 illustrates that for all but one of the poorly performing perceptron-trained vectors, the number of positives available is quite low.  This observation and Fig.1 confirm the idea that linear classifier performance requires a sufficiently large concept space. For most of these categories in Table 2, the RCDC category fingerprint also did not fare well.  Perhaps a method based on pattern matching may be more appropriate than one based on similarity of linear concept vectors for these categories.  Should it be possible to characterize and anticipate poorly performing categories (using straightforward descriptors), and select an alternative machine-learning method for those categories, aggregate F-scores well above 0.8 may be achievable.

**Experimental Results**
- Classification performance of the 2006 RCDC category fingerprints generated by human experts have been compared against that of perceptrons and linear SVMs trained on 2006 grant data. Table 1 provides an overall performance summary; Figs. 2--4 provide histograms for each of the  three classifiers on 178 DCs using 2006 grant data.  SVM training was limited to 1000 negatives due to limitations in computational resources and time.  Had the SVM-trained classifiers been trained on larger training sets with 5000 negatives (as with the perceptron), the SVM’s performance would have likely exceeded the perceptron’s.
- For 11 of the worst-performing perceptron-trained category vectors, perceptron training was also conducted with randomly labeled training sets.  In each of these “random” tests, recall and precision were 0.0, as shown in Table 2.  A pairwise T-test between the means of the randomly labeled and properly labeled perceptrons revealed P-values of 5.21×10-5 for recall and 1.59×10-8 for precision.

**Fig. 1.  **In training the perceptron, a strong relationship exists between the number of positives in the training set and the size of the concept space.

**References**
- http://rcdc.nih.gov
- Brynko B, Collexis:  “The research fingerprint,” *Information Today*, 2008;25:2:21-22.
- Bhandari A and McGowan C: “Using fingerprints to mine and search text,” *Horizons: Perspectives in Healthcare Management and Information Technology*, Fall 2005:37-40.

**Fig. 3.  **Histogram of F-scores of the perceptron-trained concept vectors on testing sets of 5000 negatives, 40% of positives.   Training sets consisted of 60% of positives and 5000 negatives.

**Fig. 2.  **Histogram of F-scores of RCDC category fingerprints on testing sets with 5000 negatives and 40% of positives.

**Fig. 4.  **Histogram of F-scores of the linear SVM-trained concept  vectors on testing sets of 5000 negatives and 40% of positives.  Training sets consisted of 60% of positives up to a maximum of 300, and 1000 negatives.

**Table 2. **Classification performance of the categories whose perceptron-trained concept vectors had among the worst F-scores; comparison with perceptron trained with randomly labeled data, and RCDC category fingerprints on the same data.

**Table 1.  **Summary result comparing size of the concept space and overall performance of the RCDC fingerprint, perceptron, and linear SVM.  Numerical columns provide mean values, with standard deviations in parentheses.

|  |  | Perceptron with random training |  | Perceptron with labeled training |  |  | RCDC category fingerprints |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Disease Category | # pos | recall | precision | recall | precision | F-score | recall | precision | F-score |
| Stroke education | 5 | 0 | 0 | 0.00 | 0.00 | 0.00 | 1.00 | 0.03 | 0.05 |
| Burden of  illness | 578 | 0 | 0 | 0.07 | 0.72 | 0.13 | 0.05 | 1.00 | 0.10 |
| Food defense | 9 | 0 | 0 | 0.11 | 1.00 | 0.20 | 0.22 | 1.00 | 0.36 |
| Hepatitis-A | 8 | 0 | 0 | 0.13 | 0.50 | 0.20 | 0.75 | 1.00 | 0.86 |
| Hyperbaric oxygen | 6 | 0 | 0 | 0.17 | 1.00 | 0.29 | 0.67 | 1.00 | 0.80 |
| Homicide and legal interventions | 14 | 0 | 0 | 0.21 | 0.75 | 0.33 | 0.07 | 1.00 | 0.13 |
| Human fetal tissue transplantation | 9 | 0 | 0 | 0.22 | 0.67 | 0.33 | 0.11 | 1.00 | 0.20 |
| Pelvic inflammatory disease | 7 | 0 | 0 | 0.29 | 0.67 | 0.40 | 0.29 | 1.00 | 0.44 |
| Pick’s disease | 6 | 0 | 0 | 0.33 | 0.67 | 0.44 | 0.50 | 1.00 | 0.67 |
| Myasthenia gravis | 17 | 0 | 0 | 0.41 | 0.64 | 0.50 | 0.82 | 1.00 | 0.90 |
| Ataxia telangiecstasia | 34 | 0 | 0 | 0.71 | 0.63 | 0.67 | 0.21 | 1.00 | 0.34 |

- Mean = 0.73

| Classifier | # Concepts |  | Recall |  | Precision |  | F-score |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RCDC | 134 | (281) | 0.64 | (0.23) | 0.96 | (0.16) | 0.73 | (0.22) |
| Perceptron | 5291 | (2291) | 0.78 | (0.09) | 0.75 | (0.14) | 0.76 | (0.18) |
| Linear SVM | 11,524 | (558) | 0.75 | (0.14) | 0.74 | (0.16) | 0.73 | (0.12) |

- Acknowledgments:  Gyan Bhanot for providing helpful suggestions; John Farley for contributing to the early testing.