The Relationship between the Nature of Practice and Performance on a Cognitive Examination

Certification by a specialty board affiliated with the American Board of Medical Specialties is one of the most widely used markers of physician competence in the United States. To enhance the meaning of this credential and in recognition of the need for periodic reassessments of physicians, the specialty boards have timelimited their certificates. To maintain certification, most of the boards have developed programs that incorporate (1) a check of credentials, (2) self-evaluation and/or continuing medical education, and (3) a secure written or computer-based examination. The secure examination is considered an integral component of certification because it provides assurance that physicians are keeping up with changes in medical knowledge and that they possess the ability to successfully manage patients’ problems that are important but rarely encountered in practice. Moreover, from patients’ and payers’ perspectives, a secure examination lends more credibility to the certification process. Despite these benefits, the secure examination is contentious because many believe that it tests only the ability to recall factual knowledge and, as such, bears little relationship to the day-to-day practice of medicine. However, the growth of electronic records and databases has made it possible to begin to address this concern by comparing certification status and test performance with aspects of practice such as volume, process of care, and patients’ outcomes. There is considerable evidence that physicians who treat large numbers of patients with a particular condition generally provide better care for such patients. Volume is directly associated with patients’ outcomes, regardless of the discipline or procedure. Therefore, the relationship of practice volume to examination scores is an important part of any test-validation effort. A study involving the first geriatric medicine certifying examination indicated that the number of geriatric patients seen in practice was positively correlated with examination scores. Likewise, a study involving cardiologists indicated that performance on cardiovascular graphics questions was positively correlated with experience. Specifically, scores on the interpretation of echocardiograms were correlated with the numbers of echocardiograms interpreted in practice or training. Similarly, scores on the interpretation of arteriograms and ventriculograms were correlated with the numbers of angioplasties performed. More recently, a study of a cognitive recertification examination in critical care medicine showed that scores were related to the amounts of time physicians spent in the direct care of critically ill patients. This relationship persisted even after statistically removing performance on the initial certifying examination in the same discipline. There is also evidence that certification status and examination performance are related to the process of care. A study of physicians in Quebec compared consultation rates, inappropriate prescribing for the elderly, and mammography screening rates with licensing examination scores based partly on cognitive tests. Physicians with higher scores referred more patients for consultation, prescribed fewer inappropriate drugs and more disease-specific medications for symptom relief, and appropriately referred more women for mammography. Similarly, a study of certified and non-certified internists found differences in preventive care services favoring the certified physicians. Although it is difficult to measure good practice outcomes well, some progress is being made in using available data in the validation of cognitive examinations. A recent study investigated whether there were differences among certified and self-designated cardiologists, internists, and family practitioners in the mortality of their patients with acute myocardial infarction. Data for all myocardial infarctions for calendar year 1993 in Pennsylvania were analyzed. Certification was associated with a 15% reduction in mortality irrespective of specialty, and after taking account of severity of illness, hospital characteristics, patient volume, and years since graduation. Similarly, Ramsey and colleagues found differences in some outcomes that favored the certified physicians, and there are a few studies with similar positive results in other specialties. Given the significance of the topic and the need for additional investigation, the purpose of this study was to extend previous work by exploring in more detail the relationship between examination performance and the nature of practice. Specifically, candidates for recertification in critical care medicine supplied information, via a practice survey, about the amounts of time that they spent in the care of patients with cardiovascular and pulmonary problems (i.e., practice volumes). Moreover, they rated the complexity of the problems they saw. These practice data were compared with performances on the items from the examination that dealt specifically with cardiovascular and pulmonary problems to determine whether patient complexity, in addition to patient volume, was associated with test scores.