Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training

Surgery has always been a dynamic discipline. It has a long history of groundbreaking innovations that dramatically improved the way patients were treated and surgery was practiced. However, nothing could have prepared the surgical community for the revolutionary advances of the last 2 decades. Almost overnight, the traditional open approach to surgery was replaced by a minimally invasive surgery (MIS) approach [2]. Practitioners quickly realized that this novel approach would require new types of training. Suddenly, the directors of surgical residency programs had to deal with faculty members who were questioning the training paradigm that had served surgery well for a century. The traditional Halsteadian method did not seem to be suitable for the new types of skills required for MIS. To make matters worse, the length of time allocated to training residents was contracting. In the European Union in 2001 the European Working time Directive was extended to include physicians in training and this year in the United States of America the mandatory 80 hour work week restriction has been instituted, both dramatically reducing work/training hours for residents. At the same time, the practice of surgery came under scrutiny in another, apparently unrelated area when the value and relevance of surgical research became an issue of controversy. In an unprecedented attack, the editor of the Lancet [6] publically called into question the quality of current surgical research. In an editorial entitled ‘‘Surgical Research or Comic Opera?’’ Dr. Horton expressed grave reservations about its utility, validity, and usefulness. Although the editorial was met by the surgical community with the predictable indignant response, Horton did make some valid points. Not necessarily in response to this editorial but certainly contiguous to it, surgical research appears to be moving away from reliance on retrospective data to prospective, randomized, and—where possible— blinded data. It is our contention that these two issues, i.e., MIS skills and surgical research, have now converged around the very essence of our surgical heritage—the stringent use of the ‘‘scientific method’’ and objective assessment. Perhaps this is best exemplified by the relatively new editorial practice of prioritizing prospective, randomized, blinded surgical trials for publication. Although blinded, randomized, controlled trial design can be employed to answer definitively many surgical research questions, we must remember that this design is only one of a quiver full of research methodologies, and may not always be the experimental design of choice. The function of good science is to show clear cause-and-effect relationships, and this does not always require a prospective, randomized, double-blind experimental design. Indeed the optimal method may not even be experimental, but statistical, such as Structural Equation Models [3]. This methodology will almost certainly be required to answer the question of which fundamental psychological abilities predict MIS skills/performance or what factors predict obesity or treatment outcomes in obesity. In both of these examples the answer will be multifactorial, which precludes the use of traditional experimental design. The control exerted with traditional experimental design will be replaced with the equally powerful (but methodologically more sophisticated) ‘‘statistical’’ control. The point we want to make here is that there is no ‘‘magic bullet’’ experimental methodology for surgical research; instead, there is a range of experimental designs that should be applied appropriately to answer the pertinent research question. Research design should therefore vary depending on the circumstances. An eloquent example can be found in a report form the Dundee MIS group, who used four subjects and, in essence, a single-subject design to evaluate the utility of three-dimensional camera systems [5]. The research design they used answered the question unequivocally, but Correspondence to: A. G. Gallagher Surg Endosc (2003) 17: 1525–1529 DOI: 10.1007/s00464-003-0035-4