REVIEW

Getting Started in Medical Education ScholarshipDavid A. Cook

Division of General Internal Medicine and Office of Education Research,
Mayo Clinic College of Medicine,
Minnesota, USA(Received for publication on January 7, 2010)(Accepted for publication on February 18, 2010)

Education scholarship and research are critically important in extending our ability to teachand assess effectively. Those considering a scholarly project in medical education should consider the following tips, learned from personal experience and supported by literature: 1) getsome training, 2) find a mentor, 3) ask important questions, 4) start small and grow, 5) aim
high, 6) don’t wait for the perfect study, 7) plan for adequate time and other resources, 8) attend to ethical issues, 9) network with others in the field, and 10) recognize that this is hard
work. By following these steps and planning ahead, scholars will be better poised to makemeaningful contributions to the art and science of medical education. (Keio J Med 59 (3) : 96－
103, September 2010)

Keywords: medical education, research design, research methods, outcome assessment, writing
Introduction

integration, application, and teaching. Boyer argued thateach scholarship type should be valued and recognized
for the contribution it makes to a field.
Scholarship of discovery includes most original research activities. One example of scholarship of discovery in medical education (selected from countless equally meritorious examples) is a study showing no difference in epidural catheter performance following training
using either a high-fidelity part-task trainer or a low-fidelity model built with a banana and bread.3 Another example of discovery is a study finding that adding multimedia images and sound to a high-stakes exam led to increased testing time and worse item psychometric performance.4 Scholarship of integration attempts to synthesize information and draw connections across disciplines, often in the form of systematic or narrative reviews. Examples include meta-analyses of Internetbased instruction5, 6 and a thought-provoking critique ofself-assessment.7 Scholarship of application employs existing theories and evidence in new ways, often borrowing ideas and research from other fields. For example,
neuroscience research on the mirror-neuron system
might explain why some computer animations facilitate
learning more effectively than others. 8 Scholarship ofMedical education is a rapidly growing field for scholarship and research, with many newcomers each year.
While some enter the field with clear ideas of whatshould be studied and how to do it, many struggle because they don’t know where to start. My intent, in thisarticle, is to introduce the concept of medical education
scholarship, and share ten suggestions for getting started.
High-quality patient care requires competent healthcare professionals. Education, like clinical medicine, isboth an art and a science. Thus, to most effectively train
healthcare professionals we must merge tradition and intuition with theory and empiric data. In short, we needevidence to guide our educational endeavors.1 Medical
education scholarship encompasses a wide range of activities2 with the common purpose of advancing our understanding of how to assess and improve the competence of future and practicing health professionals.
In 1990, in an attempt to expand the definition of academic productivity to encompass activities other than
traditional research, Boyer introduced a framework forthinking broadly about different types of scholarly activity. He proposed four types of scholarship – discovery,

Presented at the 1649th meeting of The Keio Medical Society in Tokyo, December 12, 2009.
Reprint requests to: David A. Cook, MD, MHPE, Division of General Internal Medicine, Mayo Clinic College of Medicine, Baldwin 4-A, 200 First Street
SW, Rochester, MN 55905, USA, E-mail: cook.david33@mayo.edu
96Keio J Med 2010; 59 (3): 96－103

teaching involves the development of new and evidencebased ways to facilitate and assess learning, as illustrated
by a multi-modal course in health policy9 and a needsdriven course on complementary medicine.10
These studies scarcely begin to reflect the breadth ofscholarly activity in medical education. Given the seemingly endless stream of unanswered questions, we needmore scholars committed to advancing the art and science of education. The following tips will assist thosewilling to accept this challenge.
1. Get Some Training
Most people conducting medical education scholarshiphave had years of professional training (for example, in
medicine or nursing), and many have advanced degrees.
Yet most will still need some additional preparation for
the rather different demands of educational activities.
Education scholars must understand and use teachingstrategies, learning theory, and assessment methods.
Training in research design, survey development, program evaluation, and statistical analysis will also prove
useful.
This training needs not be extensive; even a basic introduction to core concepts and vocabulary will greatly
help as you work with colleagues to plan, implement,
and share your results with others. Such training mightstart locally, through workshops or courses in your institution’s college of education or division of statistics. National and international meetings such as the Association
for Medical Education in Europe (AMEE), the Ottawa
Conference, the Association of American Medical Colleges (AAMC), and the Asia-Pacific Medical Education
Conference all offer workshops and precourses on education scholarship. For those wishing more formal training, the AAMC sponsors the Medical Education Research Certificate, and there are a growing number ofother advanced courses and master’s degrees in medical
education. Of course, self-directed learning (e.g., readingbooks, review articles, and journal reports) plays an essential role as well.
In my own career, I received training in epidemiology
and statistics during medical school and took courses on
evidence-based medicine during residency, but my firsttraining in medical education took place at the Ottawa
Conference and AMEE meetings. Because I enjoyedthese courses so much, and with a growing interest in
education research, I pursued a Master’s degree in
Health Professions Education at the University of Illinois at Chicago. I continue to attend workshops and readbooks and journals to enhance my skills. The successful
education scholars I know have followed diverse paths,
but all made conscious and consistent efforts to enhancetheir scholarly skill set.

97

2. Find a Mentor
Mentors play an essential role in fostering an academic
career,11 and this is nowhere more true than in medical
education scholarship. Mentors can guide you in obtaining appropriate training, asking the right questions, designing a project to yield defensible results, sharing results with others, and establishing connections with others in the field. Unfortunately, you may find yourself inthe situation I did early in my career, when I could not
find mentors at my institution skilled in education research. However, I did find someone. Although not an
education researcher, Denise Dupras counseled me
through several early and important decisions, helped
me write my first abstract, co-authored my first papers,
and assisted in my first workshop. I also found support
from local colleagues with specific skills in clinicalteaching, biostatistics, and oral presentation, and gradually incorporated associates from other institutions into
my growing circle of mentors. I’ve found that some of
the most influential mentoring has come through mypeers rather than my superiors. Collaborating, critiquing,
and most of all listening, we have been able to grow together in our expertise.
From this I’ve learned four lessons: First, mentors
need not be experts in your chosen field. Second, it’s often helpful to identify multiple mentors, each meeting a
specific need or playing a specific role in your scholarly
development. Third, don’t be afraid to look for mentors
outside your own institution. Finally, peer mentors1 2 areinvaluable.
3. Ask Important Questions
The most important part of an education scholarly activity is to ask and answer an important question.1 3, 1 4
The most rigorous study, if not focused on an importantquestion, will fail to attract much interest. Your work
must make a contribution to the field. While there is always some subjectivity in what constitutes “important,”
there are steps you can take to maximize the chance that
others will view your work as a contribution.
First, identify a question that’s important to you.
Where will you find such questions? Some questions
will be handed to you (e.g., an assignment from yourDean). Others will press upon your mind as you read anarticle or attend a conference. Most, however, will
emerge from your daily work. As you teach, assess, andadministrate, you will find yourself asking, “I wonder
what is the best way to do this?”, “How can we do thisbetter?”, or “What would happen if we …?” Each suchquestion is a potential opportunity for scholarship.
Second, make sure the answer isn’t already known.
Just because an idea or question is new to you, doesn’tmean someone else hasn’t thought of it or answered it.
As a journal editor and as a peer reviewer, the most com-

98

mon flaw I encounter in education manuscripts is a failure to critically examine the literature to see what others
have done. Empiric evidence confirms that critical literature reviews are important1 5 yet infrequent.1 6 Thus, thediligent literature review is a critical step in successful
scholarly activity. Searching the medical education literature isn’t always easy17, 18 but you can’t skip this step.
Third, highlight the gap in understanding. If your literature review didn’t answer your question, then by definition a gap exists in our understanding. Gaps might arise
from an unexplained finding in a research study, absent
or deficient evaluations of a given teaching approach fora new topic or group of learners, or a new or incompletely evaluated assessment method. The more clearly you
can identify this gap (also known as a problem statement) and the larger the gap, the easier it will be to see
how answering your question will advance the field.
Fourth, frame your question in the context of a broader
conceptual framework if possible. Conceptual frameworks (theories, models for how things work, or study
approaches) allow you to develop hypotheses about expected results, and help you to interpret results by suggesting potential explanations for why somethingworked or failed. They also enable you and others to apply (generalize) findings to new situations.1 9 Conceptual
frameworks help most when they form the foundation
for a study, rather than being tacked on at the end (forexample, when interpreting results).
Fifth, try to ask questions that enable improvements
and application in new contexts. To this end, I find it
helpful to classify education questions or studies into
three groups: description, justification, and clarification.2 0 Description studies describe what was done, oftenwith little or no evaluation. Such studies are useful whena field is in its infancy (as Web-based learning was 15
years ago), but people quickly demand more than just a
description. Justification studies typically come next,comparing the new approach (intervention) to no intervention (a control group) or the current standard (an active comparison group). While such studies demonstrate
the utility of a given intervention for a specific context
and learner group, they do not tell us what might happen
with different learners, contexts, or comparison interventions, or how to improve the current course. Thus, justification studies have limited value to people at other institutions. Clarification studies, by contrast, help explain
why or how things work and how to make them better.
Conceptual frameworks are a central component of clarification studies.2 0 Unfortunately, clarification studies
constitute a small minority of published work in medical
education.2 0 The field would benefit from more frequent
use of conceptual frameworks and more clarificationtype questions.
Finally, write your question clearly. A clear question(which can interchangeably take the form of a research
hypothesis or study purpose or goal) should state the in-

Cook DA: Medical Education Scholarship

dependent variable (what you, the investigator, are doing; often an intervention or [for validity studies] assessment instrument), the dependent variable (the outcome),
and the population (usually the group of learners towhom you wish to apply your results).1 3, 1 6 This question
should be “FINER”2 1 – feasible, interesting (to you andto others), novel, ethical (more on this later), and relevant to your practice.
Remember that the question must always precede theanswer. At some point you will likely find yourself
teaching a new course or analyzing a set of evaluation
data, and wonder how you can turn this activity into ascholarly project. Such situations present wonderful opportunities but they are really means to an end, which is
to answer an important question. The question mustcome first. Projects in which the opportunity overshadows the question are difficult to interpret and generalize.
By way of analogy, if I were given an electric saw as a
gift I would not start cutting the nearest piece of wood.
Rather, I would carefully consider ways in which I might
use this tool to create something useful or beautiful.
Whenever you encounter an opportunity (a new course
or a dataset, for example), first use the steps above to
identify a clear knowledge gap and an important andnovel question.
4. Start Small and Grow
By now you may be saying, “I could never do that.” Ifso, you may be partly correct. It might not be possible
for you to lead a hypothesis-driven clarification-type
study right now. But with some training and experience
you could most likely do it (or something similar) down
the road. Thus, I encourage new scholars to start small
and grow at their own pace.
My first research project – which had nothing to dowith education – was never published. However, the experience I gained in that project proved invaluable in later activities. My first publication was a descriptive study– no outcomes – of a Web-based course. 2 2 My second
publication reported a pretest-posttest single-group study
evaluating the same course – a justification study (comparison with baseline).2 3 My third study was a randomized controlled trial comparing a different Web-based
course to the then-standard paper course.2 4 This justification study had limited generalizability, because a)even minor changes to the paper course or the Webbased course would require an entirely new evaluation,and b) it didn’t help me design a better course the following year. Several years and several projects later, I finally designed my first clarification study, which foundthat adding self-assessment questions to a Web-based
course improves learning outcomes.25
I share this to illustrate three points. First, one shouldn’t
expect to start off with a large-scale project; it’s entirely
reasonable – in fact, often preferable – to start realisti-

Keio J Med 2010; 59 (3): 96－103

cally small, and grow as you gain experience and momentum. Second, no single study is likely to revolutionize the world. On the contrary, most research involves
incremental steps that accrue over time to enhance ourunderstanding of a field.2 6 Third, I found success as I engaged in continuity of research – building on my previous results. Such programmatic research is key to academic success.27
So, go ahead and do that descriptive study of a novel
instructional approach (making sure, of course, that it really is novel). Or accept the Dean’s request to evaluate acourse using a pretest-posttest design. You can present
your work at a national conference, and perhaps publish
it. In doing so you will build momentum, gain experience, and be better poised to ask and answer insightfulquestions in the future.
5. Aim HighIt is acceptable to start small, but don’t settle for mediocrity. Whatever you do, you will want to do it well.
Glassick proposed six standards2 8 by which to assess thequality of scholarly activities. These standards apply
equally to all of Boyer’s scholarship types.
First, your activity should have clear goals. If you don’tknow where you are going, it is unlikely that you will
get there.
Second, you will need to demonstrate adequate preparation. This involves a diligent search to understand thecurrent state of the field, including previous scholarshipand research, relevant theories, ongoing activities at other institutions, and essential next steps. It also involves
making sure you have appropriate skills, resources, andmentoring.
Third, you need to employ appropriate methods. The
specific methods will vary according to the activity. For
scholarship of discovery (original research) these areyour research methods. For scholarship of integration
you might employ rigorous methods to conduct a systematic review. For application and teaching, you willneed to use methods appropriate to the objectives andcontexts, and based on current best practices.
Fourth, you need outstanding results. This does notmean statistically significant results (p<0.05), but rather
that the results make a substantial contribution to thefield. Making this judgment often involves review by apeer, either internal (a local colleague) or external. Outstanding results are most likely to be achieved if you
identified clear and important goals, prepared well, andused appropriate methods.
Fifth, you need to communicate results effectively.
Your communication must be transparent, well-organized, and complete. Many resources are available to assist you in this important step.29-31
Sixth, you need to demonstrate reflective critique. Thisrequires you to step back and analyze the strengths and99

limitations of what you did. What went well? Whatwould you change if you were to do it again? How could
you improve? And, what are the next steps?
Again, these standards apply to all scholarly activities,
large or small. It is not the size or the expense of a project that determines its quality, but rather the degree towhich your results are defensible and advance the field.
Plan ahead, and be sure to consider all of Glassick’s criteria.
6. Don’t Wait for the Perfect Study
No study is perfect. You might worry because you cannot randomize, or because your sample size is small.
True, these are important considerations. However, evenrandomized trials can have fatal flaws, and having alarge sample size does not guarantee quality results. If
your question is important and novel, then it merits answering – and any answer is probably better than none.
Moreover, as noted in Tip 4, the results of an imperfect
study will provide the starting point for future investigation. Rather than focusing on study designs, I encourage
you to think about threats to study validity, study reproducibility, and meaningful and relevant outcomes.
Ultimately, we don’t really care about the study methods or even the results. We care most about our interpretations of those results, and how those interpretations
can inform future actions. Of course, the methods, results, and interpretations are all inextricably intertwined;
stronger methods permit stronger inferences and greater
confidence in our actions. Anything that interferes withinferences is called a validity threat. A detailed discussion of validity threats is beyond the scope of this article,
but Table 1 summarizes several threats; other resources
contain more information. 3 2- 3 4 Overcoming validity
threats generally involves one of four solutions: using acontrol group (if possible), randomizing groups (if possible), standardizing conditions between groups (if possible), and obtaining more information on the participants
and on what actually happened during the study (always
possible).
Naturally, there are trade-offs in cost and feasibility
with each solution. How can one know if the right balance has been achieved? Cronbach proposed a three-stepthought experiment to address this issue.3 5 The scholar
asks: Would we expect sufficiently similar results if: a)
we used the same design and a different sample (different group of study participants), b) a different investigator used the same design and sample, or c) we used a different design? If the answer to all three questions is yes,then the design is likely to yield acceptable conclusions.
If not, then perhaps some revision is in order. Of course,“sufficiently similar” is a judgment call.
In recent years we have seen greater emphasis on“higher-order” outcomes such as physician behavior andeffects on patient care.3 6 While I appreciate the intent of

100

Cook DA: Medical Education Scholarship

Table 1 Threats to study validity
ExampleHow to Minimize the Threat

Subject character- Differences among participants at start
isticsof study

ThreatDescription

A college teacher compares face-to-face andonline versions of the same course; learners
can self-select which section they join (problem: people who elect face-to-face format
might be systematically different than thoseelecting online)

Randomization

History

Unplanned events unrelated to the intervention that might impact outcome

Pretest-posttest single-group study of online
modules on bioterrorism begins August 2001
(problem: it is impossible to distinguish theeffects of the intervention from the generally
increased awareness of bioterrorism that began September 2001)

Concurrent control group

Maturation

Changes in participants over time unrelated to particular events

To evaluate a 1-hour lecture on ECG interpretation on day 1 of a 2-month cardiology
rotation, investigators give a pretest, and thena posttest at the end of the rotation (problem:
it is impossible to distinguish the effects of theintervention from other learning experiences
over the 2-month rotation)

Concurrent control group

Instrumentation

Changes in scoring rubric or instrument
calibration, including rater fatigue

A single rater codes 100 five-minute videotapes in a single session (problem: rater fatigue may affect scoring consistency over thecourse of the session)

Control group; rating quality
control; information on instrumentation procedures

Regression to themeanParticipants selected or groups assigned
based on high or low performance will
be closer to average upon subsequenttesting47

Students in the lowest 5% of the class participate in a special remediation activity and arethen retested (problem: on average students
will show improvement upon retest even without remediation)

Control group (intervention
not based on baseline performance)

Testing

Taking a pretest can affect study outcomes48

To evaluate a series of Web-based modules,
an investigator administers a pretest, then themodules, and then the same test as a posttest (problem: learners might do better on
the posttest for reasons unrelated to the intervention itself, such as studying to the test,
familiarity with questions, and motivation to
improve low scores)

No pretest

Mortality (loss tofollow-up)

Participants leave study

Thirty-six people attend day 1 of a four-session course on communication skills, whereas
only 17 attend on day 4; the course evaluation
survey indicates very positive ratings (problem: the 19 people who didn’t return likely
would have given lower ratings)

Prevent loss; collect information on those lost

Participant attitude and motivation

Learners involved in something they
consider novel, or who are being observed, tend to be more motivated (conversely, those in the comparison groupmay be demotivated)

Students are told they will be part of a “veryexciting research project” (problem: participants will likely be motivated to demonstratebetter than normal behavior)

Blind participants to study hypothesis

Implementation

Variation in the learning experience, A teacher with extensive training and experie.g., differences in the expertise of the ence in problem-based learning (PBL) plans a
instructors, the opinions of instructors randomized trial comparing PBL to lecture; toregarding the efficacy of the interven- control for content, the teacher will both leadtion, or the actual amount of instruction the PBL groups and give the lecture (probreceived (e.g., did participants skip lem: the teacher is an expert at PBL but not atclass?). Learning outside the curriculum lecture, which would favor the PBL interven(how much learners studied on the topic tion;49 moreover, the teacher may unwittingly
beyond that intended by the interven- favor the PBL group)tion) falls into this category as well.

Careful planning of study interventions; collect informationon actual experiences (bothwithin and without the study)

See Fraenkel and Wallen,33 Campbell and Stanley,32 and Cook and Beckman34 for more information on these threats.

Keio J Med 2010; 59 (3): 96－103

this movement and agree with it in principle, in many instances it is counterproductive. Overemphasis on higher
outcomes risks failure to establish a causal link between
the intervention and outcome3 7 and can result in undue
focus on measurable outcomes rather than the most important outcomes. It also serves to discourage research ingeneral by setting the bar so high as to be achievable byonly a select few. I prefer to choose outcomes based on
the educational objectives (what will determine learner
success?) and audience needs (what will be meaningful
to the people who read your report?). Intermediate outcomes such as knowledge or skill, process measures
such as attendance, and narrative comments often provide more insight than higher-order outcomes. Other
sources contain more information on selecting outcomes.38-40
7. Plan for Adequate Time and Other Resources
Leading a scholarly activity to successful completion
will require many resources, not the least of which istime. How to find time for your project will depend on
the circumstances, but it is essential that you plan forthis up front. You should also ensure you have otherneeded resources such as administrative support, money,
and willing participants.
8. Attend to Ethical Issues
The ethical implications of education research are increasingly recognized.4 1, 4 2 Learners constitute a “vulnerable population” because teachers (who are often theinvestigators in such studies) also control grades, lettersof recommendation, and even job offers. Moreover,learners may worry about actual learning that might begained or lost by participation in a research study. Investigators must be sensitive to these issues and take appropriate measures to safeguard learner rights. Such measures might include obtaining informed consent, making
participation truly voluntary, keeping data confidential,and ensuring that reported data cannot be traced to theindividual. Many journals, including Academic Medicine4 3 and Medical Education,4 4 now require a statementof institutional review board (IRB) approval or other indicators of ethical research conduct before a manuscript
will even be considered.
9. Network with Others in the Field
As noted above, one of the great benefits of engaging
in medical education scholarship is the opportunity toestablish relationships with wonderful people around theworld. However, networking with colleagues is morethan just a social activity; it is also essential for your career. Colleagues locally and at other institutions can provide support when morale is low, a sounding board for

101

ideas, and opportunities for collaboration on projects.
You can build these networks within your own institution, at national and international meetings, through involvement in societies and multi-institutional committees, and informally over the Internet.
In my experience, relationships are built most effectively when you approach an individual with a purposein mind. I’ll never forget the time I went up to a renowned
investigator and introduced myself. He said, “It’s nice tomeet you,” and then – as I had nothing more to say – hewalked on. As I stood there, I realized that I had done
nothing to establish a relationship or make myself memorable, and I was quite certain he would soon forget me.
However, six months later we had the opportunity to discuss some research of mutual interest over breakfast, andit was the beginning of a friendship that has lasted years
and resulted in several productive collaborations. Purposeful meetings might involve asking a question, sharing an idea, inviting involvement on a project, or enjoying a meal or beverage together.
10. Remember This is Hard Work (Plan Ahead)
As Gruppen noted, medical education scholarship isfun, but it is not easy.4 5 Study groups are often biased,
treatments are typically confounded and rarely standardized, assessments are imprecise and poorly validated,
and funding is difficult to find.
I don’t say this with the intent of discouraging you, butrather to emphasize that quality work requires an investment of time and energy. Making the most of this investment requires careful planning. The more you plan (inadvance, if possible), the easier and more successful
your activities will be. I advocate a three-step approach:4 6 1) identify the scholarly question, 2) use appropriate methods, and 3) select appropriate outcomes.
Closing
Education research and scholarship are critically important to extending our ability to teach and assess effectively. If you are considering a scholarly project in medical education, I encourage you to get some training, finda mentor, ask important questions, start small and growwhile continuing to aim high, accept that your study will
likely be imperfect, plan for adequate time and other resources, attend to ethical issues, network with others in
the field, and recognize that this is hard work. As you
follow these steps and plan ahead, you will be better
poised to make meaningful contributions to the art andscience of medical education.
Acknowledgments
I thank Dr. Denise M. Dupras and Dr. Thomas J. Beckman (Mayo Clinic) for years of sage counsel and pro-

102

Cook DA: Medical Education Scholarship

ductive collaboration and for critical review of this manuscript. I also thank Dr. Haruo Kashima and Dr. Michito
Hirakata (Keio University) for stimulating the development of this manuscript through their invitation to present at Keio University.
References
1. Harden RM, Grant J, Buckley G, Hart IR: BEME Guide No. 1:
Best Evidence Medical Education. Med Teach. 1999; 21: 553–
562.
2. Regehr G. Trends in Medical Education Research. Acad Med
2004; 79: 939–947.
3. Friedman Z, Siddiqui N, Katznelson R, Devito I, Bould MD, Naik
V: Clinical impact of epidural anesthesia simulation on short- andlong-term learning curve: High- versus low-fidelity model training. Reg Anesth Pain Med 2009; 34: 229–232.
4. Holtzman KZ, Swanson DB, Ouyang W, Hussie K, Allbee K: Useof Multimedia on the Step 1 and Step 2 Clinical Knowledge Components of USMLE: A Controlled Trial of the Impact on Item
Characteristics. Acad Med 2009; 84(10 suppl): S90–S3.
5. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM: Internet-Based Learning in the Health Professions: A
Meta-Analysis. JAMA 2008; 300: 1181–1196.
6. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM: Instructional Design Variations in Internet-Based Learning for Health Professions Education: A Systematic Review andMeta-Analysis. Acad Med 2010; 85: 909–922.
7. Eva KW, Regehr G: Self-assessment in the health professions: areformulation and research agenda. Acad Med 2005; 80(10 suppl): S46–54.
8. Ruiz JG, Cook DA, Levinson AJ: Computer animations in medical education: a critical literature review. Med Educ 2009; 43: 838–846.
9. Greysen SR, Wassermann T, Payne P, Mullan F: Teaching Health
Policy to Residents – Three-Year Experience with a Multi-Specialty Curriculum. J Gen Intern Med 2009; 24: 1322–1326.
10. Cook DA, Gelula MH, Lee MC, Bauer BA, Dupras DM, Schwartz
A: A Web-based course on complementary medicine for medical
students and residents improves knowledge and changes attitudes.
Teach Learn Med 2007; 19: 230–238.
11. Sambunjak D, Straus SE, Marusic A: Mentoring in Academic
Medicine: A Systematic Review. JAMA 2006; 296: 1103–1115.
12. Pololi L, Knight S: Mentoring faculty in academic medicine. A
new paradigm? J Gen Intern Med 2005; 20: 866–870.
13. Bordage G, Dawson B: Experimental study design and grant writing in eight steps and 28 questions. Med Educ 2003; 37: 376–385.
14. Prideaux D, Bligh J: Research in medical education: asking theright questions. Med Educ 2002; 36: 1114–1115.
15. Bordage G: Reasons Reviewers Reject and Accept Manuscripts:
The Strengths and Weaknesses in Medical Education Reports.
Acad Med 2001; 76: 889–896.
16. Cook DA, Beckman TJ, Bordage G: Quality of reporting of experimental studies in medical education: A systematic review.
Med Educ 2007; 41: 737–745.
17. Haig A, Dozier M: BEME Guide No 3: Systematic searching forevidence in medical education – Part 1: Sources of information.
Med Teach 2003; 25: 352–363.
18. Haig A, Dozier M: BEME guide no. 3: systematic searching forevidence in medical education – part 2: constructing searches.
Med Teach 2003; 25: 463–484.
19. Bordage G: Conceptual frameworks to illuminate and magnify.
Med Educ 2009; 43: 312–319.
20. Cook DA, Bordage G, Schmidt HG: Description, Justification,
and Clarification: A Framework for Classifying the Purposes of

Research in Medical Education. Med Educ 2008; 42: 128–133.
21. Hulley SB, Cummings SR, Browner WS, Grady D, Hearst N,
Newman TB: Designing clinical research: An epidemiologic approach. 2nd ed. Philadelphia: Lippincott Williams & Wilkins;
2001.
22. Cook DA, Dupras DM: Flexible teaching for inflexible schedules:
An online resident curriculum in acute ambulatory care. Med
Teach 2003; 25: 330–331.
23. Cook DA, Dupras DM: Teaching on the Web: Automated online
instruction and assessment of residents in an acute care clinic.
Med Teach 2004; 26: 599–603.
24. Cook DA, Dupras DM, Thompson WG, Pankratz VS: Web-Based
Learning in Residents’ Continuity Clinics: A Randomized, Controlled Trial. Acad Med 2005; 80: 90–97.
25. Cook DA, Thompson WG, Thomas KG, Thomas MR, Pankratz
VS: Impact of Self-assessment Questions and Learning Styles in
Web-based Learning: A Randomized, Controlled, Crossover Trial.
Acad Med 2006; 81: 231–238.
26. Cook DA: One Drop at a Time: Research to Advance the Scienceof Simulation. Simul Healthc 2010; 5: 1–4.
27. Bordage G: Moving the field forward: going beyond quantitativequalitative. Acad Med. 2007; 82(10 suppl): S126–128.
28. Glassick CE, Huber MT, Maeroff G: Scholarship Assessed: Evaluation of the Professoriate. San Francisco, CA: Jossey-Bass;
1997.
29. American Educational Research Association: Standards for reporting on empirical social science research in AERA publications. Educ Res 2006; 35: 33–40.
30. von Elm E, Altman DG, Egger M, et al: The Strengthening theReporting of Observational Studies in Epidemiology (STROBE)
Statement: Guidelines for Reporting Observational Studies. Ann
Intern Med 2007; 147: 573–577.
31. Cook DA, Bowen JL, Gerrity MS, et al: Proposed Standards for
Medical Education Submissions to the Journal of General Internal Medicine. J Gen Intern Med 2008; 23: 908–913.
32. Campbell DT, Stanley JC: Experimental and Quasi-Experimental
Designs for Research. Chicago: Rand McNally; 1963.
33. Fraenkel JR, Wallen NE: How to design and evaluate research ineducation. New York, NY: McGraw-Hill; 2003.
34. Cook DA, Beckman TJ: Reflections on experimental research inmedical education. Adv Health Sci Educ Theory Pract. 2008;Epubahead of print 22 April 2008; doi 10.1007/s10459-008-9117-3.
35. Cronbach LJ: Designing Evaluations of Educational and Social
Problems. San Francisco: Jossey-Bass; 1982.
36. Chen FM, Bauchner H, Burstin H: A Call for Outcomes Researchin Medical Education. Acad Med 2004; 79: 955–960.
37. Shea JA: Mind the gap: some reasons why medical education research is different from health services research. Med Educ 2001;
35: 319–320.
38. Worthen BR, Sanders JR, Fitzpatrick JL: Program Evaluation: Alternative approaches and practical guidelines, 2nd edition. New
York: Addison Wesley Longman; 1997.
39. Kern DE, Thomas PA, Howard DM, Bass EB: Curriculum development for medical education: A six-step approach. Baltimore,
MD: The Johns Hopkins University Press; 1998. 178 p.
40. Cook DA: Twelve Tips for Evaluating Educational Programs.
Med Teach 2010; 32: 296–301.
41. Henry RC, Wright DE: When Do Medical Students Become Human Subjects of Research? The Case of Program Evaluation.
Acad Med 2001; 76: 871–875.
42. Tomkowiak JM, Gunderson AJ: To IRB or Not to IRB? Acad Med
2004; 79: 628–632.
43. Kanter SL: Ethical approval for studies involving human participants: academic medicine’s new policy. Acad Med 2009; 84: 149–
150.
44. Eva KW: Research ethics requirements for Medical Education.
Med Educ 2009; 43: 194–195.

Keio J Med 2010; 59 (3): 96－103
45. Gruppen LD: Is medical education research ‘hard’ or ‘soft’ research? Adv Health Sci Educ Theory Pract 2008; 13: 1–2.
46. Beckman TJ, Cook DA: Developing scholarly projects in education: a primer for medical teachers. Med Teach 2007; 29: 210–
218.
47. Bland JM, Altman DG: Statistic Notes: Regression towards the103
mean. BMJ 1994; 308: 1499.
48. Cook DA, Thompson WG, Thomas KG: Case-based or non-casebased questions for teaching postgraduate physicians: a randomized crossover trial. Acad Med 2009; 84: 1419–1425.
49. Devereaux PJ, Bhandari M, Clarke M, et al. Need for expertise
based randomised controlled trials. BMJ 2005; 330: 88–91.