Technical Review

Number 11 

Developing a Methodology for Establishing a 

Statement of Work for a Policy-Relevant Technical 

Analysis

Prepared for: 

Agency for Healthcare Research and Quality 

U.S. Department of Health and Human Services 

540 Gaither Road 

Rockville, MD 20850 

www.ahrq.gov 

Contract No. 290-02-0025 

Prepared by: 

Duke Evidence-based Practice Center, Durham, NC 

Investigators 

David B. Matchar, MD, FACP 

Meenal Patwardhan, MD, MHSA 

Antonio Sarria-Santamera, MD, PhD 

Emma V. Westermann-Clark, BS 

AHRQ Publication No. 06-0026 

January 2006 
 

This report is based on research conducted by the Duke Evidence-based Practice Center 

(EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), 

Rockville, MD (Contract No. 290-02-0025). The findings and conclusions in this document 

are those of the author(s), who are responsible for its content, and do not necessarily 

represent the views of AHRQ. No statement in this report should be construed as an official 

position of AHRQ or of the U.S. Department of Health and Human Services. 

The information in this report is intended to help clinicians, employers, policymakers, and 

others make informed decisions about the provision of health care services. This report is

intended as a reference and not as a substitute for clinical judgment. 

This report may be used, in whole or in part, as the basis for the development of clinical 

practice guidelines and other quality enhancement tools, or as a basis for reimbursement 

and coverage policies. AHRQ or U.S. Department of Health and Human Services 

endorsement of such derivative products may not be stated or implied. 
ii

This document is in the public domain and may be used and reprinted without permission except 

those copyrighted materials noted for which further reproduction is prohibited without the 

specific permission of copyright holders. 

Suggested Citation: 

Matchar DB, Patwardhan M, Sarria-Santamera A, Westermann-Clark EV. Developing a 

Methodology for Establishing a Statement of Work for a Policy-Relevant Technical Analysis. 

Technical Review 11. (Prepared by the Duke Evidence-based Practice Center under Contract No. 

290-02-0025.) AHRQ Publication No. 06-0026. Rockville, MD: Agency for Healthcare Research 

and Quality. January 2006.
iii

Preface 

 The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-Based 

Practice Centers (EPCs), sponsors the development of evidence reports and technology 

assessments to assist public- and private-sector organizations in their efforts to improve the 

quality of health care in the United States. The reports and assessments provide organizations 

with comprehensive, science-based information on common, costly medical conditions and new 

health care technologies. The EPCs systematically review the relevant scientific literature on 

topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to 

developing their reports and assessments. 

 To bring the broadest range of experts into the development of evidence reports and health 

technology assessments, AHRQ encourages the EPCs to form partnerships and enter into 

collaborations with other medical and research organizations. The EPCs work with these partner 

organizations to ensure that the evidence reports and technology assessments they produce will 

become building blocks for health care quality improvement projects throughout the Nation. The 

reports undergo peer review prior to their release. 

 AHRQ expects that the EPC evidence reports and technology assessments will inform 

individual health plans, providers, and purchasers as well as the health care system as a whole by 

providing important information to help improve health care quality. 

 We welcome comments on this evidence report. They may be sent by mail to the Task Order 

Officer named below at: Agency for Healthcare Research and Quality, 540 Gaither Road, 

Rockville, MD 20850, or by e-mail to epc@ahrq.gov. 

Carolyn M. Clancy, M.D. 

Director 

Agency for Healthcare Research and Quality 

Kenneth S. Fink, M.D., M.G.A., M.P.H. 

Director, EPC Program 

Agency for Healthcare Research and Quality 

Jean Slutsky, P.A., M.S.P.H. 

Director, Center for Outcomes and Evidence 

Agency for Healthcare Research and Quality 

Margaret Coopey, R.N., M.G.A., M.P.S.

EPC Program Task Order Officer 

Agency for Healthcare Research and Quality
iv
v

Structured Abstract 

Context: The Duke Evidence-based Practice Center (EPC) initiated a project designed to 

improve the process of generating reports that synthesize and evaluate the scientific literature on 

topics of particular interest to health care policymakers, clinicians, and other decisionmakers. 

The project focused on two improvement opportunities: process efficiency and stakeholder 

satisfaction. We organized the analysis around the concept of a “statement of work (SOW).” As 

a contractual document, the SOW is the tangible manifestation of the objectives of each EPC 

project, serving as a foundation for interaction between involved parties, including technical 

details such as budget, timeline, and deliverables. 

Objective: To identify potential solutions to the core constraints identified by the study. 

Methods: The project proceeded in four steps: an interview of participants in the EPC process; a 

review of the technical contracting literature relating to formulating SOWs for the production of 

policy reports and similar intellectual products, and policy literature related to the determinants 

of a successful policy analysis; a synthesis of both the interview results and literature review 

utilizing tools suggested by the Theory of Constraints (TOC). 

TOC methods were applied to the initial results in order to identify a common undesirable 

effect (UDE) related to EPC reports: a report that does not get used is unsuccessful and is 

deemed fundamentally undesirable. A current reality tree (CRT) was constructed by working 

from this fundamental UDE at the top through proximate causes, and finally root causes 

identified during the process. 

Conclusions: The analysis identified several problems that ultimately result in a report not 

meeting the partner’s needs, and therefore not being used. The most significant problem, i.e. the 

core constraint (the most important target for improvement), identified was that the partner does 

not know how to conceptualize and articulate needs, objectives, and specifications. Several 

major potential change strategies for this core constraint were then explored. Two change 

strategies: establish an ongoing relationship between partners, EPC, and AHRQ early in the 

process, and utilize a changes clause were deemed to be the most practical means to exert impact 

on the SOW process by addressing this core constraint. In the final step, a Future Reality Tree 

was constructed representing an illustration of the impact of the proposed solutions on the EPC’s 

SOW process. 
vi
vii

Contents 

Executive Summary................................................................................................................. 1 

Introduction............................................................................................................................ 5 

Methodological Overview ..................................................................................................... 7 

 

Step 1: Informant Interviews ................................................................................................. 9

 Informant Interview Methods .......................................................................................... 9 

 Informant Interview Results ............................................................................................ 9 

 The Concept of a Successful Report.......................................................................... 9 

 Determinants of a Successful Report......................................................................... 10 

 The Role of AHRQ in Promoting Success ................................................................ 11 

 Summary of Step 1: Informant Interviews....................................................................... 12 

Step 2: Literature Review ...................................................................................................... 15 

 Literature Review Methods.............................................................................................. 15 

 Literature Review Results................................................................................................ 16 

 Culture/Education Factors ......................................................................................... 16 

 Strategic Factors......................................................................................................... 17

 Resource Factors........................................................................................................ 17 

 Summary of Step 2: Literature Review ........................................................................... 19 

Step 3: Synthesis—Development of the Current Reality Tree Based on the 

 Theory of Constraints ...................................................................................................... 21 

 Synthesis Methods ........................................................................................................... 21 

 Synthesis Results ............................................................................................................. 22 

Step 4: Identify Promising Solutions for the Core Constraints to Develop the Future Reality 

 Tree Based on the Theory of Constraints ........................................................................ 27 

 Solutions Methods ........................................................................................................... 27 

 Solutions Results.............................................................................................................. 27 

References .......................................................................................................................... 31 

Figures 

 Figure 1. Template of a Current Reality Tree.................................................................. 22 

 Figure 2. Current Reality Tree for the EPCs.................................................................... 25 

 Figure 3. Future Reality Tree of the SOW process for EPCs .......................................... 29 
viii

Table 

 Solutions (injections) that can address root causes of EPC-SOW problems ................... 28 

Appendix 

 Technical Description of the Statement of Work ............................................................ 33 
Technical Review 
2
3

Executive Summary 

In 1997, the Agency for Healthcare Research and Quality (AHRQ) created 12 Evidencebased Practice Centers (EPCs). The EPCs’ mission is to “develop evidence reports and 

technology assessments on topics relevant to clinical, social science/behavioral, economic, and 

other health care organization and delivery issues.” The principal products of the EPCs are called 

evidence reports. EPC evidence reports are evaluations of the scientific literature on topics of 

interest to policymakers, clinicians, and other decisionmakers. The reports are requested by 

organizations representing these interested parties. Like any process, the EPC process is subject 

to improvement. The importance of process improvement to the EPC program is reflected both 

in the existence and the content of the annual AHRQ meeting with EPCs and partner 

organizations. 

Two areas have been evident targets for improvement: process efficiency and stakeholder 

satisfaction (where stakeholders are both direct participants in the EPC effort as well as other 

parties with an interest in EPC products). This report describes a project initiated by the Duke 

Evidence-based Practice Center to contribute to the improvement of the EPC process, focusing 

on specific actions that could serve to improve process efficiency and stakeholder satisfaction. 

We organized our efforts around the concept of a “statement of work (SOW).” As a contractual 

document, the SOW is the tangible manifestation of the objectives of each EPC project, serving 

as a foundation for interaction between involved parties, including technical details such as 

budget, timeline, and deliverables. 

The project proceeded in four steps. In Step 1 we interviewed participants in the EPC process 

in order to establish a deeper understanding of the current process. In Step 2 we reviewed the 

technical contracting literature relating to formulating SOWs for the production of policy reports 

and similar intellectual products, and policy literature related to the determinants of a successful 

policy analysis. In Step 3 we applied the Current Reality Tree (CRT) technique from the Theory 

of Constraints (TOC) to synthesize the information provided in the first two steps in a way that 

highlights core constraints (i.e., constraints that are the most important targets for improvement.) 

In the final step (Step 4) we reviewed the information collected in the interviews and literature 

review in the context of the TOC exercise. The objective was to identify promising solutions for 

the core constraints, which may serve as input to an action plan by AHRQ in collaboration with 

EPCs and partners. With these solutions in mind, we constructed a Future Reality Tree, an 

illustration of the impact of the proposed solutions on the EPC’s SOW process. 

We utilized the Theory of Constraints for the next steps in the project. Our interviews with 

EPC researchers, partners, and AHRQ allowed us to identify a common undesirable effect 

(UDE) related to EPC reports: a report that does not get used is unsuccessful and is deemed 

fundamentally undesirable. We formulated a current reality tree (CRT) working from this 

fundamental UDE at the top through proximate causes, and finally root causes. In the next stage 

we extracted a list of root causes from our interviews. 

Following TOC methods, we next explored the relationship between these root causes in 

order to identify one (or two) major problems that lead to the rest, and ultimately result in a 

report not meeting the partner’s needs, and therefore not being used. Of these, the first (partner 

does not know how to conceptualize and articulate needs, objectives, and specifications) rates as 

a core problem or the constraint that prevents an EPC from achieving its goal because it affects 

most other problems. This is the reason why specifications are insufficient in the first place, 
4

which leads to an underestimation of the volume of work, results in new cost and time 

requirements that lead to delays in the report, and finally results in the report not being used. 

The interviews, literature search, and input from an EPC directors’ meeting identified several 

potential injections/change strategies: Two injections: establish an ongoing relationship between 

partners, EPC, and AHRQ early in the process, and utilize a changes clause were deemed to be 

the most practical means to exert impact on the SOW process by addressing the core constraint, 

partner does not know how to conceptualize and articulate needs, objectives and specifications 

of the SOW.

With these concepts of UDEs, root causes, and injections we constructed the Future Reality 

Tree for the SOW process; the ideal process that EPCs should follow in order to develop 

evidence reports that meet the needs of the partners. 
5

 

Introduction 

In 1997, the Agency for Healthcare Research and Quality (AHRQ) created 12 Evidencebased Practice Centers (EPCs). The objective of the EPC program was to provide a public 

service information resource for policymakers, clinicians, and other decisionmakers. According 

to AHRQ,1

 the EPCs’ mission is to “develop evidence reports and technology assessments on 

topics relevant to clinical, social science/behavioral, economic, and other health care 

organization and delivery issues.” In 2002, AHRQ announced a second 5-year cycle of funding 

for the program and expanded the total number of EPCs to 13. 

The principal products of the EPCs are called evidence reports or technology assessments; 

for simplicity’s sake, we refer to them here as evidence reports. EPC evidence reports are 

detailed evaluations of the scientific literature on specific clinical, behavioral, organizational, and 

financing topics of interest to policymakers, clinicians, and other decisionmakers. The reports are 

requested by organizations representing these interested parties. Termed “clients” in general 

management parlance, these organizations are called “partners” in the context of the EPC 

process. 

From the moment an EPC report is proposed, to its production and use by a partner, all 

parties are engaged in a process. Like any process, the EPC process is subject to improvement. 

The importance of process improvement to the EPC program is reflected both in the existence 

and the content of the annual AHRQ meeting with EPCs and partner organizations. 

Two areas have been evident targets for improvement: process efficiency and stakeholder 

satisfaction (where stakeholders are both direct participants in the EPC effort as well as other 

parties with an interest in EPC products). Process efficiency assures that the immediate goals of 

any EPC project are accomplished without wasting public resources. Stakeholder satisfaction 

directly impacts the long-term sustainability of the EPC program; stakeholders who have a sense 

of satisfaction in the process and the products are more likely to value and support the public 

service provided by the EPC program. 

This report describes a project initiated by the Duke Evidence-based Practice Center to 

contribute to the improvement of the EPC process, focusing on specific actions that could serve 

to improve process efficiency and stakeholder satisfaction. To maintain a level of concreteness, 

we organized our efforts around the concept of a “statement of work (SOW).” As a contractual 

document, the SOW is the tangible manifestation of the objectives of each EPC project, serving 

as a foundation for interaction between involved parties, including technical details such as 

budget, timeline, and deliverables. A well-constructed SOW is also the keystone for clear 

communication and effective interaction between naturally distinct parties. It should provide a 

level of guidance that maximizes the chance that the final product will serve the purposes of the 

partner or sponsor. When poorly constructed, the SOW can lead to significant delays and can 

ultimately result in an unsatisfactory report. 

 It should be noted that while the SOW document is of potential importance, it is the 

concept of the SOW that is fundamental. This project is aimed at identifying strategies for 

enhancing the success of the EPC program; some, if not all, of the enhancements should be 

reflected in an improved SOW document. (For a more detailed definition of the statement of 

work, see Appendix .)
6
7

Methodological Overview

The project proceeded in four steps: 

• In Step 1 we interviewed participants in the EPC process in order to establish a deeper 

understanding of the current process, focusing on the definition and determinants of a 

successful report. Although the primary goal was to understand current processes, we 

also solicited ideas about solutions, including strategies for producing a more useful 

SOW. 

• In Step 2 we reviewed the technical contracting literature relating to formulating SOWs 

for the production of policy reports and similar intellectual products, as well as the policy 

literature related to the determinants of a successful policy analysis. This step was 

intended to complement the interviews in Step 1 by placing the EPC activities in a 

broader context, and particularly to identify existing concrete solutions to problems noted 

in the interviews. 

• In Step 3 we applied the Current Reality Tree (CRT) technique from the Theory of 

Constraints (TOC) to synthesize the information provided in the first two steps in a way 

that highlights core constraints (i.e., constraints that are the most important targets for 

improvement.) Again, in the spirit of maintaining concreteness, we were especially 

attentive to distinguishing constraints that were within the control of the EPC 

participants. As part of this step we presented the work from Step 1 and Step 2, as well as 

preliminary reflections on the core constraints, at the 2004 annual EPC meeting and 

solicited written feedback. 

• In the final step (Step 4) we reviewed the information collected in the interviews and 

literature review in the context of the TOC exercise. The objective was to identify 

promising solutions for the core constraints, which may serve as input to an action plan 

by AHRQ in collaboration with EPCs and partners. With these solutions in mind, we 

constructed a Future Reality Tree, an illustration of the impact of the proposed solutions 

on the EPC’s SOW process. 
8
9

Step 1: Informant Interviews 

Informant Interview Methods 

Semi-structured interviews were conducted with the three groups involved in the EPC 

activities: EPC staff (research center directors and project managers), AHRQ staff, and 

representatives from partner organizations (both public and private). Interviews were conducted 

by telephone, were audio-taped, and lasted about 45 minutes each. The interviews were 

conducted over a 2-month period by a team of three researchers. An interview guide was 

followed. Questions were general in nature and focused on areas related to the establishment of 

appropriate processes for evidence-based reporting and health technology assessment. All topics 

in the interview guide were covered during each interview. Interviews were conducted with 11 

EPC directors, one EPC project manager, three AHRQ representatives, two Federal partners, two 

non-federal partners, two academic experts, and one management expert. 

All interview tapes were transcribed and then read several times to identify key issues. A 

structured coding scheme was devised based on a reading of the transcripts and the objectives of 

the project; sections of text were coded to the identified themes. Coded segments were then 

examined and compared, first to identify similarities and differences and later to produce a 

conceptual framework for analysis and interpretation. 

Informant Interview Results

An analysis of the coded transcripts was organized by the three major interview topics: 

1. The concept of a successful report. 

2. Determinants of a successful report. 

3. The role of AHRQ processes and procedures in promoting success. 

Quotes presented here have been selected as typical of the perceptions and experiences recorded; 

they are not, however, statistically representative of a larger population. 

The Concept of a Successful Report 

For researchers, success is perceived to occur at different levels. Primarily, a report is 

considered successful when the partners use the report (either to produce guidelines or develop 

policy), confirm that the report met their requirements, and provide the researchers with 

feedback from end users. In addition, for researchers, a successful report must be a rigorous 

synthesis (in accordance with evidence-based principles), intellectually satisfying, conducted 

using a credible synthesis of data, and provide new insights. A few researchers noted that reports 

should allow junior faculty in pursuit of an academic career to publish, while others did not view 

publication as an important endpoint. The metric by which success is measured has not been 

firmly established; there are some reports that, regardless of whether they have technically been 

good analyses or not, have either not been used to implement policy or have not received “a lot 

of press.” In addition to issues related to the nature of the application of the report, EPC directors 

generally agreed that a successful report must break even financially. 
10

AHRQ staff also reflected on what constitutes a successful EPC report, one that blends the 

notions of excellence and usefulness. First, a good report has to “meet the needs” of the partners. 

It answers questions that the partners will find useful and have asked. It has to be an excellent, 

well-written synthesis of the qualitative issues related to the topic addressed, and the highest 

methodological standards must have been applied. A successful report shows how each piece of 

evidence fits together to answer a question. It has clear key questions, identifies all the relevant 

literature, utilizes a technical expert panel knowledgeable about the subject, incorporates clinical 

opinion, and demonstrates an understanding of evidence, while remaining non-normative. A 

successful report requires “clinical content expertise, knowledge of the technology and 

identification of the policy implications.” 

The concept that partners have of a successful report is one that meets their needs. A 

successful report “includes what we told you to do.” In order to be a good report, it has to 

include the “universe of evidence,” not just a subset, with tables that organize evidence in a 

manner that is easy enough to understand so that committees can weigh the evidence themselves. 

They understand that in order to have a successful report, they must “have an outcome in mind.” 

Our non-EPC interviewees (two academic experts in technical analysis for policymaking and 

one management expert) confirmed the importance of a useful report as a prime measure of 

success. One academic expert suggested that one should not underestimate the potential value of 

academic products (e.g., manuscripts and technical reports) as they may influence the way 

people think about a problem; this effect may be slower but potentially more profound. 

Determinants of a Successful Report

For researchers, the most important determinant for a successful report is having a welldefined scope of work. Narrow and clearly worded questions facilitate research work and make it 

feasible to complete the report with the available time and resources. Other major conditions are 

to understand the audience and the intent of the report. Researchers need explicit information to 

know “the context of what users are looking for.” They also need to know more about the desired 

structure and tone of the report (scientific vs. lay tone). 

There is an overall concern among researchers, however, that these requirements are often 

not explicit enough or not clearly established in the original statement of work. If reports have to 

meet partners’ needs, those needs have to be identified. “Clear expectations need to be identified 

by the partner.” “What they want” has to be well articulated to result in a credible product, and it 

has to be identified up front using relevant questions. “A crucial component is to define the 

scope of the project up front.” Several problems have been identified with this process. In many 

instances, what EPCs are asked to do is completely “out of feasible bounds.” Broad scopes and 

questions are the norm, not the exception. In many cases, EPCs have perceived that partners 

were not sure what they really wanted to do. “It took us a while to figure out what they actually 

wanted.” When research questions have to be refined, not only is the scientific success of the 

report at risk, but there can be financial implications because “budget is the concrete 

manifestation of a statement of work.” This problem is particularly relevant because of the strict 

time frame for completion of reports. Because the work with partners changes over time, some 

change is inevitable; researchers indicate “we need flexibility.” However, when partners do not 

develop a clear conceptual framework up front, the result is that questions are not specific 

enough and researchers cannot define the type of analysis needed. 

Researchers indicated that one factor that can influence a project’s success is consistency of 

role expectation. Specifically, it is sometimes difficult to keep up with how partners define their 
11

roles; a partner may be “hands-off” in the beginning, and then decide that it wants to be 

intimately involved in completing the project, or it may request that the EPC become deeply 

involved in dissemination and implementation aspects of the project that were beyond the initial 

project scope. 

AHRQ recognizes that reports need to start with a well-defined conceptual framework, 

identifying what the problems are and where the complexities lie. Developing this conceptual 

framework requires time and commitment. In constructing the final report, the technical analysis 

often has to be adjusted to match the conceptual framework. 

Partners have also identified that a major determinant of a successful report involves asking 

focused questions. They consider this to be their main responsibility. The partners also raised the 

point that this is a learning process and that they have come to recognize this through their 

participation. On the other hand, several partners said that they were disappointed with certain 

reports that were too detailed to support decisionmaking. Even if the reports were technically 

perfect, they did not meet the needs of the end users because they did not present information in 

a useful manner. Partners said it is necessary to “identify the audience for the report early,” to 

“show how each piece of evidence fits together to answer a question,” and to structure the 

evidence report so that it is facile for decisionmakers to weigh the evidence themselves. 

Two other factors were generally agreed to be determinants of a successful report. The first is 

having an abundance of communication. All parties involved mentioned that establishing a 

“face-to-face relationship” is essential. Partners realize that it is crucial to have greater 

interaction with AHRQ and researchers in the very beginning stages of the project. “Our input 

can’t come too late in the process.” Establishing this relationship outlines the nature and content 

of the report in an interactive, refined process. For researchers there is an inherent tension: “the 

more we want the reports to have impact, the more we have to negotiate with the partners.” 

The second factor determining a successful report is the presence of a formal process for 

creating a standard report: establish a good working team with diverse and substantial expertise, 

develop collaborations with clinical and content experts, identify the literature, synthesize the 

evidence, and write a technically correct and readable report. The role of external experts is 

considered to be very helpful. The experts know who the leaders are in the field (professional 

societies, advocacy groups, industry), and can fill in the story behind the solicitation and 

agendas. 

One academic expert mentioned the importance of establishing a process for dealing with 

especially contentious topics (e.g., independent oversight panels) in circumstances where 

potentially important information might otherwise be suppressed or enhanced for reasons other 

than good science. The management expert highlighted the importance of a well-constructed 

SOW; this was the only person who volunteered the SOW as a determinant of a successful 

report. 

The Role of AHRQ in Promoting Success

Overall, both partners and researchers indicated a high level of satisfaction with AHRQ. 

Respondents also indicated that, in some areas, the role of AHRQ could be enhanced in order to 

improve the success of EPC reports. 

There is a widespread perception, both among researchers and partners, that AHRQ should 

mediate—or at least facilitate—the interaction between analysts and partners. “AHRQ has to 

make sure that the project they select creates the greatest value.” “I would be happy to see 

AHRQ take a stronger role in making partners prioritize questions, in order to avoid scope creep 
12

when the partner suddenly discovers he wants to ask another question.” There is a need to 

prioritize among all possible questions, focusing on those questions whose answers provide the 

greatest value. Partners have to understand that “they are going to receive what they need, but 

perhaps not what they want.” Researchers would like AHRQ to “push the partner to define what 

they want.” Partners should understand that they are receiving “an enormous gift,” but they have 

to be prepared to collaborate on the project and be ready to use it. They need to have an 

understanding of evidence-based medicine and systematic reviews. “We need to educate them.” 

Researchers advocate that AHRQ accomplish this work up front, “they could set the stage better 

with partners.” This educational process would allow for aligned expectations and a focus on the 

“right questions.” 

From the researchers’ perspective, AHRQ needs to minimize the gap between what partners 

want and what is expressed in the scope of work. Current mediation strategies were not seen as 

adequate. AHRQ has to identify “other ways of doing the intermediary work, other 

mechanisms.” Partners also recommended establishing a formal process, with clearly established 

parameters, to ensure that all issues are addressed. “We need a consistent, routinely followed 

process.” 

All parties expressed that the process has improved over time. Earlier reports were more 

“painful.” Both the EPCs and AHRQ commented that the more recent reports have been better. 

EPCs are more experienced doing things for the price, can better estimate the amount of 

literature they are going to have to review, and have developed more experienced teams. While 

“EPCs have learned by doing,” EPCs expressed a strong desire to have more opportunity to 

benefit from the experience of others. 

Three issues involving AHRQ were also mentioned. First among these was the existence of 

some variability among the different Task Order Officers overseeing the different reports in the 

way they conduct their work. Having a “fabulous project officer” is a key factor. A second issue 

was the role of the Coordinating Center. Although the establishment of the Coordinating Center 

raised a lot of expectations, the overall perception is that its impact is very limited at the 

moment. The third issue involves the budgets. The EPCs perceive that they are not receiving the 

funding they need to conduct their work. “This work is more than pulling together a few RCTs 

(randomized controlled trials).” Limited resources place a significant constraint on doing a 

quality job. “We need more money, more time.” This is still a largely boutique industry model 

and it is, consequently, quite expensive. Each product has to be developed largely from scratch, 

by hand. Although some standardization can help, tailoring is one of the most attractive and 

useful features of the EPC reports. While academic institutions seem to be better suited to find 

solutions (junior faculty can sometimes augment the contributions of assigned staff); for other 

institutions this is not always possible.

Summary of Step 1: Informant Interviews

Three salient themes emerge from the Step 1 interviews relating to the definition of a 

successful report, the determinants of a successful report, and the role of AHRQ in the EPC 

process. These themes relate to: (1) culture/education factors; (2) strategic factors; and (3) 

resource factors. 

 Cultural factors influence the definition of a successful report as one that is used 

productively. Researchers and users of evidence reports often have different values, goals, and 

perspectives about what constitutes productive use; these are shaped by their professional 
13

cultures. The process of linking the work of researchers and users is especially challenging 

because it requires the meshing of two worlds. The differences relate to familiarity with the 

principles of evidence-based medicine on the one hand and the practical realities of clinical 

medicine or the exigencies of policymaking on the other. The process of identifying and 

synthesizing best medical evidence into policy is subject to the risk that participants do not share 

the same cultural foundation, experiences, and expectations. Cultural understanding must occur 

at an early point in the development of evidence reports—policymakers and researchers 

clinicians need to teach each other. Ideally, the process of mutual education should begin with a 

face-to-face meeting immediately after the evidence report is commissioned. Establishing trust 

and expectations early is essential to the success of an evidence report. 

Regarding strategic issues, early development of a conceptual framework appears key to the 

development of a successful EPC report. A conceptual framework takes into account not only 

scientific evidence, but clinically relevant factors and stakeholder interests and serves as a 

vehicle for communication. 

Finally, resource factors can strongly influence the success of any implementation effort. It 

has been a traditional practice for organizations seeking to develop various products (such as 

clinical guidelines) to rely on the goodwill of their academic grantees to provide expert input for 

dissemination and implementation efforts. Several EPC participants suggested that this 

unfortunate historical precedent caused partners to underestimate the true economic cost of an 

EPC evidence report. A more troubling concern raised by EPCs was scope creep—the tendency 

to expand the work required by the EPC—without due consideration of the costs and the 

implications for timely production. 

Notably, the formulation or nature of the SOW was not perceived by EPC participants as an 

especially important aspect of a successful report, but rather seemed to be a necessary 

component of the EPC activity. The management expert was the only interviewee who 

independently noted the potentially vital importance of a well-crafted SOW in assuring a 

successful technical report. 
14
15

Step 2: Literature Review

Literature Review Methods 

MEDLINE® database searches were conducted in an iterative manner during April-July 

2004 to retrieve articles related to technical contracting literature relating to formulating SOWs 

for the production of policy reports and similar intellectual products, as well as the policy 

literature related to the determinants of a successful policy analysis. Search terms included 

“statement of work,” “policy and research,” “translational research,” “research transfer,” and 

“dissemination and implementation.” No specific key words were required as inclusion criteria; a 

relatively small number of studies exist on the topic, so a “bottom-up” search strategy was 

required. Journal articles were retrieved from diverse fields of study: policy analysis, 

organizational behavior, project management, health services research, environmental research, 

and Federal documents. The reference lists of each article were reviewed in detail to find 

additional articles. 

Two reviewers independently read each article in full text (n = 51 articles, including 4 

Federal documents), evaluated the relevance of retrieved articles, and recorded the main findings 

of each study in a table. Reviewers slated each article in the table for “inclusion” or “exclusion” 

based on the article’s relevance to the topic. Included articles described problems and solutions 

associated with interactions between researchers and decisionmakers (38/51 articles were 

included). The main problems identified in each article and their proposed solutions were 

recorded. A set of 11 key themes emerged early, and key themes were recorded. 

Key themes were: 

1. Timing of the technical report 

2. Communication between researchers and decisionmakers 

3. Aligning goals 

4. Managing expectations 

5. Defining role/responsibility 

6. Understanding cultural differences between researchers and decisionmakers 

7. Using a knowledge transfer expert 

8. Defining key questions 

9. Ensuring non-normativity 

10. Creating a useful product 

11. Addressing ethical, legal, or social implications (ELSI) of a research report 

The few initial articles that had been reviewed before the key themes emerged were rereviewed to specifically identify key themes. A third reviewer read all of the articles deemed 

relevant by the two primary reviewers and resolved any disagreements about the relevance of a 

given article. 
16

In addition, to address the goal of understanding the technical aspects of developing an SOW, 

a standard text entitled Delivering Project Excellence with the Statement of Work by Michael G. 

Martin2 was read by one reviewer, who provided the other two reviewers with a written 

summary. This book describes the development of a successful statement of work from a 

management perspective. Information gathered from this text provided a foundation for thinking 

about statements of work. Other books from project management,3-7 risk analysis,8

 business,9

government contracting,10-13 and qualitative analysis,14-16 were used for background information, 

and written summaries of the relevant information were shared with all reviewers. 

In reviewing the literature, we concluded that the major themes were conceptually similar to 

those uncovered in the informant interview, although they were perhaps articulated a bit 

differently. Indeed, we were able to regroup the 11 themes above into the three areas covered by 

the informant interview: cultural/educational factors (the problem of two cultures—analysts and 

decisionmakers), strategic factors (establishing mechanisms for facilitating a seamless link 

between the analysis and the ultimate application), and resource factors (effectively addressing 

resource issues initially and over time). Thus, for simplicity of exposition, we focus here on 

specific insights proposed in the literature linked to each of the three themes. 

Literature Review Results

Culture/Education Factors 

The differences between researchers and decisionmakers are manifest in multiple 

fundamental ways, ranging from the way they choose to craft questions (as researchable bites vs. 

broader policy questions),17-19 to their attitudes towards time,20 to their perceptions of success.21

The literature suggests that these differences are firmly rooted in professional culture. This 

polarization is described frequently in the literature in terms of researchers valuing “excellence,” 

while decisionmakers prize “relevance” most highly.20,22 On the issue of attitudes towards time, 

researchers have been dubbed “chronophilic,” while decisionmakers are “chronophobic.” 20

Researchers are stimulated by key questions that may take years to answer, because answering 

compelling scientific questions is their life’s work. Researchers are thus “chronophilic,” always 

seeking more time to answer questions in greater depth. Decisionmakers, in contrast, are 

“chronophobic,” chasing time against the next deadline. Decisionmakers have frequent, absolute 

deadlines; if a report is not available for a meeting at which a decision is made, it is useless. 

Assuming that researchers and policymakers articulate key questions in researchable form, 

and bridge misunderstandings during project development, cultural differences may still 

sabotage the final product. Lomas points out that researchers and decisionmakers tend to connect 

more frequently at the end of a research project, when trying to generate a final product.23 

Different professional cultures generate different visions of the end product. Sorian and 

Baugh’s survey of 292 State government decisionmakers revealed that aides to decisionmakers 

may read longer evidence reports, but decisionmakers often need shorter, simpler end products 

than researchers envision.24 It has been suggested in that researchers should be prepared to do 

simple, quick evidence syntheses as well as detailed ones if that is what the partner truly needs.25 

The literature supports the notion of approaching cultural differences through active 

education and integration. Many studies advocated communication between researchers and 

decisionmakers as early as possible, even during the proposal stage.26 In addition to formal 

efforts at educating each other in their languages, tools and values, one recommendation in the 
17

Canadian Health Services Research Foundation (CHSRF) conference report entitled Knowledge 

Transfer: Looking Beyond Health is that researchers should ask about the policy implications of 

the questions they are considering, if they are not explicitly told.27,28 Many forms of interactive 

communication have been suggested to help break down cultural barriers: posting draft 

guidelines on a Web site for public comment, Internet discussion boards, and even “exchanges,” 

in which researchers and policymakers actually trade jobs for a period of time.29 

Strategic Factors 

In addition to concerns raised in the Step 1 interviews, the literature provided additional 

insight into strategic concerns. Eight of the 38 articles we reviewed stressed the importance of a 

“knowledge transfer expert.” Knowledge transfer experts are individuals who are trained to 

promote communication, reduce cultural tensions, guide the development of a conceptual 

framework, manage expectations, and otherwise bridge the gap that researchers and 

decisionmakers have neither time nor incentive to bridge. As a measure of what resources such 

work might require, a study of 265 applied research organizations in Canada revealed that on 

average, 14 percent of organizational budget went to knowledge transfer.30 

One strategic determinant of a successful technical report is clear definition of roles and 

responsibilities. As described by Ross et al., there are several roles that partners can play in the 

research process.31 First, a partner can be a “formal supporter” (financial support only); second, a 

partner can be a “responsive audience;” third, a partner can be an “integral partner.”31 Some 

studies suggest that role and responsibility definition is important because it encourages all 

parties to treat each other as equals.21 

A related issue is the importance of establishing the operational definition of nonnormativity. Typically, EPC reports are intended to inform, not direct, policy. Sorian and 

Baugh’s survey of 292 State government decisionmakers revealed that decisionmakers want to 

see potential implications of various decisions within evidence reports, but not researcher 

opinion.24 The importance of the researcher avoiding the role of policymaker is further 

reinforced by a review of the topic of technical policy analysis by Asher.32 

Resource Factors

The general management/contracting literature is largely focused on balancing work and 

costs, and is an area in which the SOW is noted to be of particular relevance. Two crucial 

resource issues are accounting for project costs when the true costs are not known until work 

begins, and adjusting funding to account for changes in project scope, as when new issues 

become apparent. 

One issue raised in the literature is the critical importance of the due diligence phase.

33,34 The 

due diligence phase allows the contractor to understand the objectives, problems, and constraints 

of the requesting organization. It is customary to pay contractors during the due diligence phase. 

A contracting concept relevant to the production of policy reports is that of the relational 

contract. Relational contracts are “service contracts in which the contract performance 

requirements are continually evolving and accurately specifying the requirements in advance will 

be difficult, if not impossible; the relationship if the government and contractor...will evolve 

cooperatively.”33 Recent trends in Federal government contracting have moved away from a 

strict SOW process that must be followed at all costs. Relational contract theory suggests that 

current Federal contracting practices overemphasize fixed-pricing and bottom line price 
18

competition.33 Relational contracts are well suited to the concept of the Statement of Objectives 

(SOO), a variation of the SOW. The SOO formally requires parties to work together to define 

the contractual requirements. The five-step approach to using an SOO as described by Mather 

and Costello requires the requesting organization (in this case, the partners) to first “conduct 

market research,” which would imply gathering information from all stakeholders who matter to 

the partner, including policymakers and patient groups, if relevant.34 Second, the partner would 

develop an initial SOO and identify constraints that might affect the ultimate deliverable. Third, 

the partner (or knowledge broker) would conduct an initial competition. Fourth, the partner (or 

knowledge broker) would support contractors during the due diligence phase, which should take 

6 weeks or more according to Mather and Costello.34 Finally, a contract award would be made. 

A major cause of resource problems is scope creep, which occurs when the objectives of the 

partner organization change midway through the project. The literature supports the importance 

of controlling scope creep. The Federal government contracting literature addresses the issue of 

scope creep through the mechanism of changes clause. 

Changes clauses constitute a formal process by which scope creep can be managed. As a 

reflection of their importance, the changes clause is one of the most litigated clauses in 

government contracts.13 In fixed-price supply contracts, the changes clause provides that the 

contracting officer may at any time, by written order, “make changes, within the general scope” 

of the contract.13 When a contractor receives a change order, he has 30 days to submit a 

“proposal for adjustment;” the term “equitable adjustment” is the process by which contractors 

recover “the reasonable value for the work under which no agreement was reached on price [in 

the fixed-price contract].”13 

A cardinal change is a unique subset of changes to the initial contract, a modification 

“beyond the original scope” of the contract, which requires the contractor to perform “duties 

materially different from those originally bargained for.” 13 The Court of Claims has ruled that a 

cardinal change is a breach of contract, entitling the contractor to damages. Cardinal changes are 

considered potential attempts at “unauthorized procurement,” according to the theory of unjust 

enrichment,13 meaning that the government (or requesting organization) is overstepping its legal 

bounds by attempting to derive extra benefits from the contractor. 

Finally, a constructive change is a change that the contractor argues he has to make in order 

to meet project objectives, even though he has not received an official written change order. In 

other words, a constructive change is the formal process by which the contractor can change 

scope. It is important that a formal process for change orders be followed, because courts have 

not upheld oral change orders in recent years. If a contractor performs extra work without a 

written change order, he is considered a “volunteer.”13 

Research and development (R&D) contracting within the Federal government is not usually 

done according to fixed-price contracts. According to the Federal Acquisition Requisition (FAR) 

35.005, the R&D SOW should allow contractors “freedom to exercise innovation and creativity,” 

and be “individually tailored.”35 In R&D contracts, “the difficulties in estimating costs... 

normally precludes fixed-price contracting.”13 

 A pragmatic approach for particularly complex projects, termed scope planning, is 

advocated in the PMBOK (Project Management Body of Knowledge).3

 Scope planning starts 

before the project charter is written and includes “the initial definition of constraints and 

assumptions.” A project charter is a “document issued by the project initiator or sponsor that 

formally authorizes the existence of the project, and provides the project manager with the 

authority to apply organizational resources to project activities.”3

 Scope definition also includes 
19

consideration of constraints and assumptions and the development of a work breakdown 

structure (WBS). The WBS is defined as “a deliverable-oriented grouping of project elements 

that organizes and defines the total work scope of the project. Each descending level represents 

an increasingly detailed definition of the project work.” The WBS helps enumerate tasks and 

sub-tasks needed to complete a project, and assigns responsibility for those tasks. It is then 

utilized to develop the SOW.2 Scope verification is the process of “obtaining formal acceptance 

of the project scope by the stakeholders.” Scope change control is a formalized method of 

monitoring changes in scope according to pre-defined criteria for documenting the change and 

ensuring adequate resources to support the change. Partner advantages gleaned from scope 

change control include performance measurement, which are necessary “to assess the magnitude 

of any variations [in scope] that do occur.”3

Summary of Step 2: Literature Review

The literature review regarding the challenges to technical analysis for policymaking 

reinforced the general themes raised in the interviews in Step 1, and these themes fell into the 

categories of cultural/educational factors, strategic factors, and resource factors. The literature 

suggests that cultural differences are deeply rooted, are reflected in the polarizing distinction 

between “excellence” and “relevance,” can persist throughout an entire project if not addressed, 

and, if allowed to persist, can sabotage the entire effort. Strategic factors highlight the 

importance of a conceptual framework, and specific, well understood and communicated 

processes for establishing that framework, as well as roles and responsibilities. Resource factors 

are noted to be crucial and are more extensively addressed in the management and contracting 

literature, stemming from the tendency for technical assessment projects to incur “scope creep.” 

Several potential solutions are offered, related to each category. To address cultural factors, 

approaches include education and integration, using various informal and formal techniques. To 

address strategic factors, suggestions include the identification and funding of a trained 

“knowledge transfer agent,” formal assignment of roles and responsibilities, and establishment of 

the definition of “non-normativity.” Problems related to resource factors can be approached with 

strategies such as institutionalizing a due diligence phase, and use of relational contracts, 

changes clauses, as well as more involved techniques such as scope planning. 
20
21

Step 3: Synthesis—Development of the Current 

Reality Tree Based on the Theory of Constraints 

Synthesis Methods 

Primed with the insights provided in Step 1 and Step 2, we moved to synthesis—that is, to 

drawing up a representation of the current process, focusing on factors constraining the success 

of EPC products. We applied techniques described in the Theory of Constraints (TOC), a wellestablished, explicit, and reproducible means of identifying and addressing complex problems in 

a systematic and comprehensive manner.4 

TOC is similar to traditional process improvement techniques, with the exception that it is 

designed to accommodate complex processes, such as provision of health services, which, unlike 

many industrial processes, are non-linear. TOC understands that processes do not function in 

isolation, but are part of a larger, intertwined system. Therefore addressing root causes at process 

levels does not result in sustainable solutions. Hence, as an alternative to investigating and 

addressing problems (or one root cause) at a process level, TOC demands a much more involved 

investigation into the relationship between several root causes of all the failures of processes in 

the system. It then forces the strategic thinker to explore the one root cause that leads to most 

undesirable effects (UDEs), and labels it as the core problem. TOC proposes to focus attention 

on this core problem. The core problem is called the “constraint” in TOC terminology. This 

constraint prevents an organization (e.g., an EPC) from reaching its goal (namely, developing a 

“useful” evidence report). Because the TOC thinking strategy has thought through other 

processes in a system that can affect the root causes of problems, its change strategies produce 

more sustainable results. 

TOC proposes that this detailed investigation can be best performed through five systematic 

exercises. Each of these exercises requires construction of corresponding logic “trees.” Most of 

these logic trees can be used as stand-alone tools, depending upon the nature of the questions 

under consideration. 

We based our analysis for the current problem on the first of these logic trees, the Current 

Reality Tree (CRT). The CRT informs about the existing situation (what to change); this is 

followed by the design of new reality through the construction of a Future Reality Tree (what to 

change to). This involves introducing new ideas—strategies for change. These change strategies 

are called “injections” in TOC terminology. Identification of these injections and creating the 

Future Reality Tree were the objectives of Step 4. 

Figure 1 illustrates a template of the CRT. It represents the various undesirable effects 

(UDEs) associated with the production and use of evidence reports, as well as a series of 

hypothesized causal relationships. At the top of the tree is the most important UDE, the UDE that 

is fundamental to the entire process. TOC proposes that strategic thinkers then explore other 

UDEs and their root causes, the relationship between them, and the core constraint (one root 

cause) that leads to the fundamental UDE. 

We utilized the interview results, and the literature search to organize our thoughts to 

develop the CRT for the SOW process. For this purpose, we did the following: 

• Identified the most relevant UDE. 

• Explored other UDEs. 
22

• Listed a set of root causes that led to the UDEs. 

• Explored the relationship between the UDEs and the root causes, so that one (or two) 

core constraint (s) could be identified. 

Figure 1. Template of a Current Reality Tree

Synthesis Results

Our interviews with EPC researchers, partners, and AHRQ allowed us to identify a common 

UDE, described from their perspectives: 

• For researchers a report was considered “unsuccessful” when the partners failed to use 

the report and complained that the report did not meet their requirements. 

• Partners considered a report successful if it met their needs. 

• AHRQ staff believed that even a technically excellent EPC report was unsuccessful if it 

was not useful. 
23

Therefore we concluded that that a report that does not get used is unsuccessful and among all 

UDEs, this effect is deemed fundamentally undesirable. 

We formulated the CRT (Figure 2) working from this fundamental UDE at the top through 

proximate causes, and finally root causes. For this purpose we made several assumptions: 

• The partner had some potential use for a report for which they could articulate needs. 

This precludes the possibility that a report is requested for no reason at all. This led to the 

principal reason a report would not get used—that the report does not meet those needs. 

We distinguish here between: 

o Needs—the ultimate goal of the information seeking activity 

o Objectives—how the partner articulates their understanding of their needs, e.g., in the 

form of key questions 

o Specifications—how the work is described for purposes of communicating with the 

contractor in the form of a Task Order or Statement of Work 

• EPCs have expertise sufficient to produce a technically perfect report given adequate 

resources and specifications. This assumption derives from the vetting process used by 

the AHRQ in selecting EPCs for the program. 

In the next stage we extracted a list of root causes from our interviews: 

• Partner does not know how to conceptualize and articulate needs, objectives, and 

specifications required for the desired report. 

• Specifications for a report are not initially sufficient. 

• Specifications for a report change “mid-stream.” 

• Partner’s initial objectives regarding the final report are clarified late in the process. 

• Partner’s needs change late in the process. 

• Volume of anticipated work required for the completion of the report is initially 

underestimated by the EPCs. 

• True cost and time requirement of the project are underestimated. 

• Project cannot be accomplished within the timeframe allotted because new costs emerge 

later during the course of the project. 

• Budget and timelines cannot be revised. 

Following TOC methods, we explored the relationship between these root causes in order to 

identify one (or two) major problems that lead to the rest, and ultimately result in a report not 

meeting the partner’s needs, and therefore not being used (Figure 1). This could be the result of 

causes in one of two categories: (1) resources (budget or time), and (2) specifications 

(stipulations of content or process for development of content). 

Whether a resource issue relates to budget or to time schedule, process failure can result from 

underestimation of resource requirements at the outset of the project, or because unanticipated 

requirements emerge later. Given the assumption about the capabilities of the EPCs, the 

proximate cause is insufficiency of the initial specifications. 
24

The specifications can be insufficient because they do not provide an accurate guide to the 

ultimate volume or nature of work (e.g., number of studies to evaluate or specific techniques of 

synthesis to use) at the outset, or because the specifications change after the project is well 

underway. The latter problem (referred to in the interviews and literature as “scope creep”) is the 

result of partners stating objectives at the outset that do not actually reflect their needs, or the 

partner’s needs change due to external forces (e.g., new science, new political or other practical 

considerations). Insufficiency of initial specifications and scope creep are linked to the inability 

of the partners to conceptualize and articulate their needs, objectives and specifications (referred 

to as the “conceptual framework”). 

From this list and from Figure 2 we identified four basic “root causes”: 

1. Partner does not know how to conceptualize and articulate needs, objectives, and 

specifications (the SOW). 

2. Partner’s needs change late in the process. 

3. Insufficiency of specifications is never appreciated. 

4. Budget and timeline cannot be changed in response to down-line changes in project 

specifications. 

Of these, the first (partner does not know how to conceptualize and articulate needs, 

objectives, and specifications) rates as a core problem or the constraint that prevents an EPC 

from achieving its goal because it affects most other problems. This is the reason why 

specifications are insufficient in the first place, which leads to an underestimation of the volume 

of work, results in new cost and time requirements that lead to delays in the report, and finally 

results in the report not being used. If this one problem is solved, it is expected that most other 

issues can be resolved. Note that two other root causes that are not technically core constraints 

could also independently impact the success of the SOW. These causes are: 3. insufficiency of 

specifications is never appreciated and 4. budget/timeline cannot be revised. 

The second root cause, partner’s needs change late in the process, is isolated from the rest of 

the CRT because resolving this root cause is beyond the span of control of the EPC. TOC 

suggests that we eliminate it from our logical thinking process. 
25

Figure 2. Current Reality Tree for the EPCs 
26
27

Step 4: Identify Promising Solutions for the Core 

Constraints to Develop the Future Reality Tree Based 

on the Theory of Constraints

Solutions Methods

The objective here is to identify promising solutions/change strategies to address the root 

causes that lead to a report that is not used, and to depict a desired future through a Future 

Reality Tree. The Future Reality Tree is one of the five logic trees that TOC proposes. The UDE 

from the CRT (e.g. “Report is not used) is the desired effect (DE) of the Future Reality Tree 

(“Report is used”), and it results from one or more proposed injections (e.g. “establish on 

ongoing relationship between partner, EPC, and AHRQ starting early on”). 

Note that, as described in the Introduction, we are particularly interested in strategies that can 

be articulated in the SOW while including approaches that need not be incorporated into a SOW 

document. These approaches however must be consistent with the concept of SOW and can 

function as clear principles guiding the EPC process. 

The result of the Step 3, the CRT exercise, was the identification of the core constraint 

(partner does not know how to conceptualize and articulate needs, objectives, and 

specifications). Our interviews and literature search allowed us to explore possible change 

strategies and promising solutions to address this core constraint. In addition, at the 2004 annual 

EPC directors meeting we presented the CRT and solicited solutions (injections in TOC 

terminology). We listed all these solutions, chose the most promising ones from our discussions 

with the EPC directors (and from Steps 1 and 2), and developed a Future Reality Tree (Figure 3). 

Solutions Results

The interviews, literature search, and input from the directors’ meeting identified several 

potential injections/change strategies: 

• Establish a formally supported relationship between partners, researchers and the EPC 

early in the process. 

• Involve a knowledge transfer expert. This expert will enable communication, reduce 

cultural tensions between researchers and partners, manage expectations, and guide the 

development of the SOW. 

• Ensure frequent and ongoing communication, as well as the development of long-term 

partnerships between researchers and decisionmakers, in order to develop cultural 

sensitivity. 

• Consider the current EPC SOW a “proposal SOW.” This rough SOW would be utilized 

during the competitive bidding situation, while a revised SOW is created with input from 

the EPC, AHRQ, and partners within the first 30 days after contract award. 

• Provide funding specifically for the EPCs to perform preliminary research in order to 

estimate the time and budgetary resources required for the stated scope of work (e.g. 

based on volume of literature, nature of expected end product). 
28

• Utilize a changes clause in the SOW; in more complex tasks include formal Scope 

Change Control. 

The relation between the injections, and the root causes they could potentially attack are 

illustrated in the following table:

Solutions (injections) that can address root causes of EPC-SOW problems 

Injection Root causes that the injection can address 

Establish an ongoing relationship 

between partners, EPC, and AHRQ 

early in the process 

1. Partner does not know how to conceptualize a SOW 

2. Specifications for a report are not initially sufficient 

Involve a knowledge transfer expert 1. Partner does not know how to conceptualize a SOW 

2. Specifications for a report are not initially sufficient 

Concept of “proposal SOW” that is 

revised with EPC-input 

1. Partner does not know how to conceptualize a SOW 

2. Specifications for a report are not initially sufficient 

Provide funding for EPCs to 

perform preliminary research 

1. Volume of anticipated work is under-estimated by the EPCs 

2. True cost and time requirement of the project are underestimated 

Utilize a changes clause 1. Specifications for a report change mid-stream 

2. Partners needs change late in the process 

3. Budget and timeline cannot be changed in response to changes in 

project specifications 

With these concepts of UDEs, root causes, and injections that lead to converting a UDE to a 

DE, we constructed the Future Reality Tree for the EPC-SOW process (Figure 3). 

• The two injections (establish an ongoing relationship between partners, EPC, and AHRQ 

early in the process, and utilize a changes clause) were deemed to be the most practical 

means to exert impact on the SOW process by addressing the core constraint, partner 

does not know how to conceptualize and articulate needs, objectives and specifications of 

the SOW.

• The injections (change strategies) lead to the ability of a partner to conceptualize and 

articulate needs, objectives and specifications of the SOW (the core constraint is 

addressed by the injection). 

• This ultimately results in the production of a successful EPC report that meets partner 

needs and therefore will be used (opposite of the UDE, the DE). 
29

Figure 3. Future Reality Tree of the SOW process for EPCs 

Partner does not know how to conceptualize and articulate needs, objectives and specs of SOW 

Report does not get used

CURRENT REALITY TREE 

(concise version) 

UDE 

Core constraint 

Report gets used

Establish on ongoing relationship between partner, EPC and AHRQ starting early on 

Partner is educated about effective SOW processes 

Partner has an exrationale for beinproactive in the S

Include SOW (change 

Partner conceptualizes and articulates needs, objectives and specs of SOW

Report meets partner needs

FUTURE REALITY TREE 

Opposite of the UDE: the DE 

Injection 

The core constraint is addressed by the injection (the change strategy) 
30
31

References

1. Evidence-based Practice Centers. Overview. 

AHRQ Publication No. 03-P006, March 2003. 

Agency for Healthcare Research and Quality, 

Rockville, MD. Available at: 

http://www.ahrq.gov/clinic/epc/. Last accessed on 

March 15, 2005. 

2. Martin, MG. Delivering project excellence with 

the SOW. Vienna, VA: Management Concepts; 

2002. 

3. Project Management Institute. A guide to the 

Project Management Body of Knowledge 

(PMBOK). Pennsylvania: Project Management 

Concepts; 2000. 

4. Dettmer HW. Goldratt’s Theory of Constraints: 

A systems approach to continuous improvement. 

Milwaukee: ASQ Quality Press; 1997. 

5. Cleland DI, King WR. Management: a systems 

approach. New York: McGraw-Hill; 1972. 

6. Kerzner H. Project management: a systems 

approach to planning, scheduling, and 

controlling. New York: John Wiley; 2001. 

7. Loring R, Kerzner H. Proposal preparation and 

management handbook. New York: Van 

Nostrand Reinhold Publishers; 1982. 

8. Stern PC, Fineberg HV. Understanding risk: 

informing decisions in a democratic society. 

Washington, DC: National Academy Press; 1996. 

9. Walker DM, Walker T, Schmitz J. Doing 

business internationally: the guide to crosscultural success. New York: McGraw Hill; 2002. 

10. Cibinic J, Nash RC. Administration of 

government contracts. Washington, DC: George 

Washington University Press; 1995. 

11. Cibinic J, Nash RC. Formation of government 

contracts. Washington, DC: George Washington 

University Press; 1998. 

12. Keyes WN. Government contracts: under the 

Federal Acquisition Regulation. St. Paul, MN: 

Thompson/West; 2003. 

13. Keyes W. Noel. Government contracts: in a 

nutshell. St. Paul, MN: Thompson/West; 1990. 

14. Oppenheim, AN. Questionnaire design, 

interviewing and attitude measurement. London: 

Pinter Publishers; 1992. 

15. Moser CA, Kalton G. Survey methods in social 

investigation. New York: Basic Books; 1972. 

16. Bernard HR. Social research methods: qualitative 

and quantitative approaches. Thousand Oaks, 

CA: Sage Publications; 2000. 

17. Banta HD, Andreasen PB. The political 

dimension in health care technology assessment 

programs. Int J Technol Assess Health Care 

1990;6(1):115-23. 

18. Walshe K, Rundall TG. Evidence-based 

management: from theory to practice in health 

care. Milbank Q 2001;79(3):429-57, IV-V. 

19. Lilford R, Jecock R, Shaw H, et al. 

Commissioning health services research: an 

iterative method. J Health Services Research 

Policy Jul 1999;4(3):164-7. 

20. Frenk J. Balancing relevance and excellence: 

organizational responses to link research with 

decisionmaking. Soc Sci Med Dec 

1992;35(11):1397-404. 

21. Canadian Health Services Research Foundation. 

Conference Report from Can Health Serv Res 

Fdn Ann Invitational Workshop. March 2002. 

Productive Partnership. Available at: 

http://www.chsrf.ca/knowledge_transfer/pdf/200

2_workshop_report_e.pdf. Last accessed: March 

10, 2005. 

22. Goering P, Butterill D, Jacobson N, et al. Linkage 

and exchange at the organizational level: a model 

of collaboration between research and policy. J 

Health Services Research Policy Oct 2003;8 

Suppl 2:14-9. 

23. Lomas J. Connection research and policy. 

ISUMA-Canadian Journal of Policy Research 

2000; Vol 1(1),pp140-4. 

24. Sorian R, Baugh T. Power of information: 

closing the gap between research and policy. 

When it comes to conveying complex 

information to busy policy-makers, a picture is 

truly worth a thousand words. Health Affairs 

Mar-Apr 2002;21(2):264-73. 

25. Alberta Heritage Foundation for Medical 

Research. A study of the impact of 2000-01 HTA 

Products. Jan 2002. Available at: 

http://www.inahta.org/Reports.asp?name=Conten

t12/Dokument/4+AHFMR+Impact.pdf. Last 

accessed: March 10, 2005. 
32

26. Eagar K, Cromwell D, Owen A, et al. Health 

services research and development in practice: an 

Australian experience. J Health Services 

Research Policy Oct 2003;8 Suppl 2:7-13. 

27. Canadian Health Services Research Foundation; 

Central West Planning Information Network; 

Centre for Health Economics and Policy 

Analysis; Health Evidence, Linkage and 

Application Network (HEALNet); Institute for 

Clinical Evaluative Sciences; Institute for Work 

and Health. Conference Report from Knowledge 

Transfer: Looking beyond Health. Toronto, 

October 26-27, 2002. Available at: 

http://www.chsrf.ca/knowledge_transfer/pdf/ktra

nsfer_e.pdf. Last accessed: March 10, 2005. 

28. Canadian Research Transfer Network; Health 

Resarch Transfer Network of Alberta. 

Conference Report from Knowledge Transfer: in 

Health Alberta, Oct 24-25, 2002. Available at: 

http://www.chsrf.ca/knowledge_transfer/pdf/200

2_workshop_report_e.pdf. Last accessed: March 

10, 2005. 

29. Bartunek J, Trullen J, Bonet E, et al. Sharing and 

expanding academic and practitioner knowledge 

in health care. J Health Services Research Policy 

Oct 2003;8 Suppl 2:62-8. 

 

30. Lavis JN, Robertson D, Woodside JM, et al. 

Knowledge transfer study group. How can 

research organizations more effectively transfer 

research knowledge to decisionmakers? Milbank 

Q 2003;81(2):221-48, 171-2.

31. Ross S, Lavis J, Rodriguez C, et al. Partnership 

experiences: involving decision-makers in the 

research process. J Health Services Research 

Policy Oct 2003;8 Suppl 2:26-34. 

32. Ascher W. Forecasting: an appraisal for policymakers and planners. Baltimore: The John 

Hopkins University Press; 1978. 

33. Cole PS. How to write a statement of work. 

Vienna, VA: Management Concepts; June 2003. 

34. Mather C, Costello A. An innovative approach to 

performance-based acquisition: Using a SOO. 

Acquisition solutions, Inc. May 1, 2001. 

Available at: 

http://www.acqsolinc.com/docs/adv0501.pdf. 

Last accessed: March 10, 2005. 

35. Federal Acquisition Regulation, FAC 24, July 19, 

2004. Available at: http://www.arnet.gov/far/. 

Last accessed: March 10, 2005.
33

Appendix 

Technical Description of the Statement of Work 

The Statement of Work (SOW) is most fundamentally a contractual agreement 

stipulating how a job, in this case a research study, will be done. The Statement of Work 

establishes expectations and mediates the relationship between contractor and sponsor (in 

the case of the EPC program, between research team and AHRQ, which contracts on 

behalf of a “partner”). According to Michael G. Martin, author of Delivering Project 

Excellence with the Statement of Work,1 “the single most significant cause of project 

failure is the lack of a clearly defined and detailed statement of work.” Martin identifies 

the first formal SOW written in modern history as the 1908 Army Signal Corps document 

entitled “Agreement and Specification for the Wright Brothers’ Heavier-than-Air Flying 

Machine,” (though he admits that Noah’s biblical instructions for building the ark 

essentially constituted a Statement of Work). The Department of Defense and the 

National Aeronautics and Space Administration (NASA) have published handbooks that 

describe the step-by-step process of writing a Statement of Work.2-4 NASA4

 defines the 

SOW as a document that: 

. . . . describes the work to be performed or the services rendered; defines 

the respective responsibilities of the Government and the contractor; 

serves as a basis for contractor response, evaluation of proposals and 

source selection; and ultimately provides an objective measure so that both 

the Government and the contractor will know when the work is 

satisfactorily completed and payment is justified. 

Definitions from the Department of Defense and the project management literature 

have much in common with NASA’s definition, but NASA’s definition was selected for 

repetition here because NASA specifically addresses the responsibilities of both the 

contractor and the client (the Government). The SOW is not just a work plan detailing the 

services that the contractor (or researcher) will offer; rather, it lays the groundwork for a 

relationship between contractor and client. 

All Federal agencies use the Federal Acquisition Regulation (FAR) to acquire needed 

supplies and services, and the FAR defines the SOW. One subsection of the FAR, 

subsection 37.602-1-Statements of Work, is given below (FAC 24, effective July 19, 

2004). 

FAR Subsection 37.602-1: Statements of work.5

 

(a) Generally, statements of work shall define requirements in clear, concise 

language identifying specific work to be accomplished. Statements of work 

must be individually tailored to consider the period of performance, 

deliverable items, if any, and the desired degree of performance flexibility 

(see 11.106). In the case of task order contracts, the statement of work for the 

basic contract need only define the scope of the overall contract (see 

16.504[a][4][iii]). The statement of work for each task issued under a task 

order contract shall comply with paragraph (b) of this subsection. To achieve 
34

the maximum benefits of performance-based contracting, task order contracts 

should be awarded on a multiple award basis (see 16.504[c] and 16.505[b]). 

(b) When preparing statements of work, agencies shall, to the maximum extent 

practicable— 

(1) Describe the work in terms of “what” is to be the required output rather 

than either “how” the work is to be accomplished or the number of 

hours to be provided (see 11.002[a][2] and 11.101); 

(2) Enable assessment of work performance against measurable 

performance standards; 

(3) Rely on the use of measurable performance standards and financial 

incentives in a competitive environment to encourage competitors to 

develop and institute innovative and cost-effective methods of 

performing the work; and 

(4) Avoid combining requirements into a single acquisition that is too 

broad for the agency or a prospective contractor to manage 

effectively.5

 
35

References for Appendix 

1. Martin, MG. Delivering project 

excellence with the SOW. Vienna, VA: 

Management Concepts; 2002. 

2. Department of Defense. Handbook for 

preparation of statement of work 

(SOW). MIL-HDBK-245D. April 3, 

1996. Available at: 

http://www.acqnet.gov/Library/OFPP/B

estPractices/pbsc/library/DODhandbook

.pdf. Last accessed: March 10, 2005. 

3. Department of Defense. Handbook for 

work breakdown structure. MILHDBK-881. January 2, 1998. Available 

at: 

http://www.acq.osd.mil/pm/currentpolic

y/wbs/mil_hdbk_881/mil_hdbk_881.ht

m. Last accessed: March 10, 2005. 

4. NASA. Statement of Work (SOW): 

Guidance for writing work statements. 

NASA NPG 5600.2B. December 1997. 

Available at: 

http://www.hq.nasa.gov/office/procure

ment/newreq1.htm. Last accessed: 

March 10, 2005. 

5. Federal Acquisition Regulation, FAC 

24, July 19, 2004. Available at: 

http://www.arnet.gov/far/. Last 

accessed: March 10, 2005. 
36