Chapter 2. Methodology

	The management of acne was a topic nominated to the Agency for Healthcare Research and Quality (AHRQ, then called the Agency for Health Care Policy and Research [AHCPR]) by the

American Academy of Dermatology (AAD) and the American Academy of Pediatrics (AAP). In

October 1998, AHRQ awarded a contract to the Johns Hopkins Evidence-based Practice Center

(EPC) to prepare an evidence report on this topic. With the assistance from the partners from

AAD and AAP, the Johns Hopkins EPC established a team and work plan to develop a report

that would identify and synthesize the best available evidence on the management of acne. The

project consisted of recruiting technical experts, identifying the patient population, formulating

and refining the specific questions, performing a comprehensive literature search, summarizing

the state of the literature, constructing evidence tables, and submitting the report for extensive

peer review.

Recruitment of Technical Experts

	Experts were sought who could provide content and/or methodological guidance. The group

of experts included representatives from the project partners AAD and AAP, as well as from the

American Pharmaceutical Association (AphA), Society for Investigational Dermatology (SID),

and the Skin Collaborative Review Group (CRG) from the Cochrane Collaboration. Their input

was sought through ad hoc correspondence as well as through more formal requests for feedback

during the project.  Specific requests for feedback were made for key decisions throughout the

project, such question refinement, completeness of literature search, treatment and acne severity

classification, and format and layout of evidence tables and synthesis of literature.

	Additional representation of AAD, AAP, American Academy of Family Physicians (AAFP),

as well as from the Society for General Internal Medicine (SGIM), SID, and Society for

Adolescent Medicine (SAM) joined the technical experts as peer reviewers for the report. These

reviewers included dermatologists, pediatricians, and generalists (Appendix A).

Patient Population

	The target population for this report is all patients with acne who do not have complicating

comorbidities such as endocrinopathies. In consultation with the partners and technical experts,

we decided to exclude evidence from populations with the following conditions: acne fulminans,

acne necroticans, acne venenata, rosacea, and chloracne. We believed that to include

endocrinopathies would necessitate an examination of underlying medical conditions that was

beyond the scope of this report. The search and synthesis of evidence was not limited by age,

gender, or any other patient characteristic, though synthesis includes an analysis by these

characteristics, where possible.

	The report sought evidence concerning all primary care and dermatology settings. In

addition, self-treatment is an important modality for acne therapy. Therefore, the community-

based setting in which a consumer of health care acts without the involvement of a health care

provider was also considered.

Questions

	Questions were identified through discussions among the team members, preliminary

searches of MEDLINE®, a search of the MicroMedex pharmaceutical database, and consultation with our technical experts. We also reviewed over 14,000 consumer-generated questions posted

to a major consumer health Internet Web site (InteliHealthTM) for queries about acne. The

questions identified through this process formed the basis of a brief questionnaire distributed to

the technical experts (Appendix B). The experts were asked to rate the importance of each

question, to provide a sense of the data available to address each question, and also to provide

any additional comments or suggestions.

	Based on the results from the questionnaire, we developed questions in factorial format to

embrace the various patient treatments, outcomes, patient characteristics, practice settings, and

care providers of interest (Appendix C). The factorial formatted questions, as well as the

proposed exclusions, were distributed to the technical experts for feedback (Appendix D).

From the factorial questions, the technical experts requested that the evidence report address the

following questions:

1.	What treatments are effective in the management of acne?

2.	What is the responsiveness of acne to first-line, second-line, and third-line (referral) treatments?

3.	What are the side effects of treatment?

	The following secondary questions were also to be addressed as the evidence allowed: 

•	What baseline characteristics raise the likelihood of acne?

•	What care do patients provide for themselves at home?

•	How effective is initial home care?

•	What is the presentation of acne to physicians?

•	How compliant are patients with their treatment?

•	What is the residuum of untreated acne? Of treated acne?

•	What is the quality of life (psychological morbidity) in untreated acne? In treated acne?

•	What is the financial cost of therapy?

Causal Pathway

	The accumulation of data alone is not sufficient for decisionmaking. The evidence needs to

be organized into meaningful units, and the units must be organized to aid in making decisions.

To this end, we structured a causal model, using the conventions of an influence diagram23 (Figure 1). The diagram is an abstraction of a decision tree, and, when fully specified, can derive

the flowchart of a practice guideline. For instance, rather than specifying all possible self-care

options, the diagram simply shows that the choice of self-care is influenced by a patient’s

baseline characteristics which, in turn, influence the possibility and choice of initial medical care. It was developed through structured interviews with two technical experts and a review of the

literature, and it was critiqued by the other technical experts.

	Preliminary definitions and assessments were made of the nodes (states or variables) of the

influence diagram:

	•	Patient characteristics—These are characteristics such as race, gender, and skin type. 	

•	Self-care—Refers to over the counter treatments and medications, such as benzoyl peroxide. This also includes self-referral to a medical provider.

•	Initial acne assessment—Refers to the acne present at initial assessment. The arrow from baseline characteristics suggests that this acne severity is influenced by baseline

characteristics, although our initial review of the evidence suggested that few

investigators go beyond lesion count and type in making a severity assessment (see

Chapter 3).

•	Initial medical care—Represents all possible prescription and non-prescription treatments.

•	Acne reassessment—The same as initial acne assessment, but later in time. An arrow is drawn from the initial acne assessment node, suggesting that response to treatment (from

initial medical care) is influenced by the initial acne status (or baseline characteristics).

•	Initial compliance—Acknowledges that success depends on patient compliance and that initial medical choices must be made keeping compliance in mind.

•	Followup medical care—Refers to the provider’s options after assessment, including continuing present therapy, changing to another treatment, or referral.

•	Followup compliance—Similar to initial compliance.

•	Followup acne reassessment—Similar to acne reassessment.

•	Further medical care—Focuses on referral if acne reassessment shows no improvement. Refers to the provider’s option after assessment, including continuing present therapy,

changing to another treatment, or referral. We left the full definition open, hoping to find

evidence in the literature that would help us to define the therapeutic choices.

•	Acne residual—This is the long-term dermatological morbidity of acne, with or without treatment.

•	Quality of life—Outcomes such as self-esteem and social and familial relationships.

•	Cost—Includes costs related to loss of productivity in school or work, as well as medical costs. 

•	Other morbidity—Other morbidity outcomes, such as the presence of lesions themselves, and the possible effects of treatment. Possible treatment effects include sun sensitivity

and, of the most concern, more serious side effects such as those related to isotretinoin

use.

	Influence diagrams terminate in a utility node that integrates the outcomes, their

probabilities, and the decisionmaker’s valuation of preference for those outcomes. We plan to

leave that integration to the users of our final evidence synthesis for creating their guidelines.

Literature Search Methods

	Searching the literature included the steps of identifying reference sources, formulating a

search strategy for each source, and executing and documenting each search.

Sources

	Several literature sources were used to identify all studies potentially relevant to our study

questions. The first source that we used was the CENTRAL database of the Cochrane

Collaboration. This is a database of all clinical trials (primarily randomized controlled trials and

controlled clinical trials) identified through the searching efforts of the Cochrane Collaboration.

The CENTRAL database includes search results from many electronic databases, including

MEDLINE® and EMBASE®, as well as results from the hand searching of more than 1,000 journals, for all publication years starting in 1948. The CENTRAL database also includes the

specialized register of controlled trials developed by the Cochrane Skin Collaborative Review

Group. The Skin CRG has completed extensive searching of electronic databases including

MEDLINE®, EMBASE®, Current Contents® in Dermatology, and the American Academy of Dermatology trials register. In addition, members of this CRG hand search a number of key

dermatology journals such as Dermatology and Clinical and Experimental Dermatology. The CENTRAL database is made available on The Cochrane Library, which is issued quarterly. Both Issue 4 of the 1998 The Cochrane Library and Issue 1 of the 1999 The Cochrane Library were  searched.

	Our second source was MEDLINE®, or MEDlars onLINE, the database of bibliographic citations and author abstracts from approximately 3,900 current biomedical journals published in

the United States and 70 foreign countries, which dates back to 1966. MEDLINE® was accessed through PubMed, the Internet access to MEDLINE® provided by the National Library of Medicine (NLM), which also includes the PREMEDLINE database of basic citation information

and abstracts of full records before they are prepared and added to MEDLINE®. The searching of PubMed helped to confirm the completeness of the search conducted with the Cochrane

Collaboration's CENTRAL database and also helped to identify new studies published while we

were working on the evidence report. In addition, for the questions where there were few or no

controlled clinical trials (i.e., those related to the costs of acne), PubMed was used to identify

other types of studies.

	Internet GratefulMed, provided as a Web-based service by the NLM, was used to access

OLDMEDLINE. OLDMEDLINE contains citations published in the 1960 through 1965

Cumulated Index Medicus and covers the fields of medicine, preclinical sciences, and allied

health sciences. We felt it important to include OLDMEDLINE as some of the key articles

concerning treatment effectiveness, for instance of benzoyl peroxide, may have been published

prior to the MEDLINE® start date of 1966. Internet GratefulMed was also used to access HealthSTAR. This electronic database combines the former HEALTH (Health Planning and Administration) and HSTAR (Health Service/Technology Assessment Research) databases and

includes citations from 1975 to present.

	Using the access provided by the Johns Hopkins Welch Medical Library, the PsycINFO® and CINAHL® databases were also searched. PsycINFO® covers the professional and academic literature in psychology and related disciplines, including medicine, psychiatry, nursing,

sociology, education, pharmacology, physiology, linguistics, and other areas (including

literature) from 1887 to present. CINAHL® is a multidisciplinary database covering the nursing, allied health, biomedicine, and consumer health literature from 1982 to present. 

	Two final steps were taken to ensure a comprehensive literature search. A database of

reference material, identified through the electronic searching, through discussions with experts,

and by the article review process, was created in the reference management software, ProCite®. A listing of titles and abstracts from this database, the Acne References Database, was reviewed by

the investigators to identify key articles. The reference lists from these key articles were then

examined to identify any additional articles for inclusion in the Acne Citations Database. Finally,

a listing of all identified controlled trials was distributed to our technical experts, and they were

asked to identify any additional articles of which they were aware that should be considered for

inclusion in the review process.

Search Terms and Strategies

	The search strategies were designed to maximize sensitivity and were developed in

consultation with team members. A core search strategy for identifying evidence from the

CENTRAL database was developed. PubMed was searched by combining the core search

strategy with phases 1 and 2 of a search strategy developed for the optimal identification of

controlled clinical trials from PubMed.24 Search strategies were also modified, as needed, based on the source being accessed, with separate strategies developed for OLDMEDLINE, CINAHL®, and PsycINFO®. Because results were limited on this topic, a supplemental search was conducted on PubMed for the retrieval of cost-related articles. (See Appendix E for search strategies.)

Organization and Tracking of Literature Search

	The results of the searches of electronic databases were downloaded and, using the

duplication check in the bibliographic software ProCite®, articles not previously retrieved were included in the Acne Citations Database. This ProCite® database was used to store citations and to aid in the tracking of search strategies and sources. The use of this software allowed citations

and abstracts to be directly imported into a database as well as catalogued with important

attributes. The results of the abstract review process were also tracked using ProCite®. 

Abstract Review

	Specific inclusion and exclusion criteria were applied at each of three levels of review, with

criteria becoming more stringent as the process moved from use of specific search terms to the

review of abstracts to the review of articles. After identifying a citation, its title and abstract were

reviewed, and articles were included or excluded from the article review on this basis.

Identification of Inclusion and Exclusion Criteria

	During the abstract review process, the emphasis was placed on identifying all articles that may possibly have original data pertinent to the questions.   As previously described, the

technical experts were consulted during the development of inclusion and exclusion criteria.  

	In evaluating titles and abstracts, the following criteria were used to exclude articles from

further consideration:

•	Article does not address management of acne.

•	Article does not include human data.

•	Article addresses a topic excluded from this evidence report:

	–	chloracne

	–	rosacea

	–	acne venanta

	–	acne fulminans

	–	acne necroticans

•	Article contains no original data.

•	Article is not in English.

	Articles excluded with criterion “no original data” included those that were duplicate

publications of the same trial. In those cases, the article with a more comprehensive reporting of

the data was kept (if otherwise eligible).

	It is recognized that the decision to exclude non-English language reports of trials is a

limitation of this report because it means the possible exclusion of potentially high quality and

highly relevant reports of trials.  Time and other resources, however, prevented this project from

searching, retrieving, translating, and then reviewing non-English reports of trials. 

	The abstract review process was also used to classify the articles by topic and to identify the

study design. For those articles without an abstract, the abstract review process was performed

using the entire article. 

Abstract Review Process

	Titles and abstracts for all articles retrieved by the literature search were printed on an abstract form and distributed to two reviewers (Appendix F). When reviewers agreed that,

because of insufficient information, a decision regarding eligibility could not be made, the full

article was retrieved and then reviewed. The principal investigators reviewed abstracts when

abstract reviewers disagreed regarding eligibility of the article.

	The results of the abstract review process were entered into the Acne Citations Database.

Articles deleted through the abstract review process were tagged with reason for exclusion and

removed to the Acne Deletions Database. The maintenance of this database allowed for the

ongoing search results to be compared to already retrieved citations (in Acne Citations and Acne

Deletions Databases) for the identification of newly retrieved articles and also allowed for the

efficient tracking of the abstract review results.

Article Review

	The purpose of the article review was to confirm relevance of each article to the questions at

hand, to determine methodological characteristics pertaining to study quality, and to collect

evidence pertaining to the questions.  Where articles included reports of more than one trial,

reviewers were instructed to complete the relevance assessment (i.e., comparison to inclusion and

exclusion criteria), quality assessment, and data abstraction for each trial separately.

Quality Assessment and Data Abstraction

	Forms were developed to confirm eligibility for full article review, assess study

characteristics, and abstract the relevant data to address the study questions. The forms were

developed through an iterative process including the review of forms used for previous EPC

projects, discussions among team members and experts, and through pilot testing.

	Three forms were developed and color-coded to aid reviewers and data entry personnel

(Appendix G).

Study Characteristics and Quality

	The first form completed for all trials comprised three sections. The first section included the same exclusion criteria used during the abstract review phase. In this way, reviewers could

confirm the eligibility of the trial before proceeding with the full article review. The second

section (questions 1 to 5) contained general study characteristic questions such as the number of

treatment groups, number of sites, and location of trial. A classification of study design was also

included in this section. Once the decision was made to limit full article review to controlled

trials, this question was added to the previous section in that a trial classified as any study design

other than a controlled trial would not be reviewed. The final section (questions 6 to 26)

contained questions designed to provide an assessment of study quality. The questions were

designed to assess characteristics such as research design, treatment allocation, and blinding, as

well as the potential for bias and generalizability. These questions allowed for the identification

of methodological strengths and weaknesses used in the evidence tables and summary of

evidence.

Arm Characteristics

	The characteristics of each arm or group in the trial were assessed on the second form. The first section gathered information concerning the treatment or intervention. For example,

treatment name and dose were noted. The following pages abstracted information concerning the

patients and these 15 questions were completed for each arm and/or for the entire trial, as

possible. It was in this section that baseline characteristics such as age and gender, as well as

source of care and phase of care, were noted.

Outcomes

	Two forms were developed for outcomes: one for discrete outcomes and one for continuous outcomes. These one-page forms gathered definitions and data for each outcome. For discrete

outcomes, data collected included the categories and raw numbers for each measurement, while

for continuous outcomes the means and standard deviations or standard error were collected.

Where available, P-values provided by the investigators were also collected. The number of

patients included in outcome assessment, the number of patients based on intention to treat and

the number of patients lost to followup were also collected where possible. Finally, reviewers

judged the objectivity and consistency of the assessment of each outcome. These questions,

combined with those on the study characteristics form, provided the information used to identify

the methodological strengths and weaknesses of each trial.

Article Review Process

	Because of the large number of articles to review, a serial article review process was employed. In this process, the quality assessment and abstraction forms were completed by the

primary reviewer. The secondary reviewer, after reading the article, checked each item on the

forms for completeness and accuracy. The secondary reviewer also indicated the number of

disagreements with the primary reviewer for those items related to study quality. The secondary

reviews were completed by the principal investigators. We did not mask reviewers to author,

institution, or journal.

	All information from the article review process was entered in a Microsoft Access database

(Acne Evidence Database) to collect, organize, display, and report the evidence.

Peer Review

Feedback was sought from the technical experts throughout the project through ad hoc and

formal requests for guidance. Drafts of the report were sent to technical experts, as well as to

peer reviewers identified by AAD and AAP. (See Appendix A for a list of peer reviewers and

technical experts who reviewed one or more drafts of this evidence report.) We logged the dates

each reviewer’s forms were received and entered feedback requiring responses into a database.

Revisions were made to the report, as appropriate. Responses to the comments, descriptions of

the revisions and/or explanations of decisions, were sent back to reviewers and a conference call

was scheduled to discuss the comments and responses. The review process was then repeated

with the revised report.