Research Quality: Critique of Quantitative Articles in the Journal of Counseling & Development

Research, defined as "an activity conducted to increase knowledge by systematically collecting, analyzing, and interpreting data to answer carefully formulated questions about publicly observable phenomena" (Hadley & Mitchell, 1995, p. 4), is imperative to advancing a profession. Research informs what services should be provided, to whom services should be delivered, and how services should be implemented. In the counseling field, this mandate equates conducting research to help our clients, supervisees, and students. However, if research is truly to inform and advance theory and practice, then it needs to be of quality (Sink & Mvududu, 2010). "The faith one places in the conclusions made from a study is related to the quality of the study" (Wampold, 2006, p. 94), and quality is dictated by the weakest element in design, execution, and analysis (Berger, Matthews, & Grosch, 2008). Thus, quality of research lies within all stages of a study, including the literature reviewed, questions asked, research design and analyses selected, and results reported. To ensure research is meeting the goal of informing practice and theory, counselors need to make a thorough assessment of the quality of published research. Within the counseling field, most examinations have been content analyses of research articles rather than indicators of quality. Typically, a content analysis includes types of articles published (e.g., conceptual, empirical), authors and their institutions, and topics covered (e.g., Blancher, Buboltz, & Soper, 2010). Recently, a focus on samples has been included in some content analyses (e.g., Blancher et al., 2010; Erford et al., 2011; Ray et al., 2011), and Erford et al. (2011) also described the methodology and statistical analyses used as well as trends of empirical publications over time. Each of these studies included frequency counts of the samples, methodologies, and statistical analyses used but did not address the appropriateness of these nor other statistical factors, such as statistical power and effect size, that affect findings. Although content analyses are important in understanding trends in the counseling field, lacking is a focus on the quality of published research. Barrio Minton, Fernando, and Ray (2008) specifically stated they made no effort to evaluate the quality of the articles in their review. However, examining research quality is necessary as an indicator of how well researchers are informing the field. Fong and Malone (1994) published one of the few studies focused on quality of research in the field. They examined over 100 quantitative manuscripts submitted to Counselor Education and Supervision and found both research design and data analysis errors. The most common research design errors included lack of or unclear research questions, sampling errors, and instrumentation problems. For data analyses, the most predominant problems were using the wrong statistical technique and conducting piecemeal analyses for studies that included multiple variables. Although Fong and Malone's study provided important information, they focused solely on manuscripts submitted for publication in Counselor Education and Supervision, not studies actually published. Manuscripts submitted for publication and rejected may be more flawed than those that are accepted and published. In addition, manuscripts that are submitted may not necessarily reach public awareness and thus do not affect the field. Exploring the quality of published research in counseling journals is needed if counselors are to know how research is affecting their field (both negatively and positively). In addition, examining published research would provide researchers information about how to enhance quality of research, offer consumers questions to ask when reading published articles, and inform educators regarding changes in research training needed to enhance research quality. Focusing on quality or error in research can be overwhelming, because different methodologies (e. …