TRAINING IN ASSESSING COGNITIVE ERRORS AND COPING 1 Running Head : TRAINING IN ASSESSING COGNITIVE ERRORS AND COPING Training Effects in Using the Observer-Rated Cognitive Errors and Coping Action Patterns Rating Scales

The use of observer-rated rating scales requires the training of raters until they become reliable in using the scales. However, few studies properly report how training in a given rating scale is conducted or how it should be conducted. This study examined progress in inter-rater reliability over 6 months of training in two observer-rating scales, the Cognitive Errors Rating Scale and the Coping Action Patterns Rating Scale. The evolution of the IntraClass Correlation Coefficients was modelled using Hierarchical Linear Modelling (HLM). Results showed an overall training effect, as well as an effect of the basic training phase and of the rater calibration phase, with the latter being smaller than the former. The results are discussed in terms of implications for rater training in psychotherapy research. Key-Words: Observer-Rating, Process Research, Rater Training, Reliability, Cognitive Errors, Coping Action Patterns TRAINING IN ASSESSING COGNITIVE ERRORS AND COPING 3 Training Effects in Using the Observer-Rated Cognitive Errors and Coping Action Patterns Rating Scales Observer-rated scales applied to different types of interview formats (e.g., therapy session, assessment interview) are common in psychotherapy research. By means of observerbased methods, relevant clinical and research concepts, such as defense mechanisms (Perry & Henry, 2004) or core conflictual relationship themes (Luborsky, & Crits-Christoph, 1998) may be assessed reliably by trained raters. An important implication of the use of rating scales in psychotherapy research is the necessity of rater training, which is usually highly timeand cost-intensive (e.g., Lorber, 2006; Schanche et al., 2008). Thus, there is a need to define and investigate efficient and cost-effective training formats. The present report aims at investigating a two-phase training format formulated for the training for assessment of defense mechanisms (Perry & Henry, 2004): (1) basic training and (2) rater calibration phase. During the first phase, the participants learn in a small group the concepts of the rating scales, accommodate to the rating manual that describes the method and learn to apply the rating system to selected training cases. Problems and questions are discussed in the group, in the presence of an expert. At the end of this phase, the raters should be able to rate the training cases reliably. During the second phase, the raters are confronted to new material, not pre-selected, but stemming from various research projects. The participants rate individually and discuss their views with the expert view, and later with another participant’s view. Questions and problems are discussed in individual supervision sessions provided by the expert. The calibration phase aims at avoiding “rater drift”, i.e., ratings becoming more and more divergent over time without regular supervision sessions (Perry, & Henry, 2004). Such a training format is a time-consuming process; the duration, total time investment and total cost may depend on the complexity of the constructs assessed, the TRAINING IN ASSESSING COGNITIVE ERRORS AND COPING 4 precision of their operationalization and of the rater training, the initial competency of the raters, the size of the training group, among other factors. A few studies have investigated these factors influencing efficiency and costeffectiveness of the rater training. For the basic rater training, Schanche et al. (2008) presented data showing that the reliability after a training with university clinical psychology students with minimal clinical experience was satisfying, in particular when the training procedure focused on less concepts at a time. In a controlled study, Lorber (2006) showed that ratings of harsh parental discipline provided by minimally trained raters were as good as the ones provided by raters that were trained according to a very time-consuming “golden standard” procedure, assessed at a one-time characterization of the subjects. However, when looking at the differences between means of the raters, Lorber reported very poor reliability coefficients for the minimally-trained raters. Finally, for the rater calibration phase, a rater training format focusing specifically on the problematic categories for the specific rater may be a cost-reducing strategy, as suggested by Tractenberg, Schafer and Morris (2001) in the context of clinical dementia ratings. In conclusion, there remains a dearth of research into the training effects of psychotherapy research observer-rated scales (Cichetti et al., 2006; Invancevich, 1979). In particular, no study has examined the evolution of training effects – defined as the increase of rater competency in the particular rating scale as a function of training phase. The present study aims at addressing this question by examining change in inter-rater reliability coefficients over the course of a training in applying two observer-rated methods used in cognitive behavioral psychotherapy process and outcome research : the Cognitive Errors Rating Scale (Drapeau, Perry, & Dunkley, 2008) and the Coping Action Patterns Rating Scales (Perry, Drapeau, Dunkley, & Blake, 2005). We aim at describing the evolution of training effects in these specific rating scales across both training phases.