title: Concensus About Program Quality: An Australian Study In Early Childhood Special Education
Creators: name: Beamish, Wendi
Publisher: name: Griffith University
Year: 2004
Subjects: subject: Special education
subject: educational program quality
subject: early childhood intervention
subject: Queensland Department of Education
subject: Australian Early Intervention Association

description: The research presented in this thesis involved an innovative exploration of program quality in the Australian early childhood intervention (ECI) service system. The overall aim of the inquiry was to develop a local service evaluation tool for use in quality measurement and improvement activities. Tool development and research effort focused on the generation of a manageable, credible, and contextually relevant list of Program Quality Indicators (PQIs). The inquiry drew on the literature from ECI service systems and practice, program evaluation and service quality, and best practice educational research. These converging literatures rendered support for a stakeholder-led, contextual approach to evaluative tool development. Hence, this research was couched within a participatory action research (PAR) framework, guided by best practice principles, and undertaken in specific Australian service contexts. Across a 5-year period (1994-1998), a compact list of PQIs was generated by staff from a large governmental ECI service in Queensland (Stage 1), validated by staff and parents throughout that service (Stage 2), and validated across the nation (Stage 3). In Stage 1 (1994-1995), the researcher worked in partnership with staff from the Queensland Department of Education's early intervention service, ESE(Q), to develop a contextualised list of PQIs for the specialised service. List development was undertaken across two interconnected PAR-based activities. First, a small focus group of ESE(Q) staff (N = 10) participated in a search-based workshop in order to complete a functional environmental scan of conditions underlying practice at a systems level. Second, an expert group of ESE(Q) teachers (N = 19) from programs throughout the state participated in a modified Delphi procedure in order to develop a PQI listing. Seven rounds of Delphi activity were necessary to generate practices, to refine judgements, and to reach consensus about which practices were PQIs for the service. During the iterative process, these teachers drew on the contextual information from the environmental scan and on their local theory and practice to generate a comprehensive list of 164 practices and then to extract a compact set of 31 core PQIs. In Stage 2 (1996-1997), ESE(Q) staff and families throughout the state were surveyed in order to socially validate the identified PQIs and to gather additional information about their implementation. For each of the 31 PQIs, participating teachers, therapists, teacher aides, and parents were asked to (a) indicate their level of acceptance, (b) report on current use, and (c) comment on barriers to implementation. A comprehensive exploration of the large PQI data set followed, with analyses affording opportunity to apply two novel methodological procedures: data slicing and a PAR-based acceptance consensus criterion. Results from participants (N = 370) revealed a pattern of 'collective' consensus on PQI acceptance (i.e., agreement across all participants), which was matched by a pattern of group consensus on PQI acceptance (i.e., agreement among participating teachers, therapists, teacher aides, and parents). Moreover, high PQI acceptance was accompanied by somewhat lower PQI implementation. Furthermore, lack of time and lack of staff were identified as the primary barriers to PQI implementation throughout the service. In Stage 3 (1997-1998), targeted members from within a national organisation, Australian Early Intervention Association, were surveyed in order to probe PQI acceptance in contexts outside the ESE(Q) service. National PQI acceptance data were examined using procedures similar to those adopted for the analysis of ESE(Q) survey data. A key finding from this nationwide sample of participating staff and parents (N = 236) in specialised ECI services pointed to the broad applicability of the Queensland PQIs within Australia. Despite participants being drawn from a range of geographical, historical, disciplinary, and service contexts, results revealed a pattern of 'collective' consensus on PQI acceptance. Results also extended understandings about the Queensland PQIs and confirmed the usefulness of the data slicing procedure and the PAR-based acceptance consensus criterion. Taken collectively, this lengthy research program provided a new contextual model for best practice documentation, a unique service-specific practice listing, and some fresh insight into practice relevance. It also provided occasion to demonstrate the benefits of using the two alternative data analysis procedures. These functional outcomes, however, were made possible only through the considered application of PAR. PAR proved to be a robust methodology that facilitated communication, collaboration, and consensus about program quality among Australian ECI stakeholders.