ICT teacher training: Evidence for multilevel evaluation from a national initiative

This paper is one of a pair that re-examines the evidence from a national initiative to train all teachers in England to bring them up to the level of newly qualified teachers, who are required to know when to use and when not to use information and communication technologies (ICT) in their professional practice. Reanalysis of data gathered for the 2004 evaluation uncovers the complexity of such professional development. The effectiveness of contrasting approaches to ICT-related teacher training was analysed using the national survey of 496 trainees and experts’ reports on 11 of the 47 training providers. multilevel evaluation of professional development was shown to be robust for ICT teacher training, including a significant correlation between the views of experts and those of teachers. The presence of the middle level of ‘organisational support and change’ emerged as a particularly discriminating factor, indicating that higher-quality teacher training supports change in the classroom and in the school. Therefore, we recommend that all five of Guskey’s levels be consistently adopted for the evaluation of ICT teacher training, and that research should also adopt a multilevel model. A second paper delves deeper to describe and contrast the highest and lowest-rated approaches to ICT teacher training. Introduction Information and communication technologies (ICT) continue to permeate society, accompanied by calls for equity, including United Nations Educational, Scientific and Cultural Organization’s (UNESCO) mandate resulting from the World Summit on the Information Society. There are calls for ICT teacher training on a massive scale (IFIP, 2005; Hinostroza, 2008). Research has also established that effective technologyBritish Journal of Educational Technology Vol 40 No 1 2009 135–148 doi:10.1111/j.1467-8535.2007.00808.x © 2007 The Authors. Journal compilation © 2007 Becta. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA. related professional development is an important pillar for successful integration and sustainability of ICT in education (Culp, Honey & Mandinach, 2003). But what is effective ICT teacher training and how can it be evaluated and scaled up globally? One extreme argues for a machine-centred, computer-based training, and the other argues for a people-centred approach that scales up expertise within the school and region. This paper is the first of a pair that provides evidence from a national initiative to support the more people-centred approach, which recognises that ICT teacher training impacts multiple ecologies, including the classroom, school and region. An important framework to evaluate teacher professional development in general was provided by Guskey (2002) in his award-winning paper in Educational Leadership. Guskey laid out five critical levels for the evaluation of professional development in general: (1) participants’ reactions, (2) participants’ learning, (3) organisational support and change, (4) participants’ use of new knowledge and skills, and (5) students’ learning outcomes. Guskey (2002) cautioned that ‘[w]ith each succeeding level, the process of gathering evaluation information gets a bit more complex. And because each level builds on those that come before, success at one level is usually necessary for success at higher levels’ (p. 46). Detailed case studies of teachers innovating with ICT over the period of a year provide evidence to support Guskey’s levels—for example, studies by Zhao, Pugh, Sheldon and Byers (2002) illustrate the importance of school support in addition to ICT teacher training. However, studies that compare the effectiveness of different approaches to ICT teacher training are rare, because ICT-related teacher training normally varies with each context. A national initiative in England that stimulated a range of approaches to ICT teacher training within a common evaluation framework provided an opportunity to study different approaches of ICT teacher training with the same population of teachers. The British government had been funding computers for schools since the 1980s, but as Opie and Katsu noted in 2000, the use to which ICT could be put across the curriculum was limited. This provided the impetus for the national ICT teacher training in 1999– 2003 known as ‘New Opportunities Funding’ or NOF, after its source of charitable funding. NOF training stimulated development of a consortia of alternative training providers that were subject to accreditation by the TTA (Teacher Training Agency, currently the Training Development Agency). Some providers also aimed to export their services abroad following deployment in Britain. The ambitious NOF ICT teacher training initiative became controversial when it received critical reviews in the press, which resulted in several rounds of evaluation leading to improvements in the teacher training. Condie and Munro’s (2007) landscape review of the impact of ICT in schools provides the following widely held view of the NOF ICT teacher training initiative: These programmes were resourced through New Opportunities Funding (NOF) and often involved attendance at a series of staff development sessions which focused primarily on developing technical competence and understanding rather than on pedagogical principles. Beastall (2006) describes the NOF programmes as ‘largely unsuccessful’ and evidence from other studies supports this (Conlon, 2004; McCarney, 2004; Condie et al, 2005; Stark et al, 2002; Ofsted, 2004c). (p. 19) 136 British Journal of Education Technology Vol 40 No 1 2009 © 2007 The Authors. Journal compilation © 2007 Becta. However, as noted by Preston (2004, p. 7), the TTA’s original request for proposals called for a consortia of training providers ‘to design a programme which focused on the pedagogy of using ICT in the classroom, rather than on basic skills training. The intention was that where basic ICT skills were required, then the training would be given within a pedagogical context or the schools would arrange separate training prior to starting the NOF programme’. Further study of the variety of approaches to ICT teacher training is warranted, particularly now that the NOF ICT teacher training controversy has subsided, as the data gathered on this initiative (Preston, 2004) could inform the understanding of ICT professional development and identify the characteristics of effective ICT teacher training. This first paper of a pair provides evidence to support multilevel evaluation, while a second paper delves deeper to inform design of ICT-related teacher training and theoretical understanding of the process (Davis, Preston & Sahin, submitted). The data sources The TTA was responsible for quality assurance of the national ICT teacher training in England from 1999 to 2003. It undertook this by two methods: (1) quality assurance evaluation by ICT teacher training experts; and (2) surveys of a nationally representative sample of trainees. One nationally representative survey of teachers and the ICT experts’ reports on 11 approaches to teacher training are the source of evidence for this study. This paper briefly describes the data collected through the survey of teachers and analyses it to examine the applicability of Guskey’s (2002) five-level model of professional development evaluation, followed by a complementary analysis of the experts’ reports. The survey of a nationally representative sample of 1000 teachers in England was undertaken at the end of 2002, while some NOF-funded teacher training was still underway. The large majority of these teachers had finished their training because this was at the end of the national initiative. This final survey, along with other evidence, was used by the MirandaNet Fellowship to provide a national report on the initiative (see Preston, 2004). The consortium that undertook this evaluation was led by MirandaNet, and included researchers from the Institute of Education, University of London, and that university’s procedures were followed for ethical approval. The reanalysis also fell within the procedures of the same institute. TTA office staff had introduced the original two-page survey questionnaire to widen its sample base. It sent the survey by postal mail, and it was returned in the same way or by fax. The cover letter noted that the data would be utilised both for quality assurance and for research by MirandaNet. No reminders or inducements were used beyond the incentive to stimulate improvement of ICT teacher training through feedback to providers based on this information. The MirandaNet research team coded the data, and the final section of comments was entered into a separate file grouped by approach to the training. These MirandaNet files were analysed further by the authors for this study. ICT teacher training multilevel evaluation 137 © 2007 The Authors. Journal compilation © 2007 Becta. Teachers’ evaluation of their ICT training: descriptive results At the end of 2002, MirandaNet had gathered a total of 638 surveys for the national evaluation of ICT teacher training, covering 40 of the 47 ICT teacher training providers that had participated in the national initiative (Preston, 2004). The evidence discussed in this paper draws upon this opportunity sample covering 13 different approaches from 11 providers of ICT teacher training in England, for which quality assurance reports and at least 15 survey responses were accessible. The use of a sample of the national data has helped the authors maintain the anonymity of the providers, which is important for ethical reasons. A total of 496 surveys were used in this analysis, slightly over half of which were from secondary teachers, and the remainder were from primary teachers (Table 1). Responses from librarians and those who taught both primary and secondary students were excluded due to the small number of such respondents. As can be seen i