The power of PISA – limitations and possibilities for educational research

On 6 December 2016, the Programme for International Student Assessment (PISA) releases its report on the achievements of 15-year-olds from 72 countries and economies around the world. This triennial international survey aims to evaluate education systems across 72 contexts by testing skills in Mathematics, Science and Reading Literacy. This is the sixth cycle of PISA and the OECD suggests countries and economies now have the capability to compare the results over time to ‘assess the impact of education policy decisions’1. Compared to other education studies, the media coverage of PISA must be described as massive (Baird et al., 2016; Meyer & Benavot, 2013) and, as with previous years, it is expected that PISA will attract considerable discussion among policy-makers, educators and researchers (Wiseman, 2014). It is therefore timely to present a thematic issue of Assessment in Education, where we publish four articles that have analysed previous data-sets from the PISA studies each commenting upon the challenges, limitations and potential future assessment research on the PISA data. The articles touch upon issues regarding sampling, language, item difficulty and demands, as well as the secondary analyses of students’ reported experiences of formative assessment in the classroom. One important message from the authors in this thematic Special Issue is the need for a more complex discussion around the use and misuse of PISA data, and the importance of pointing to the limitations of how the results are presented to policy-makers and the public. In an area where the media produces narratives on schools and education systems based upon rankings in PISA, researchers in the field of large-scale assessment studies have a particularly important role in stepping up and advising on how to interpret and understand these studies, while warning against potential misuse. In 2014, Yasmine El Masri gave a keynote at the Association for Educational Assessment-Europe conference in Tallinn, Estonia, following her Kathleen Tattersall New Researcher Award. We are pleased to publish the paper based upon her DPhil research: Language effects in international testing: the case of PISA 2006 science items. Together with Jo-Anne Baird and Art Graesser, El Masri investigates the extent to which language versions of the PISA test in Arabic, English and French are comparable in terms of item difficulty and demand (El Masri et al., 2016). As there is an ongoing discussion on whether it is possible to assess in a fair manner and compare science, mathematics and reading performances across countries and cultures, this present study offers important findings for future research. Using released PISA items, El Masri et al. show how language demands vary when comparing Arabic, English and French versions of the same item, and hence could impose different cognitive demands on the students participating in the PISA test in different countries. With the expansion of PISA to other countries through PISA for Development and the need for fair comparisons across countries, El Masri et al. suggest that subsequent research could explore the possibility of investigating computational linguistics approaches in test transadaptation as an alternative to the use of expert judgement which is the current practice in international test development. The next article in this issue by Freitas, Nunes, Reis, Seabra, and Ferro (2016), Correcting for sample problems in PISA and the improvement in Portugese students’ performance, reports a