Moving towards the Assessment of Collaborative Problem Solving Skills with a Tangible User Interface.

The research on the assessment of collaborative problem solving (ColPS), as one crucial 21 Century Skill, is still in its beginnings. Using Tangible User Interfaces (TUI) for this purpose has only been marginally investigated in technology-based assessment. Our first empirical studies focused on light-weight performance measurements, usability, user experience, and gesture analysis to increase our understanding of how people interact with TUI in an assessment context. In this paper we propose a research agenda for assessing ColPS of individuals using the Microworlds methodology implemented on TUIs. In a first example item, we use so-called MicroDYN items, which are independent microworld scenarios that rely on structural linear equations as underlying model. As the MicroDYN approach has been thoroughly empirically investigated for the assessment of complex problem solving of individuals, it offers a good basis for a reliable and valid assessment. We describe how this approach was applied to create an assessment item for a collaborative setting. This item described in this paper implements a simplified model of a MicroDYN item related to climate change using knowledge of previous studies. Therefore, the focus of the item’s construction lies on meeting the requirements for a standardised high quality assessment. Finally, a research agenda is proposed to sketch the main research issues. INTRODUCTION In the last few years the term 21 Century Skill gained substantial visibility in scientific literature, for instance in the latest reports on technology-based assessment (OECD, 2012) as well as in the Digital Agenda published by the European Commission (Ferrari, 2012). These so-called 21st Century Skills refer to skills such as complex problem solving, collaborative problem solving, creativity, critical thinking, learning to learn, decision making, etc. (Binkley et al., 2012). Several researchers have stated that the acquisition of these 21 Century Skills and their development are only marginally investigated (Bennett & Gitomer, 2009) despite their unquestioned importance and relevance. This paper focuses on a particular 21st Century Skill, collaborative problem solving (ColPS), which encompasses the ability to successfully deal with untransparent and dynamically changing problems in a collaborative setting. ColPS is considered an important component for the success in life and for this reason will be included as transversal domain in one of the most prestigious large-scale assessments worldwide, the Programme for International Student Assessment (PISA) in its 2015 cycle (OECD, 2014). As emphasised by educators and policy makers, ColPS plays an important role in different life contexts at schools, at home, or at work (Rummel & Spada, 2005). Especially on the labour market, the organisational researchers (e.g. (CannonBowers & Salas, 1997; Hellinghausen & Myers, 1998)) emphasise more and more the importance of teamwork for organisational success and continuously look for new ways to assess collaborative skills in a recruitment process and in the course of personnel development. Despite the apparent importance of ColPS for success in different areas of life, up until now there has not been much progress in its assessment. However, recently, fast development of new technologies for gathering data in real settings allows us to assess specific 21 Century Skills such as ColPS. Due to practical limitations, conventional paper-pencil tests or even the use of desktop applications known from computer-based assessment do not entirely allow for highly dynamic and interactive assessment of 21 Century Skills. More natural forms of interTOJET: The Turkish Online Journal of Educational Technology – October 2014, volume 13 issue 4 Copyright © The Turkish Online Journal of Educational Technology 96 action such as touch, speech, gestures, and handwriting support assessment researchers to explore new approaches to skill assessment. Therefore, the availability of affordable devices that can collect such natural forms of interaction motivated us to develop several so-called tangible user interfaces (TUI) for assessment. ColPS assessment. The use of TUI in assessment is new and, therefore, knowledge about how to design those systems and how to develop test items for TUI is limited. For this reason, we present the state of the art in related research topics and propose a study that will help us shed light on the use of TUIs for ColPS assessment purposes. The aim of this article is to summarise open research issues in the domain of complex collaborative problem solving and to motivate the use of tangible user interfaces to assess this 21 Century Skill. The upcoming section summarises research on relevant topics followed by the description of the overall research methodology. Further, we elaborate how a test item related to climate change has been developed for the TUI to assess collaborative problem solving of individuals. Finally we derive a first research agenda to assess complex collaborative problem solving with TUIs. BACKGROUND Collaborative Problem Solving ColPS is one of the 21 Century Skills that has attracted the most attention from researches in recent years. This comes from the fact that collaborative work is becoming ever more significant in education and in everyday life. For instance, in schools students are often required to work in teams on science tasks; at work people collaborate on common projects; and at home families take household decisions collaboratively. An everyday example of ColPS is students organising a school trip together. They need to consider different factors such as distance, transport, or price, whereby some of factors can change in the course of planning. Students take over specific roles in the process (e.g. a person who finds bus schedules, the one taking care of hotel prices, the one who makes an overview of everything), exchange their knowledge and apply different strategies to find the nearest destination, with the optimal transport and the cheapest price. In theory, ColPS incorporates two dimensions – complex problem solving as the cognitive dimension and collaboration as the interpersonal dimension. However, ColPS is not just the sum of those two dimensions, but represents the interaction of complex problem solving skills and collaboration skills (OECD, 2014). The first dimension, complex problem solving, is one of the 21 Century Skills that has been extensively investigated. It may be defined as searching for the path from the problem state to the goal state. Its main characteristics are that the problem the person is confronted with is untransparent and the problem situation is dynamic, meaning that it changes, for instance, in the course of time or under the influence of the problem solver (Fischer, Greiff, & Funke, 2011; Lai, 2011). By adding the collaborative aspects to the complex problem solving construct leads us to the definition by O`Neil et al. (2003), who define ColPS as searching for the path from the initial state to the goal state while interacting with others working on a shared goal. Further, one of the most recognised large-scale assessments worldwide, the Programme for International Student Assessment (PISA) offers a more detailed definition of ColPS describing it as: ". the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution" (OECD, 2013). Whereas the dimensions of ColPS, problem solving and collaboration, are well examined as single constructs, the ColPS as a construct that incorporates both of those skills is still insufficiently investigated. Although recently there are initiatives researching the assessment of ColPS, especially in large scale testing context, the progress is still scarce at best. As outlined by Krkovic et al. (Krkovic, Pásztor-Kovács, Molnár, & Greiff, 2014) following aspects need to be considered when constructing the assessment for ColPS how to create a standardised testing environment, to use computer agents or humans as collaborators, how to assess communication, and how to analyse the collected data. Up until now, there have been two approaches on how to assess ColPS – human-to-agent and human-to-human. The first approach incorporates human-to-agent collaboration, the participant collaborates with a computer-simulated agent to solve a problem collaboratively. The OECD will include the assessment of ColPS in its PISA 2015 cycle uses this approach. The application of human-to-agent approach can be justified by the fact that it offers standardised stimuli to each participant ensuring a standardized testing environment and reasonable scoring procedures (Graesser, Jeon, & Dufty, 2008). Another approach to the assessment of ColPS is to use human-to-human interaction as investigated in the ATC21s project (Griffin, Care, & TOJET: The Turkish Online Journal of Educational Technology – October 2014, volume 13 issue 4 Copyright © The Turkish Online Journal of Educational Technology 97 McGaw, 2011) which in comparison to the human-to-agent approach offers more face valid situation of two or more individuals working together on a problem, providing more in-depth information about the collaboration process. However, the human-to-human approach has its limitations too, such as unstandardized assessment settings, or large log files coming from open conversations that can hardly be transferred into scoring (O’Neil, et al., 2003). Considering benefits and limitations of both approaches, researchers need to search for an optimum solution for ColPS assessment that will on one hand offer scenarios that are scoreable and on the other hand be realistic and close to real life situations. Programs such as PISA aim at large scale assessment, hence, the technology they use must